Add support for an MFEM finite element backend

[lindsayad]

Motivation

We are migrating the work by UKAEA done in Platypus to MOOSE so that users may create MOOSE input files that leverage MFEM as the finite element backend

Design

• Add MOOSE configure option for MFEM
• Copy files from Platypus to MOOSE
• Update files to comply with MOOSE standards
• support mirroring of data from MFEM mesh and algebraic data structures onto a libMesh mesh and equation systems a la ExternalProblem so that no new transfers have to be written for multiphysics sims mixing mfem and libMesh

Impact

Gives users additional choices for finite elements and also potential for GPU computing through MFEM

[lindsayad]

I've added a data mirroring bullet motivated by conversation with @friedmud. @alexanderianblair I believe you all have been thinking about transfers recently? I think we could possibly avoid having to write new transfer objects

[alexanderianblair]

We went down a ExternalProblem/syncSolutions route in Platypus's predecessor Apollo to synchronise nodal and elemental variable data between MFEM and libMesh representations (eg. here).

Are you thinking of avoiding writing new transfer objects by just converting MFEM data to a libMesh representation, transfer, and then converting back from libMesh to MFEM (if required)? In the MFEM<->MFEM subapp case, I'm not sure this makes sense, since there are FE types and orders not available in libMesh that would prevent that intermediate step.

In the MFEM<->libMesh case, it's an option, but philosophically I'm hesitant in mixing MFEM and libMesh representations in the same subapp - it seems that applying DoF permutations etc. for converting between MFEM and libMesh representations should be a Transfer's job. We should definitely avoid building both a libMesh and MFEM mesh for the same subapp though unless absolutely necessary.

[lindsayad]

Very much agree that we should not unnecessarily build a libmesh representation when it's not required

@friedmud brought up the idea of mixing libmesh and MFEM objects within a single input file as one way that MOOSE users might begin dipping their toes in MFEM land. Of course this would not do much for you (or it may actively hurt you) from a performance standpoint, but it could be beneficial pedagogically. There could also be instances in which you want to do "fully coupled" physics with some capability that is only in wrapped MFEM (e.g. missing FE types and orders) and some other capability that is only in wrapped libMesh (e.g. I do not know the contact capabilities in MFEM well yet and I don't believe you all have done any wrapping of it either way)

[lindsayad]

It sounds like you ultimately concluded the ExternalProblem route wasn't the right fit?

I am definitely leery of having to write new data transfers. It has taken us years and years (work done by @friedmud, @fdkong, @GiudGiud) to increase the robustness and functionality of transfers and the process is still ongoing. And now we are throwing in the added complexity of mixing finite element backends.

Of course as you say, our existing transfers will not help in an MFEM-to-MFEM transfer context

[GiudGiud]

If you can write a functor material that provides a MFEM field value on an ElemArg / ElemPointArg, I can write a general MultiAppGeneralFieldFunctorTransfer that will do MFEM -> LibMesh pretty easily (and contribute some nice functionality for libMesh<->libMesh too)

[alexanderianblair]

Very much agree that we should not unnecessarily build a libmesh representation when it's not required

@friedmud brought up the idea of mixing libmesh and MFEM objects within a single input file as one way that MOOSE users might begin dipping their toes in MFEM land. Of course this would not do much for you (or it may actively hurt you) from a performance standpoint, but it could be beneficial pedagogically. There could also be instances in which you want to do "fully coupled" physics with some capability that is only in wrapped MFEM (e.g. missing FE types and orders) and some other capability that is only in wrapped libMesh (e.g. I do not know the contact capabilities in MFEM well yet and I don't believe you all have done any wrapping of it either way)

The main reason I'm keen to avoid returning to an ExternalProblem based route - at least how we explored things in Apollo - for sharing data with an MFEM problem is the added complexity in having a base Problem that has the responsibility to add either libMesh or MFEM objects to their respective sub-problems. The interface between the two can easily end up less well defined than in the Transfer case, since mirrored variables/meshes (and any data they contain) can be set up on both the sub-problems simultaneously. In the Transfer route, checking and setting data in mirrored variables is explicit, since it's clear that there is no guarantee that mirrored variables have consistent data until the transfer has taken place. For the ExternalProblem case, implicitly assuming that the mirroring relationship between a pair of MFEM and libMesh objects that existed at construction time is still valid at some later time in the simulation is easier to do.

The ExternalProblem route definitely doable, of course - there's no reason the above can't be designed around with sufficient checks and safeguards in place, and we have the Apollo implementation already - but I'm not currently convinced that the use case of wanting fully coupled mixed libMesh/MFEM problems in the same sub-app is significant enough to warrant the drop in simplicity of assuming that the sub-app is running one type of problem or the other.

Of course, if the effort required to introduce a new set of Transfers outweighs the benefits of avoiding an ExternalProblem, then that changes things. For the simple CopyTransfer case, we've started work transfers between MFEM subapps (here and on MFEM->libMesh transfers.

If you can write a functor material that provides a MFEM field value on an ElemArg / ElemPointArg, I can write a general MultiAppGeneralFieldFunctorTransfer that will do MFEM -> LibMesh pretty easily (and contribute some nice functionality for libMesh<->libMesh too)

We can easily define a GridFunctionCoefficient that will provide a method to return a MFEM field value from an arbitrary point in a functor-like way, if that's what's needed? GSLib can also be used to fetch field values at an arbitrary set of spatial coordinates quickly from a vector of them (which we've used previously for Apollo postprocessors).

[GiudGiud]

We can easily define a GridFunctionCoefficient that will provide a method to return a MFEM field value from an arbitrary point in a functor-like way,

sounds perfect to me.
Are you thinking of Transfers only in the context of MultiApps or are you thinking the Problem will have two meshes (one libMesh, one MFEM), two systems, and will need to pass a few (but not all) fields from one system to the other? In that case we could create "SystemTransfers" in addition to MultiAppTransfers

[alexanderianblair]

We can easily define a GridFunctionCoefficient that will provide a method to return a MFEM field value from an arbitrary point in a functor-like way,

sounds perfect to me. Are you thinking of Transfers only in the context of MultiApps or are you thinking the Problem will have two meshes (one libMesh, one MFEM), two systems, and will need to pass a few (but not all) fields from one system to the other? In that case we could create "SystemTransfers" in addition to MultiAppTransfers

I'm thinking of starting with just the former (Transfers in a MultiApp context), to keep things simple for the Problem - but SystemTransfers would indeed help if we find we do need to mix MFEM and libMesh in the same Problem later on.