In situ Data Compression

One of the goals in the design of MrHyDE is to optimize for both performance and memory usage. In many finite element applications, the user has options to use more memory, e.g., to store information at quadrature points, rather than recomputing whenever needed. While the savings in doing this can be significant, it may be impractical for large-scale applications or on computational nodes with limited memory. Indeed, many applications are memory-limited rather than compute/flop-limited. MrHyDE does allow the user to store a proportion of the data and recompute the rest as needed. While this allows the user to balance memory usage and performance, more effective strategies based on data-compression are also available.

MrHyDE has a very simple capability to detect and exploit structure in the form of redundancies in this data. As a preprocessing step, it can determine the set of unique reference-to-physical Jacobians and then store only the data associated with the unique entries. This database/dictionary is then used whenever this information is required. For some applications, e.g., those utilizing uniform or extruded meshes, the savings can be nearly 99%. As an added bonus, by storing and reusing this data frequently, less memory is moved around and the applications may run up to 20% faster in our experience. In the figure below, we plot the pareto front associated with varying the storage proportion from 0% to 100%. We also plot the data point associated with using the data-compression.

Of course, not every mesh has this redundancy. Recent work has sought to enhance this redundancy through a novel r-adaptive mesh refinement procedure, but this is not incorporated into MrHyDE. We suggest that one should only use this in situ data compression when this structure is known to be present.

To enable this database compression for the basis functions at the quadrature points, set the following flag in the Solver block in the input file:

Solver:
    use basis database: true

The default is false.

For similar reasons, the mass matrices often also have the same redundancies. To enable this compression, set

Solver:
    use mass database: true

in the Solver block. This mass matrix compression should only be used if the following settings are also enabled:

Solver:
    matrix free: true 
    fully explicit: true 
    use custom PCG: true 

The fully explicit mode is described in more detail elsewhere, but it is designed to be ultra-low memory and optimized performance for explicit time integration methods. Storing the full mass matrix gives up the ability to compress data since redundancies in the matrix entries are not exploited by Trilinos linear algebra. Finally, the custom preconditioned conjugate gradient (PCG) can utilize this compressed mass format when computing the action of the mass matrix on a vector.

Finally, it is possible to compress the quadrature points and weights themselves when the mesh is uniform and they are based on tensor products. In this case, only the 1-dimensional data needs to be stored. To enable this, set

Solver:
    use ip database: true