Feature/mpi #53
Feature/mpi #53
Conversation
This reduces the number of objects required to instantiate a model. Instead, a user can simply enable domain decomposition during mesh creation. At the moment, this option is provided only when reading in a mesh.
fluidnumerics-joe
added
the
work-in-progress
Codecov ReportAttention: Patch coverage is
📢 Thoughts on this report? Let us know! |
|
There seems to be a problem with the |
Also adds gpu-direct for vector 2d
There was a problem hiding this comment.
There's a number of cleanup items we can do here. We will want to make note in #51 and in the v0.0.1 release notes that we have not explicitly tested meshes with flip=1 in 2-D and flip>=1 in 3-D, though these bits are coded up in a way we believe is correct.
We need to add documentation on how to build and verify OpenMPI with GPU-Awareness for ROCm and CUDA platforms in the docs/Learning/dependencies.md . Additionally, we will want to add documentation for developers on how the CMake build system checks for GPU awareness, should this ever need to be updated/patched in the future.
Last, add an example for advection-diffusion-2d and advection-diffusion-3d and include a short writeup for each in the documentation that demonstrates how to run this in single-domain and multi-domain mode.
While putting in these tests, I was able to resolve a number of previously undetected errors with MPI read/write
|
Some of the adjustments required for MPI have made it quite cumbersome for a user to add their own model. To remedy this, I'm going to finalize this PR with some adjustments to the backend routines that are provided for the |
2-d and 3-d coming next Also, updated the armory superci configuration file
The null models provide coverage for the built in template functions that are supposed to provide no flux divergence and no source. The null modules also provide a template for folks that want to build their own models using SELF.
Relates to Issue #51
This feature adds fully tested MPI implementations for scalars and vectors in 2-D and 3-D. The primary changes are in the
SideExchangeroutines where asynchronous (isend/irecv) pt2pt messaging is used to sendboundarydata and receive intoextboundarydata. MPI+GPU implementations specifically assume that GPU Aware MPI is available and GPU pointers are used in the message passing.We have also added parallel reduction for entropy calculation (global grid integration) and cleaned up the model API to make adding new models quite easy. Specifically, we have removed the need to be aware of loop ordering, MPI call requirements, or GPU acceleration requirements to be able to have a model implemented rather quickly. Further optimization for GPU platforms is possible from the first model specification, but is still quite reasonable out of the box.