[Libmesh-users] Problem with parallel execution of element loop

[jwpeterson]

The following is from an email posted to the libmesh-users mailing list.

The following code fragment is part of a program that behaves correctly when invoked to run on a single processor (ie using : mpirun -n 1 main)

    libMesh::BoundaryInfo& bdry_info = tet_mesh.get_boundary_info();
    for (auto pp_elem = tet_mesh.local_elements_begin(); pp_elem != tet_mesh.local_elements_end(); ++pp_elem)
    {
        Elem* elem = *pp_elem;
        for( unsigned int is = 0; is < 4; ++is)
            if ( /* stuff */)
                bdry_info.add_side(elem, is, sideset_id);
    }

However, when I invoke it with several processors (ie using mpirun -n 4 main), the BoundaryInfo object does not contain all of the sidesets that it contains on a single processor.

I suspect that the issue here is that several processors are writing to the same memory location (ie the BoundaryInfo object) without appropriate synchonization/mutex code.

I certainly didn't add any such code.

Is my suspicion correct or are there other problems with the code?

In the former case, can someone point me to some correct code in which mesh elements are processed in parallel and a common data structure is modified by each thread/process?

dan

[jwpeterson]

I think you are seeing the expected behavior since you are looping over "local" elements. The BoundaryInfo object is not shared by the processors... there is a separate copy of the BoundaryInfo object on each processor and each will contain only boundary info about local elements after running the loop above. The BoundaryInfo class also provides the

  void parallel_sync_side_ids();
  void parallel_sync_node_ids();

APIs for syncing this data across processors.

[dan-lior]

I just naively added the line

bdry_info.parallel_sync_side_ids();

right after the loop above and it seems to have solved the problem.

Thanks John!

PS Not to seem ungrateful, but I searched for just such a synchronization function in the online documentation for the BoundaryInfo class and didn't find it. Perhaps it comes from a parent class?

dan

[jwpeterson]

libmesh/include/mesh/boundary_info.h

Lines 722 to 731 in 0f2c5cf

	/**
	* Synchronize the boundary element side and node across processors.
	* This function is needed when boundary info is changed by adding or removing
	* sides on the fly.
	* Note: if the side of a ghost element is changed, then you would need to do
	* do parallel push (see e.g., timpi/parallel_sync.h) and then sync.
	*/
	void parallel_sync_side_ids();
	void parallel_sync_node_ids();

Most likely the issue is that our Doxygen documentation is not up-to-date for this class, so if you searched for it there, you would not have found it. I would recommend using the Doxygen docs maintained by the MOOSE project, BounadryInfo, as they are automatically updated on a more regular basis.