mmap()'d Ring Space #135
mmap()'d Ring Space #135
Conversation
…ped file to create a ring buffer that is larger than the system memory at the expense of access speed. Hopefully the speed is not too much of an issue for some pipelines.
…ned up the space accesssibility checkers in both C++ and Python.
…ssociated with the mapped ring space live.
…what it controls.
|
I'm generally happy with how it works but the specification of the base directory in |
Codecov Report
@@ Coverage Diff @@
## master #135 +/- ##
==========================================
- Coverage 58.78% 58.56% -0.22%
==========================================
Files 63 64 +1
Lines 5095 5112 +17
==========================================
- Hits 2995 2994 -1
- Misses 2100 2118 +18
Continue to review full report at Codecov.
|
…ileutils.hpp a la PR ledatelescope#132.
…. Updated src/Makefile to try to make the mapped path less fiddly.
There was a problem hiding this comment.
Seems like useful functionality and at a glance all looks in order to me. Again would defer on changes to memory.cpp to others for review.
Random note: flock can act strangely over NFS mounts. But I think using an NFS mount as a disk buffer is a pretty unlikely choice and you'd kinda expect it not to work anyways.
Codecov Report
@@ Coverage Diff @@
## master #135 +/- ##
==========================================
- Coverage 61.62% 58.71% -2.91%
==========================================
Files 64 64
Lines 5300 5302 +2
==========================================
- Hits 3266 3113 -153
- Misses 2034 2189 +155
Continue to review full report at Codecov.
|
…iable can be used to override the default mmap()'d ring location for a pipeline.
…tion of where the mapped rings live.
…he user in user.mk about its existence.
|
I think the environment variable approach is viable and takes care of my last concern. |
|
Well, at least mapped compiles on MacOS now. That's something. |
|
Keep the logging directory and the mapped ring directory separate on mac seems to have helped with a lot of the testing failures but not there is a segfault when the test suite exits. |
(Noticed this ×3 in CI.) ``` memory.cpp: In destructor 'MappedMgr::~MappedMgr()': memory.cpp:130:17: warning: catching polymorphic type 'class std::exception' by value [8;; https://gcc.gnu.org/onlinedocs/gcc/Warning-Options.html#index-Wcatch-value= -Wcatch-value=8;;] 130 | } catch( std::exception ) {} | ^~~~~~~~~ ```
|
Looks like there are some problems related to the recent changes to |
This PR adds a new mmap()'d space, named "mapped", for Bifrost rings. This will allow for larger disk-based rings for buffering data.