[3.18.x] A few LMDB related fixes and improvements #5380
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Data stored in LMDB and accessed with the 'MDB_val.data' field is a pointer to a map of the LMDB file (set up with mmap()). This doesn't ensure proper alignment and so in order to avoid potential SIGBUS due to bad alignment (on architectures sensitive to these issues), we need to create local copies of the data before trying to access them in any way than just one byte at a time. Ticket: CFE-3373 Changelog: None (cherry picked from commit a7fbac9)
We have that constant so we should use it. Ticket: ENT-8201 Changelog: None (cherry picked from commit e6bd200)
So that we can check how full DBs are. For now only implemented for LMDB where we set/know the maximum size. Ticket: ENT-8201 Ticket: CFE-2136 Changelog: None (cherry picked from commit b775f2e)
Ideally, locks should never need to be cleaned because the ones
used by the current policy set should be periodically updated and
the old ones that no longer match anything in policy should be
very few in numbers and thus taking very little space. And we
want to retain locks for more than 4 weeks whenever possible
because for example locks used by our daemons to prevent multiple
instances from running should stay intact while our daemons run
which should ideally be for the whole boot time of a machine.
Also, our documentation doesn't specify any hard expire interval
for locks (any upper limit) so if possible, we should retain and
respect even very old locks.
If the locks DB grows too big, however, we need to do purging,
potentially even more aggressively than just with the 4 week
horizon if needed.
Ticket: ENT-8201
Ticket: CFE-2136
Ticket: ENT-5898
Changelog: CFEngine locks are now purged dynamically based on the
local locks DB usage/size ranging from no purging (<=25%
usage) to purging of locks older than 1 week (>75% usage)
(cherry picked from commit 0a45508)
Declare variables when they are used and give them better names. Also use the nicer names for DB types. Ticket: ENT-8201 Changelog: None (cherry picked from commit a2159f9)
Almost full LMDBs are dangerous, so they should be rotated as part of `cf-check repair`. Ticket: CFE-3374 Changelog: `cf-check repair` now rotates DB files with high usage (>95%) (cherry picked from commit 11ed41e)
If an LMDB database is full, we get a special error from LMDB
when trying to use the DB giving us an option to handle the
situation.
We could try to prune/purge the DB, but there's no general rule
for which data to preserve so we just move the file aside with a
`.rotated.$timestamp` extension.
Continuing (restarting, really) with a new empty LMDB file is
better than getting stuck. So it's a last-resort solution, but
our LMDB files should never get full.
Ticket: ENT-8201
Changelog: Full LMDB files are now handled gracefully by moving
them aside and using new empty LMDB files
(cherry picked from commit bddc4e0)
`cf-check diagnose` supports the `--test-write` option to check
if we can write into the given files. `cf-check repair` uses the
same code to identify files that need repairing so it should
provide the same option.
Ticket: CFE-3375
Changelog: `cf-check repair` now supports the `--test-write`
option to check if DBs can be written to as part of
identifying DBs that need repairing
(cherry picked from commit 5ed5585)
Plus the information if we think the DB needs rotation or
not. Almost full LMDBs are problematic and should be moved out of
the way.
Ticket: none
Changelog: `cf-check diagnose` now shows DB usage and a hint if
rotation is required
(cherry picked from commit 6141736)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Cherry-pick of #4215.