Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

flux-accounting job-archive: implement purge of old jobs #355

Closed
Tracked by #356
cmoussa1 opened this issue May 19, 2023 · 1 comment
Closed
Tracked by #356

flux-accounting job-archive: implement purge of old jobs #355

cmoussa1 opened this issue May 19, 2023 · 1 comment
Assignees
Labels
new feature new feature

Comments

@cmoussa1
Copy link
Member

Broken off from a comment in #353, fetching new jobs periodically and adding it to the flux-accounting DB will grow the DB incredibly large. And since job-usage and fair share (by default) only consider jobs up to a certain point, perhaps a purge_old_jobs () function should be considered to be added (side note: I think flux-core is considering different methods of storing old jobs "forever" in some sort of historical storage, so it wouldn't be flux-accounting's responsibility to keep jobs forever at this moment).

flux-accounting currently keeps a timestamp of the t_inactive of the last completed job it saw when it fetches jobs from the flux-core job-archive DB as well as values for how long to consider jobs for job-usage, so it could probably make use of both of those (most likely just the latter) to decide when to purge jobs from its job-archive.

@cmoussa1 cmoussa1 added the new feature new feature label May 19, 2023
@cmoussa1 cmoussa1 self-assigned this May 19, 2023
@cmoussa1
Copy link
Member Author

This should now be fixed by #459 with the introduction of the scrub-old-jobs command. Can re-open more specific issues if ones arise.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
new feature new feature
Projects
None yet
Development

No branches or pull requests

1 participant