Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

0.7.290 memory leak during apero_database.py --update #787

Open
larnoldgithub opened this issue Oct 15, 2024 · 2 comments
Open

0.7.290 memory leak during apero_database.py --update #787

larnoldgithub opened this issue Oct 15, 2024 · 2 comments

Comments

@larnoldgithub
Copy link
Collaborator

larnoldgithub commented Oct 15, 2024

We moved from the 288 to the 290 for the quicklook processing, as we couldn't fix the issue wit the obj_db.

When I ran apero_database.py --update to rebuild the db from the old folders (tellu and calib) the RAM usage increased
linearly as shown below. The full rebuilding took 33h, with 200,000 files in tellu/.

I got no error messages at the end of the apero_database.py and the new quicklook290 seems to work as expected. I'm catching up with the nights of current run.

Screenshot 2024-10-14 at 09 51 16

@njcuk9999
Copy link
Owner

I mean to me if the RAM goes down without killing python its not a "leak" of memory - its just a lot being kept open (for some reason) that is then closed at some point. Though this is just semantics, I tried with v0.7 to fix this and close more things but it is a complex problem.

I aim in v0.8 to address this and hope that having all this stuff "open" for so long goes away (and will thus fix the problem with using multiprocessing - where this seems to climb way higher).

@larnoldgithub
Copy link
Collaborator Author

At least I agree that as far as the RAM is large enough it shouldn't be an issue.

Also on the plot above I don't know why RAM usage increases by 10Gb after midnight on Sunday night.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants