Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Optionally stop data deletion after compute #641

Open
sizhky opened this issue Sep 24, 2024 · 0 comments
Open

Optionally stop data deletion after compute #641

sizhky opened this issue Sep 24, 2024 · 0 comments

Comments

@sizhky
Copy link

sizhky commented Sep 24, 2024

In the following locs I'm seeing that the module is deleting the data with del self.data can we have a flag that avoids this deletion so that we can drill down into good/bad data points at a later stage? I can raise a raise a PR if you think this is a good idea.

del self.data

...
            if self.buf_writer is not None:
                self.buf_writer = None
                del self.data
                self.data = None
            else:
                # Release locks and delete all the cache files. Process 0 is released last.
                for filelock, file_path in reversed(list(zip(self.filelocks, self.file_paths))):
                    logger.info(f"Removing {file_path}")
                    del self.data
                    self.data = None
                    del self.writer
                    self.writer = None
                    os.remove(file_path)
                    filelock.release()
...
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant