Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Implement option to use S3-compatible object storage #240

Open
manning-ncsa opened this issue May 29, 2024 · 1 comment
Open

Implement option to use S3-compatible object storage #240

manning-ncsa opened this issue May 29, 2024 · 1 comment
Labels
enhancement New feature or request Full stack Working on this requires the full blast tech stack

Comments

@manning-ncsa
Copy link
Collaborator

The static cutout files downloaded for each transient could be more efficiently and economically stored using S3-compatible object storage, particularly in production deployments. Implement an option to enable object storage instead of filesystem storage. Currently the use of FileField types in the relevant Django models that store data, such as Cutout, should make this an easier task. Combined with the django-storages Django app, it should be possible to rewrite the read/write functions such that they are agnostic to the selected storage backend.

@manning-ncsa manning-ncsa added enhancement New feature or request Full stack Working on this requires the full blast tech stack labels Aug 25, 2024
@manning-ncsa
Copy link
Collaborator Author

Before we dive in the code to implement a bunch of changes, it occurs to me that we should experiment first with s3fs to transparently mount the S3 backend to the local filesystem for Blast. I have tested this method on OSN and it works; the only question is whether the performance is sufficient.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request Full stack Working on this requires the full blast tech stack
Projects
None yet
Development

No branches or pull requests

1 participant