Cloud mirror overview page #2176
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 3 commits
December 22, 2023 16:56
This page provides a list of published projects, and shows whether each project is currently mirrored on one of the supported cloud storage platforms. This page currently just displays a read-only list, and doesn't offer any way to change the state of cloud mirrors (that must be done through the manage_published_project view.) Note that to efficiently display the projects and their associated cloud mirror information, we want to make a single query using select_related(). In particular, we don't want to select all PublishedProjects and then individually check each project's gcp and aws attributes in a Python loop, which would perform 2*N+1 queries. Furthermore, since the list of projects is likely to be large, we use only() to avoid querying unnecessary columns. This should be carefully tested to be sure that the template (cloud_mirrors.html) doesn't directly or indirectly reference unlisted columns.
|
nice, thanks benjamin! |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This adds a page to the console which gives an overview of the active cloud mirrors (Google Cloud Storage and Amazon S3).
Currently this is just a table so that we can easily see which projects are uploaded and which aren't. Since this stuff has to be managed by hand I'd like to have an easy way to see the status of all projects (and grouped by open vs restricted.) Eventually it might be nice to add management functions to this page as well.
In order to view this page, you will need to have the
project.change_publishedprojectpermission (the same permission as for manage_published_project.) I think that there should be a separate permission for managing mirrors, but this will work for now.The implementation is more fragile than I'd like, in a couple of ways.
I want to retrieve the GCP/AWS information in one SQL query, and furthermore want to select only the relevant fields. Using only() and select_related() lets us do that (and you can verify for yourself: this code is only making a single SQL query). But this could easily be broken by code changes elsewhere, and suddenly this page would become much slower to load.
I don't just want to display whether or not file access is allowed, I want to indicate the reason (deprecated, forbidden, embargoed). There's no way to do this via
project.authorization.accessand I can't immediately think of a clean way to do so.That said, I think this code works and should be useful.