Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Configurable s3 endpoint for download #1163

Merged
merged 31 commits into from
Jan 9, 2025

Conversation

aaperis
Copy link
Contributor

@aaperis aaperis commented Dec 4, 2024

Related issue(s) and PR(s)
This PR closes #750.

Description
With this PR download can either be deployed as serving either unencrypted or encrypted files. The serveUnencryptedData boolean is removed as well as the logic for download to always generate an internal c4gh key-pair (that would allow the service to serve unencrypted files in case serveUnencryptedData=true).

Main changes:

  • download can now be configured to only serve unencrypted files if a filepath to a c4gh private key file is provided. Internally, the code checks the validity of the provided key by using the supplied passphrase to retrieve the corresponding c4gh public key. The service will not start if this procedure fails and the service will serve unencrypted data only if the retrieved public key is non-empty. This ensures that there can be no misconfiguration by e.g. providing malfunctioning keys etc. For example, the crypt4gh library and therefore reencrypt will use any string provided to it and encrypt a file but with this mechanism we avoid such a scary scenario.
  • if no filepath and passphrase are provided, then download will serve only encrypted files. This is the default behavior.

Other changes include:

  • go testsuite updates and some refactoring along the way
  • update of dev environment: in order to be able to test both setups at once, the docker compose files have been modified to include two instances of download listening at different ports, one serving encrypted and the other serving unencrypted files
  • update of integration tests, including a separate bash script for error messages returned to the user (according to the issue description)
  • update of helm chart

How to test
Integration tests pass.

@aaperis aaperis force-pushed the feature/unified-configurable-s3-endpoint-for-download branch from d5fc9f1 to d9c8803 Compare December 4, 2024 20:51
@aaperis aaperis changed the title Feature/unified configurable s3 endpoint for download Configurable s3 endpoint for download Dec 5, 2024
@aaperis aaperis force-pushed the feature/unified-configurable-s3-endpoint-for-download branch from d9c8803 to 73baacd Compare December 5, 2024 14:35
@aaperis aaperis self-assigned this Dec 9, 2024
@aaperis aaperis force-pushed the feature/unified-configurable-s3-endpoint-for-download branch 4 times, most recently from 9f68afa to d4e2439 Compare December 12, 2024 23:52
@aaperis aaperis marked this pull request as ready for review December 12, 2024 23:59
@aaperis aaperis requested a review from a team December 12, 2024 23:59
@aaperis aaperis force-pushed the feature/unified-configurable-s3-endpoint-for-download branch 2 times, most recently from 32a1128 to 058887a Compare December 13, 2024 08:10
charts/sda-svc/README.md Outdated Show resolved Hide resolved
charts/sda-svc/templates/download-deploy.yaml Outdated Show resolved Hide resolved
charts/sda-svc/templates/download-deploy.yaml Outdated Show resolved Hide resolved
charts/sda-svc/values.yaml Outdated Show resolved Hide resolved
charts/sda-svc/templates/download-deploy.yaml Outdated Show resolved Hide resolved
sda-download/internal/config/config_test.go Outdated Show resolved Hide resolved
sda-download/internal/config/config_test.go Outdated Show resolved Hide resolved
sda-download/internal/config/config_test.go Outdated Show resolved Hide resolved
sda-download/dev_utils/compose-no-tls.yml Outdated Show resolved Hide resolved
sda-download/dev_utils/config.yaml Show resolved Hide resolved
@aaperis aaperis force-pushed the feature/unified-configurable-s3-endpoint-for-download branch 2 times, most recently from 402b10a to 8a9b000 Compare December 15, 2024 11:47
@aaperis aaperis requested review from jbygdell and a team December 15, 2024 11:58
Copy link
Collaborator

@jbygdell jbygdell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Crypt4gh config already exists, there is not need to add another one since the app will only need one.

Either reuse the existing one or move everything up under the app sub heading.

sda-download/dev_utils/config.yaml Outdated Show resolved Hide resolved
sda-download/dev_utils/run_integration_test.sh Outdated Show resolved Hide resolved
sda-download/internal/config/config.go Outdated Show resolved Hide resolved
sda-download/internal/config/config.go Outdated Show resolved Hide resolved
sda-download/internal/config/config_test.go Outdated Show resolved Hide resolved
sda-download/internal/config/config_test.go Outdated Show resolved Hide resolved
sda-download/internal/config/config_test.go Outdated Show resolved Hide resolved
@aaperis
Copy link
Contributor Author

aaperis commented Dec 16, 2024

I unresolved the initial the conversations that I set as resolved (these are answered) so that @jbygdell can take a look and resolve. According to how we decided to do things on our last meeting about PRs.

Copy link
Contributor

@MalinAhlberg MalinAhlberg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks like good and thorough work..!
I left some questions, but otherwise I think it looks good!

sda-download/api/api.md Outdated Show resolved Hide resolved
sda-download/api/sda/sda.go Show resolved Hide resolved
@aaperis aaperis force-pushed the feature/unified-configurable-s3-endpoint-for-download branch from 107a01a to a4a5e96 Compare January 9, 2025 10:40
@aaperis aaperis added this pull request to the merge queue Jan 9, 2025
Merged via the queue into main with commit 4e9723c Jan 9, 2025
30 checks passed
@aaperis aaperis deleted the feature/unified-configurable-s3-endpoint-for-download branch January 9, 2025 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[download] Configurable option for whether to support decrypted xor encrypted downloads.
4 participants