Feat/sagemaker llms main updates #230
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This gives Airflow teams access to a "_2" secret. This is to work around the limitation that an AWS Secret has a max size of 64KB
feat: allow multiple secrets for Airflow teams
… notebooks S3 bucket add a policy change to allow gitlab runner to put objects into notebooks S3 bucket
…ts from notebooks S3 bucket add a policy change to allow gitlab runner to put objects into notebooks S3 bucket
Added ability for dag processors and tasks to fetch from external buckets
…in a different PR
…mirror-bucket Enable intelligent tiering on mirror bucket for objects > 128KB
…vate-gitlab-package-index policy changes required for private package index for gitlab projects
Bumps [cross-spawn](https://github.com/moxystudio/node-cross-spawn) from 7.0.3 to 7.0.6. - [Changelog](https://github.com/moxystudio/node-cross-spawn/blob/master/CHANGELOG.md) - [Commits](moxystudio/node-cross-spawn@v7.0.3...v7.0.6) --- updated-dependencies: - dependency-name: cross-spawn dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]>
…awn-7.0.6 chore(deps): bump cross-spawn from 7.0.3 to 7.0.6
…ase images Because we only used tagged images in ECS, to reduce costs and to avoid alerts for vulnrabilities that have since been addressed, we should be able to safely delete untagged images. The exception are the various visualisation-base images which we do (for now) use untagged, although this is being changed.
feat: add lifecycle policies to all ECR repos, except visualisation-base images
…ut objects to their space in notebooks bucket for private python index
This adds to the lifecycle rules for preview visualisation (user provided) images. It should now expire preview images one day after they have been pushed. In order to leave production images alone robustly, they now have a "--prod" suffix so they will match the rule with pattern "*--prod" that expires them in 1000 years. While odd, it seems to be the best way to make "*--prod" images _never_ expire.
…ner-to-list-put-objects security policy change for private python index for ddat data science
…urce fixing resource format for gitlab ds runner
…ovided-actions correct gitlab_runner user_provided actions
feat: expire preview visualisation (user provided) images
rename iam role for gitlab data science runner
…r-images-after-one-day expire theia images
…n, pgadmin, remote-desktop, s3sync, and metrics
…r-images-of-all-tools applied to all tools
Bumps [katex](https://github.com/KaTeX/KaTeX) from 0.16.10 to 0.16.21. - [Release notes](https://github.com/KaTeX/KaTeX/releases) - [Changelog](https://github.com/KaTeX/KaTeX/blob/main/CHANGELOG.md) - [Commits](KaTeX/KaTeX@v0.16.10...v0.16.21) --- updated-dependencies: - dependency-name: katex dependency-type: indirect ... Signed-off-by: dependabot[bot] <[email protected]>
…ernally This does introduce a fair bit of duplication that we should probably figure out how to remove, this starts a separate runner for the AG Data Science group that can publish packages to the internal bucket. Co-authored-by: Sekhar Panja <[email protected]> Co-authored-by: Michal Charemza <[email protected]>
…spec feat: separate runner for AG Data Science for publishing packages internally
Instead of having to have files local in the filesystem (which is tricky to store securely), that are then copied to S3, which GitLab pulls from on launch, this makes it so GitLab secrets are stored in Secrets Manager, which GitLab pulls from on launch. This is a part 1 of (probably) 2 parts - this does not remove existing object, permissions or any associated config, to allow environments to keep on accessing the secrets as they were, so we don't have to migrate them all at once. Later parts will likely remove permissions and config. This is part of our move away from having to have any secrets locally on the filesystem.
…ets-manager feat: move GitLab secrets to secrets manager (part 1)
This follows up from #223 by making it possible to apply the terraform with GitLab enabled, but while not have GitLab secrets on the local filesystem.
…ecrets-in-private-tf feat: move GitLab secrets to secrets manager (part 2)
…tlab-aws using admin_environment secrets from aws secrets manager instead of file
The mirror bucket has essentially two modes: - As a "resource", which is used for the live/production environment - As a "data source", which is used for non-prod environments, and refers to the one in the production account (to not have to duplicate the bucket - it's big, and only contains public code) However, before this change the non-prod environment had to be in the same AWS account as the prod one... which is a bit odd in many setups. Now the non-prod environments can be in another AWS account, but still use the mirror bucket from the prod one.
…her-account feat: make it possible for other AWS accounts to use the mirror bucket
…16.21 chore(deps): bump katex from 0.16.10 to 0.16.21
isobel-daley-6point6
requested review from
joehearnshaw-6point6 and
aidanrussell
and removed request for
a team
aidanrussell
approved these changes
Feb 4, 2025
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Merging latest changes to the main branch of the data workspace repo