Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

chore(CI): Build test vectors from source #724

Merged
merged 58 commits into from
Nov 21, 2024
Merged

chore(CI): Build test vectors from source #724

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
ac79bc8
feat: Add MPL and hierarchy keyring example (#634)
lucasmcdonald3 Apr 23, 2024
44d9192
feat: Required encryption context CMM (#645)
lucasmcdonald3 Apr 26, 2024
91d06ae
chore: Merge from master (#670)
lucasmcdonald3 Apr 30, 2024
750bb68
chore: Added KMS keyring example (#659)
RitvikKapila May 2, 2024
88e6d58
chore: added raw rsa and raw aes keyring examples (#661)
RitvikKapila May 2, 2024
428fe95
chore: added keyring examples (kms_rsa, multi, discovery) (#671)
RitvikKapila May 3, 2024
52ad465
chore(examples): Added mrk and mrk_multi keyring examples (#673)
RitvikKapila May 3, 2024
e47e441
chore(examples): added mrk discovery keyring examples (#675)
RitvikKapila May 6, 2024
2a35192
chore(examples): Added file_streaming, migration and set_encryption_a…
RitvikKapila May 7, 2024
986f54c
feat(test_vector_handlers): TestVectors test with MPL constructs (#650)
lucasmcdonald3 May 8, 2024
70d1364
chore(keyring examples): refactor keyring examples and move master ke…
RitvikKapila May 10, 2024
bd32c79
chore: Sync from master (#682)
lucasmcdonald3 May 20, 2024
ebbc26b
chore: performance tests for ESDK-python (#680)
RitvikKapila Jun 5, 2024
39d4498
chore(performance_tests): added hierarchy keyring and caching cmm tes…
RitvikKapila Jun 7, 2024
3476816
chore(migration examples): added KMS, raw AES and raw RSA keyring/MKP…
RitvikKapila Jun 13, 2024
fc8fd38
chore(custom_cmm_example.py): added test for custom_cmm_example.py (#…
RitvikKapila Jun 26, 2024
fceb2b0
chore(MPL): Update README and primary pydocs (#658)
lucasmcdonald3 Jul 30, 2024
ca7e159
chore: Change MPL branch, remove PYTHONPATH workarounds (#683)
lucasmcdonald3 Aug 5, 2024
0de58cd
feat(test_vector_handlers): Hierarchy keyring test vectors (#674)
lucasmcdonald3 Aug 7, 2024
5e813e7
chore(examples): Added raw RSA/AES keyring multithreaded examples (#694)
lucasmcdonald3 Aug 8, 2024
ffe28d9
merge
lucasmcdonald3 Aug 9, 2024
ff29a05
merge
lucasmcdonald3 Aug 9, 2024
1c73a65
test: Required encryption context CMM integration tests (#691)
lucasmcdonald3 Aug 16, 2024
1df7dbc
fix(Examples): Validate EC on decrypt (#697)
lucasmcdonald3 Aug 19, 2024
91aca72
chore: Refer to renamed MPL libraries (#698)
lucasmcdonald3 Sep 6, 2024
0f19a45
Update README.rst
lucasmcdonald3 Oct 2, 2024
219c905
fix(CI): Fix oracle linting (#702)
lucasmcdonald3 Oct 3, 2024
45f386e
fix(test): Fix building ESDK-Net (#703)
lucasmcdonald3 Oct 4, 2024
5b2b116
chore(examples): fix comments (#705)
RitvikKapila Oct 8, 2024
ea82181
chore(MPL): Rename MPL Python package (#706)
lucasmcdonald3 Oct 23, 2024
f37b355
Merge branch 'master' into mpl-reviewed
lucasmcdonald3 Oct 23, 2024
c14f7fe
chore: Point at PyPI MPL (#709)
lucasmcdonald3 Oct 24, 2024
cb56504
chore(release): Bump ESDK version, add changelog (#710)
lucasmcdonald3 Oct 28, 2024
2bdd114
chore(CI): Fix TestVectors timeout (#711)
lucasmcdonald3 Oct 28, 2024
8383b06
Update src/aws_encryption_sdk/internal/formatting/deserialize.py
lucasmcdonald3 Oct 29, 2024
9cddd41
feat(TestVectors): Decrypt "golden" manifest (#721)
lucasmcdonald3 Oct 30, 2024
abfca22
chore(CI): Bump from macOS 12 to 13 (#723)
lucasmcdonald3 Nov 4, 2024
81d6766
m
lucasmcdonald3 Nov 5, 2024
bf0b108
m
lucasmcdonald3 Nov 5, 2024
ae72693
m
lucasmcdonald3 Nov 5, 2024
e9085b7
m
lucasmcdonald3 Nov 5, 2024
2e369a3
no tox
lucasmcdonald3 Nov 5, 2024
567533c
no tox
lucasmcdonald3 Nov 5, 2024
fa28345
no tox
lucasmcdonald3 Nov 5, 2024
bb0e64e
m
lucasmcdonald3 Nov 5, 2024
234b9a6
m
lucasmcdonald3 Nov 5, 2024
3653a2b
m
lucasmcdonald3 Nov 5, 2024
bec81a5
spread
lucasmcdonald3 Nov 5, 2024
06cd362
chore: Bump to MPL 1.7.4
lucasmcdonald3 Nov 11, 2024
a3b85c9
1.7.4
lucasmcdonald3 Nov 11, 2024
dcf2de7
chore: Bump to MPL 1.7.4 (#727)
lucasmcdonald3 Nov 11, 2024
670b97b
Merge branch 'mpl-reviewed' into build-test-vectors
lucasmcdonald3 Nov 11, 2024
c4177fc
m
lucasmcdonald3 Nov 12, 2024
f8876fb
m
lucasmcdonald3 Nov 13, 2024
a6ef4a6
Merge branch 'master' into build-test-vectors
lucasmcdonald3 Nov 14, 2024
4d99d0c
m
lucasmcdonald3 Nov 15, 2024
4aa6a26
Merge branch 'master' into build-test-vectors
lucasmcdonald3 Nov 20, 2024
d8b6517
Update ci_codebuild-tests.yml
lucasmcdonald3 Nov 21, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions codebuild/py311/awses_local_mpl.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -21,6 +21,8 @@ phases:
python: 3.11
build:
commands:
# Build Python MPL TestVector runner from source
- sh test_vector_handlers/scripts/install_mpl_test_vector_runner.sh
- pip install "tox < 4.0"
- cd test_vector_handlers
- tox
2 changes: 2 additions & 0 deletions codebuild/py311/decrypt_dafny_esdk_vectors_keyrings.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ phases:
python: 3.11
pre_build:
commands:
# Build Python MPL TestVector runner from source
- sh test_vector_handlers/scripts/install_mpl_test_vector_runner.sh
# Fetch test vectors from Dafny ESDK's most recent run
# (Assuming the first result is most recent; seems to be correct...)
- |
Expand Down
2 changes: 2 additions & 0 deletions codebuild/py311/decrypt_golden_manifest_with_keyrings.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ phases:
python: 3.11
pre_build:
commands:
# Build Python MPL TestVector runner from source
- sh test_vector_handlers/scripts/install_mpl_test_vector_runner.sh
# Download "golden manifest"
- curl -L -o python-2.3.0.zip https://github.com/awslabs/aws-encryption-sdk-test-vectors/raw/master/vectors/awses-decrypt/python-2.3.0.zip
- unzip python-2.3.0.zip -d python-2.3.0
Expand Down
2 changes: 1 addition & 1 deletion codebuild/py311/decrypt_golden_manifest_with_masterkey.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ version: 0.2

env:
variables:
TOXENV: "py311-full_decrypt-mpl"
TOXENV: "py311-full_decrypt"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This removes the MPL from the TestVector execution environment.
Before, the MPL was installed for TestVector execution in this step. But this is a masterkey step, so the MPL isn't used.

AWS_ENCRYPTION_SDK_PYTHON_INTEGRATION_TEST_AWS_KMS_KEY_ID: >-
arn:aws:kms:us-west-2:658956600833:key/b35311ef1-d8dc-4780-9f5a-55776cbb2f7f
AWS_ENCRYPTION_SDK_PYTHON_INTEGRATION_TEST_AWS_KMS_KEY_ID_2: >-
Expand Down
2 changes: 2 additions & 0 deletions codebuild/py311/decrypt_keyrings_with_keyrings.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ phases:
python: 3.11
pre_build:
commands:
# Build Python MPL TestVector runner from source
- sh test_vector_handlers/scripts/install_mpl_test_vector_runner.sh
# Download previously generated vectors
- aws s3 cp s3://generated-vectors-artifacts-bucket/$CODEBUILD_RESOLVED_SOURCE_VERSION/311_keyrings.zip 311_keyrings.zip
- unzip 311_keyrings.zip
Expand Down
2 changes: 2 additions & 0 deletions codebuild/py311/decrypt_masterkey_with_keyrings.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ phases:
python: 3.11
pre_build:
commands:
# Build Python MPL TestVector runner from source
- sh test_vector_handlers/scripts/install_mpl_test_vector_runner.sh
# Download previously generated vectors
- aws s3 cp s3://generated-vectors-artifacts-bucket/$CODEBUILD_RESOLVED_SOURCE_VERSION/311_masterkey.zip 311_masterkey.zip
- unzip 311_masterkey.zip
Expand Down
2 changes: 2 additions & 0 deletions codebuild/py311/decrypt_net_401_vectors_keyrings.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ phases:
python: 3.11
pre_build:
commands:
# Build Python MPL TestVector runner from source
- sh test_vector_handlers/scripts/install_mpl_test_vector_runner.sh
# Fetch ESDK .NET v4.0.1 Test Vectors
- VECTOR_ZIP=$CODEBUILD_SRC_DIR/v4-Net-4.0.1.zip
- VECTORS_URL=https://github.com/aws/aws-encryption-sdk-dafny/raw/mainline/AwsEncryptionSDK/runtimes/net/TestVectorsNative/TestVectors/resources/v4-Net-4.0.1.zip
Expand Down
2 changes: 2 additions & 0 deletions codebuild/py311/encrypt_keyrings.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ phases:
python: 3.11
build:
commands:
# Build Python MPL TestVector runner from source
- sh test_vector_handlers/scripts/install_mpl_test_vector_runner.sh
- pip install "tox < 4.0"
- cd test_vector_handlers
- |
Expand Down
2 changes: 2 additions & 0 deletions codebuild/py311/generate_decrypt_vectors_keyrings.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ phases:
python: 3.11
build:
commands:
# Build Python MPL TestVector runner from source
- sh test_vector_handlers/scripts/install_mpl_test_vector_runner.sh
- pip install "tox < 4.0"
- cd test_vector_handlers
- |
Expand Down
2 changes: 2 additions & 0 deletions codebuild/py312/awses_local_mpl.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -23,6 +23,8 @@ phases:
python: 3.12
build:
commands:
# Build Python MPL TestVector runner from source
- sh test_vector_handlers/scripts/install_mpl_test_vector_runner.sh
- cd /root/.pyenv/plugins/python-build/../.. && git pull && cd -
- pyenv install --skip-existing 3.12.0
- pyenv local 3.12.0
Expand Down
2 changes: 2 additions & 0 deletions codebuild/py312/decrypt_dafny_esdk_vectors_keyrings.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,8 @@ phases:
python: 3.12
pre_build:
commands:
# Build Python MPL TestVector runner from source
- sh test_vector_handlers/scripts/install_mpl_test_vector_runner.sh
# Fetch test vectors from Dafny ESDK's most recent run
# (Assuming the first result is most recent; seems to be correct...)
- |
Expand Down
2 changes: 2 additions & 0 deletions codebuild/py312/decrypt_golden_manifest_with_keyrings.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ phases:
python: 3.12
pre_build:
commands:
# Build Python MPL TestVector runner from source
- sh test_vector_handlers/scripts/install_mpl_test_vector_runner.sh
# Download "golden manifest"
- curl -L -o python-2.3.0.zip https://github.com/awslabs/aws-encryption-sdk-test-vectors/raw/master/vectors/awses-decrypt/python-2.3.0.zip
- unzip python-2.3.0.zip -d python-2.3.0
Expand Down
2 changes: 1 addition & 1 deletion codebuild/py312/decrypt_golden_manifest_with_masterkey.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ version: 0.2

env:
variables:
TOXENV: "py312-full_decrypt-mpl"
TOXENV: "py312-full_decrypt"
AWS_ENCRYPTION_SDK_PYTHON_INTEGRATION_TEST_AWS_KMS_KEY_ID: >-
arn:aws:kms:us-west-2:658956600833:key/b35311ef1-d8dc-4780-9f5a-55776cbb2f7f
AWS_ENCRYPTION_SDK_PYTHON_INTEGRATION_TEST_AWS_KMS_KEY_ID_2: >-
Expand Down
2 changes: 2 additions & 0 deletions codebuild/py312/decrypt_hkeyring_with_keyrings.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ phases:
python: 3.12
pre_build:
commands:
# Build Python MPL TestVector runner from source
- sh test_vector_handlers/scripts/install_mpl_test_vector_runner.sh
# Download previously generated vectors
# This manifest has coverage for both HKeyring and required encryption context CMM
- aws s3 cp s3://generated-vectors-artifacts-bucket/$CODEBUILD_RESOLVED_SOURCE_VERSION/312_hkeyring_reccmm_manifest.zip 312_hkeyring_reccmm_manifest.zip
Expand Down
2 changes: 1 addition & 1 deletion codebuild/py312/decrypt_hkeyring_with_masterkey.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@ version: 0.2

env:
variables:
TOXENV: "py312-full_decrypt-mpl"
TOXENV: "py312-full_decrypt"
AWS_ENCRYPTION_SDK_PYTHON_INTEGRATION_TEST_AWS_KMS_KEY_ID: >-
arn:aws:kms:us-west-2:658956600833:key/b35311ef1-d8dc-4780-9f5a-55776cbb2f7f
AWS_ENCRYPTION_SDK_PYTHON_INTEGRATION_TEST_AWS_KMS_KEY_ID_2: >-
Expand Down
2 changes: 2 additions & 0 deletions codebuild/py312/decrypt_keyrings_with_keyrings.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ phases:
python: 3.12
pre_build:
commands:
# Build Python MPL TestVector runner from source
- sh test_vector_handlers/scripts/install_mpl_test_vector_runner.sh
# Download previously generated vectors
- aws s3 cp s3://generated-vectors-artifacts-bucket/$CODEBUILD_RESOLVED_SOURCE_VERSION/312_keyrings.zip 312_keyrings.zip
- unzip 312_keyrings.zip
Expand Down
2 changes: 2 additions & 0 deletions codebuild/py312/decrypt_masterkey_with_keyrings.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ phases:
python: 3.12
pre_build:
commands:
# Build Python MPL TestVector runner from source
- sh test_vector_handlers/scripts/install_mpl_test_vector_runner.sh
# Download previously generated vectors
- aws s3 cp s3://generated-vectors-artifacts-bucket/$CODEBUILD_RESOLVED_SOURCE_VERSION/312_masterkey.zip 312_masterkey.zip
- unzip 312_masterkey.zip
Expand Down
2 changes: 2 additions & 0 deletions codebuild/py312/decrypt_net_401_vectors_keyrings.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,8 @@ phases:
python: 3.12
pre_build:
commands:
# Build Python MPL TestVector runner from source
- sh test_vector_handlers/scripts/install_mpl_test_vector_runner.sh
# Fetch ESDK .NET v4.0.1 Test Vectors
- VECTOR_ZIP=$CODEBUILD_SRC_DIR/v4-Net-4.0.1.zip
- VECTORS_URL=https://github.com/aws/aws-encryption-sdk-dafny/raw/mainline/AwsEncryptionSDK/runtimes/net/TestVectorsNative/TestVectors/resources/v4-Net-4.0.1.zip
Expand Down
2 changes: 2 additions & 0 deletions codebuild/py312/encrypt_keyrings.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ phases:
python: 3.12
build:
commands:
# Build Python MPL TestVector runner from source
- sh test_vector_handlers/scripts/install_mpl_test_vector_runner.sh
- pip install "tox < 4.0"
- cd test_vector_handlers
- |
Expand Down
2 changes: 2 additions & 0 deletions codebuild/py312/generate_decrypt_vectors_keyrings.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ phases:
python: 3.12
build:
commands:
# Build Python MPL TestVector runner from source
- sh test_vector_handlers/scripts/install_mpl_test_vector_runner.sh
- pip install "tox < 4.0"
- cd test_vector_handlers
- |
Expand Down
2 changes: 2 additions & 0 deletions codebuild/py312/generate_hkeyring_decrypt_vectors.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,8 @@ phases:
python: 3.12
build:
commands:
# Build Python MPL TestVector runner from source
- sh test_vector_handlers/scripts/install_mpl_test_vector_runner.sh
- pip install "tox < 4.0"
- cd test_vector_handlers/test/aws-crypto-tools-test-vector-framework
# Checkout WIP branch with manifest containing HKeyring and required EC CMM test cases
Expand Down
2 changes: 1 addition & 1 deletion test_vector_handlers/requirements_mpl.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1 +1 @@
aws-cryptography-internal-mpl-testvectors @ git+https://github.com/aws/aws-cryptographic-material-providers-library.git@python-main-with-dafny-code#subdirectory=TestVectorsAwsCryptographicMaterialProviders/runtimes/python
aws-cryptographic-material-providers==1.7.4
24 changes: 24 additions & 0 deletions test_vector_handlers/scripts/install_mpl_test_vector_runner.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
# Builds the Python MPL TestVector runner from source.
# This package is used by the ESDK-Python test vectors for testing with the MPL.

# This script is intended to be used by ESDK-Python's integration tests.
# You may need or want to make local changes to get this work on your machine.

# Change to the directory of the script
cd "$(dirname "$0")"

# Get MPL version specified in requirements.txt
export mplVersion=$(grep 'aws-cryptographic-material-providers==' ../requirements_mpl.txt | sed -E 's/.*==(.+)/\1/')

# Clone MPL repo to get test vectors runner source code and the Dafny version to use
git clone --branch v$mplVersion --recurse-submodules https://github.com/aws/aws-cryptographic-material-providers-library.git

# Download Dafny to build the test vector runner; get Dafny version from ESDK's project.properties file
export dafnyVersion=$(grep '^dafnyVersion=' aws-cryptographic-material-providers-library/project.properties | cut -d '=' -f 2)
curl https://github.com/dafny-lang/dafny/releases/download/v$dafnyVersion/dafny-$dafnyVersion-x64-ubuntu-20.04.zip -L -o dafny.zip
unzip -qq dafny.zip && rm dafny.zip
export PATH="$PWD/dafny:$PATH"

# Build MPL test vector runner from source
cd aws-cryptographic-material-providers-library/TestVectorsAwsCryptographicMaterialProviders/
make transpile_python
3 changes: 3 additions & 0 deletions test_vector_handlers/tox.ini
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -51,6 +51,9 @@ deps =
-rtest/requirements.txt
# Install the MPL requirements if the `-mpl` suffix is present
mpl: -rrequirements_mpl.txt
# This is the filepath used in ESDK-Python's integration tests.
# You may need to change this to run this project locally.
mpl: scripts/aws-cryptographic-material-providers-library/TestVectorsAwsCryptographicMaterialProviders/runtimes/python
..
commands =
awses_local: {[testenv:base-command]commands}
Expand Down
Loading