Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

TSS-2214-trs-dbt-migration-setup-environment-variables-to-be-compatible-with-dbt-platform #424

Merged
merged 4 commits into from
Nov 15, 2024
Merged

TSS-2214-trs-dbt-migration-setup-environment-variables-to-be-compatible-with-dbt-platform #424

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
7e0db9b
added env file for env variables
osimuka Nov 12, 2024
c323796
added env file for env variables
osimuka Nov 12, 2024
3dbeb1a
lint fix
osimuka Nov 12, 2024
0ad0888
lint fix
osimuka Nov 12, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 3 additions & 3 deletions .circleci/config.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,7 +12,7 @@ commands:
. venv/bin/activate #
pip install --upgrade pip
pip install poetry==1.6.1
poetry export --dev --without-hashes -f requirements.txt -o requirements-dev.txt
poetry export --with dev --without-hashes -f requirements.txt -o requirements-dev.txt
pip install -r requirements-dev.txt

restore_dependency_cache:
Expand All @@ -33,7 +33,7 @@ jobs:

code_quality:
docker:
- image: cimg/python:3.9.5
- image: cimg/python:3.9.20
steps:
- checkout
- restore_dependency_cache
Expand All @@ -49,7 +49,7 @@ jobs:
pflake8 trade_remedies_public --config pyproject.toml
test_and_fitness_functions:
docker:
- image: cimg/python:3.9.5
- image: cimg/python:3.9.20
environment:
DATABASE_URL: sqlite:///db.sqlite3
DJANGO_SETTINGS_MODULE: config.settings.test
Expand Down
2 changes: 2 additions & 0 deletions Makefile
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,5 +18,7 @@ help:
@echo -e "$(COLOUR_YELLOW)make prod-requirements$(COLOUR_NONE) : Generate prod requirements (requires local pip-compile)"

all-requirements:
poetry lock
poetry export --without-hashes -f requirements.txt -o requirements.txt
poetry export --with dev --without-hashes -f requirements.txt -o requirements-dev.txt
pip install -r requirements-dev.txt
1,458 changes: 937 additions & 521 deletions poetry.lock
View file
Open in desktop

Large diffs are not rendered by default.

4 changes: 3 additions & 1 deletion pyproject.toml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ requires = ["poetry-core^1.0.0"]
build-backend = "poetry.core.masonry.api"

[tool.poetry.dependencies]
python = "^3.8"
python = "^3.9.20"
boto3 = "1.29.6"
Django = "4.2.16"
certifi = "^2024.07.04"
Expand Down Expand Up @@ -47,6 +47,8 @@ v2-api-client = {git = "https://github.com/uktrade/trs_v2_api_client.git", rev =
gunicorn = "22.0.0"
gevent = "23.9.1"
lxml = "4.9.1"
dbt-copilot-python = "0.1.4"
pydantic-settings = "2.5.2"

[tool.poetry.dev-dependencies]
behave-django = "1.4.0"
Expand Down
175 changes: 100 additions & 75 deletions requirements.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,75 +1,100 @@
api-client==1.3.1 ; python_version >= "3.8" and python_version < "4.0"
asgiref==3.8.1 ; python_version >= "3.8" and python_version < "4.0"
async-timeout==4.0.3 ; python_version >= "3.8" and python_version < "4.0"
backports-zoneinfo==0.2.1 ; python_version >= "3.8" and python_version < "3.9"
boto3==1.29.6 ; python_version >= "3.8" and python_version < "4.0"
botocore==1.32.7 ; python_version >= "3.8" and python_version < "4.0"
certifi==2024.8.30 ; python_version >= "3.8" and python_version < "4.0"
cffi==1.17.1 ; platform_python_implementation == "CPython" and sys_platform == "win32" and python_version >= "3.8" and python_version < "4.0"
charset-normalizer==3.4.0 ; python_version >= "3.8" and python_version < "4.0"
deprecated==1.2.14 ; python_version >= "3.8" and python_version < "4.0"
django-appconf==1.0.6 ; python_version >= "3.8" and python_version < "4.0"
django-audit-log-middleware==0.0.4 ; python_version >= "3.8" and python_version < "4.0"
django-basicauth==0.5.2 ; python_version >= "3.8" and python_version < "4.0"
django-cache-memoize==0.2.0 ; python_version >= "3.8" and python_version < "4.0"
django-chunk-upload-handlers==0.0.11 ; python_version >= "3.8" and python_version < "4.0"
django-compressor==4.5.1 ; python_version >= "3.8" and python_version < "4.0"
django-countries==7.2.1 ; python_version >= "3.8" and python_version < "4.0"
django-csp==3.7 ; python_version >= "3.8" and python_version < "4.0"
django-environ==0.11.2 ; python_version >= "3.8" and python_version < "4"
django-extensions==3.2.3 ; python_version >= "3.8" and python_version < "4.0"
django-govuk-forms==0.5 ; python_version >= "3.8" and python_version < "4.0"
django-govuk-template==0.6 ; python_version >= "3.8" and python_version < "4.0"
django-ipware==3.0.7 ; python_version >= "3.8" and python_version < "4.0"
django-log-formatter-ecs==0.0.5 ; python_version >= "3.8" and python_version < "4.0"
django-redis==5.2.0 ; python_version >= "3.8" and python_version < "4.0"
django-sass-processor==1.4.1 ; python_version >= "3.8" and python_version < "4.0"
django-storages==1.11.1 ; python_version >= "3.8" and python_version < "4.0"
django-timezone-field==4.2.3 ; python_version >= "3.8" and python_version < "4.0"
django==4.2.16 ; python_version >= "3.8" and python_version < "4.0"
djangorestframework==3.15.2 ; python_version >= "3.8" and python_version < "4.0"
dotwiz==0.4.0 ; python_version >= "3.8" and python_version < "4.0"
dpath==2.1.6 ; python_version >= "3.8" and python_version < "4.0"
et-xmlfile==2.0.0 ; python_version >= "3.8" and python_version < "4.0"
feed-gov-back==0.4 ; python_version >= "3.8" and python_version < "4.0"
gevent==23.9.1 ; python_version >= "3.8" and python_version < "4.0"
greenlet==3.1.1 ; platform_python_implementation == "CPython" and python_version < "4.0" and python_version >= "3.8"
gunicorn==22.0.0 ; python_version >= "3.8" and python_version < "4.0"
idna==3.10 ; python_version >= "3.8" and python_version < "4.0"
jmespath==1.0.1 ; python_version >= "3.8" and python_version < "4.0"
kubi-ecs-logger==0.1.2 ; python_version >= "3.8" and python_version < "4"
libsass==0.23.0 ; python_version >= "3.8" and python_version < "4.0"
lxml==4.9.1 ; python_version >= "3.8" and python_version < "4.0"
markupsafe==2.1.5 ; python_version >= "3.8" and python_version < "4.0"
marshmallow==3.19.0 ; python_version >= "3.8" and python_version < "4"
openpyxl==3.1.5 ; python_version >= "3.8" and python_version < "4.0"
packaging==24.1 ; python_version >= "3.8" and python_version < "4.0"
phonenumbers==8.13.48 ; python_version >= "3.8" and python_version < "4.0"
pikepdf==9.2.1 ; python_version >= "3.8" and python_version < "4.0"
pillow==10.4.0 ; python_version >= "3.8" and python_version < "4.0"
psycopg2-binary==2.9.10 ; python_version >= "3.8" and python_version < "4.0"
py==1.11.0 ; python_version >= "3.8" and python_version < "4.0"
pycparser==2.22 ; platform_python_implementation == "CPython" and sys_platform == "win32" and python_version >= "3.8" and python_version < "4.0"
pyheck==0.1.5 ; python_version >= "3.8" and python_version < "4.0"
python-dateutil==2.9.0.post0 ; python_version >= "3.8" and python_version < "4.0"
pytz==2024.2 ; python_version >= "3.8" and python_version < "4.0"
rcssmin==1.1.2 ; python_version >= "3.8" and python_version < "4.0"
redis==4.4.4 ; python_version >= "3.8" and python_version < "4.0"
requests==2.32.2 ; python_version >= "3.8" and python_version < "4.0"
rjsmin==1.2.2 ; python_version >= "3.8" and python_version < "4.0"
s3transfer==0.7.0 ; python_version >= "3.8" and python_version < "4.0"
sentry-sdk==2.8.0 ; python_version >= "3.8" and python_version < "4.0"
setuptools==75.2.0 ; python_version >= "3.8" and python_version < "4.0"
six==1.16.0 ; python_version >= "3.8" and python_version < "4.0"
sqlparse==0.5.0 ; python_version >= "3.8" and python_version < "4.0"
tenacity==9.0.0 ; python_version >= "3.8" and python_version < "4.0"
trade-remedies-client @ git+https://github.com/uktrade/trade-remedies-client.git@b2853f6aade7aed082fa7abc435d470427cc2fcd ; python_version >= "3.8" and python_version < "4.0"
typing-extensions==4.12.2 ; python_version >= "3.8" and python_version < "3.11"
tzdata==2024.2 ; python_version >= "3.8" and python_version < "4.0" and sys_platform == "win32"
urllib3==1.26.19 ; python_version >= "3.8" and python_version < "4.0"
v2-api-client @ git+https://github.com/uktrade/trs_v2_api_client.git@f8a766216db0d5b50ac5e433ccf243fcf30fca24 ; python_version >= "3.8" and python_version < "4.0"
werkzeug==3.0.6 ; python_version >= "3.8" and python_version < "4.0"
whitenoise==6.6.0 ; python_version >= "3.8" and python_version < "4.0"
wrapt==1.16.0 ; python_version >= "3.8" and python_version < "4.0"
zope-event==5.0 ; python_version >= "3.8" and python_version < "4.0"
zope-interface==7.1.1 ; python_version >= "3.8" and python_version < "4.0"
annotated-types==0.7.0 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
api-client==1.3.1 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
asgiref==3.8.1 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
async-timeout==5.0.1 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
backoff==2.2.1 ; python_full_version >= "3.9.20" and python_version < "4.0"
boto3==1.29.6 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
botocore==1.32.7 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
certifi==2024.8.30 ; python_full_version >= "3.9.20" and python_version < "4.0"
cffi==1.17.1 ; platform_python_implementation == "CPython" and sys_platform == "win32" and python_full_version >= "3.9.20" and python_full_version < "4.0.0"
charset-normalizer==3.4.0 ; python_full_version >= "3.9.20" and python_version < "4.0"
dbt-copilot-python==0.1.4 ; python_full_version >= "3.9.20" and python_version < "4.0"
deprecated==1.2.14 ; python_full_version >= "3.9.20" and python_version < "4.0"
django-appconf==1.0.6 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
django-audit-log-middleware==0.0.4 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
django-basicauth==0.5.2 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
django-cache-memoize==0.2.0 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
django-chunk-upload-handlers==0.0.11 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
django-compressor==4.5.1 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
django-countries==7.2.1 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
django-csp==3.7 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
django-environ==0.11.2 ; python_full_version >= "3.9.20" and python_version < "4"
django-extensions==3.2.3 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
django-govuk-forms==0.5 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
django-govuk-template==0.6 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
django-ipware==3.0.7 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
django-log-formatter-ecs==0.0.5 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
django-redis==5.2.0 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
django-sass-processor==1.4.1 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
django-storages==1.11.1 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
django-timezone-field==4.2.3 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
django==4.2.16 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
djangorestframework==3.15.2 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
dotwiz==0.4.0 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
dpath==2.1.6 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
et-xmlfile==2.0.0 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
feed-gov-back==0.4 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
gevent==23.9.1 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
googleapis-common-protos==1.65.0 ; python_full_version >= "3.9.20" and python_version < "4.0"
greenlet==3.1.1 ; platform_python_implementation == "CPython" and python_full_version < "4.0.0" and python_full_version >= "3.9.20"
grpcio==1.67.1 ; python_full_version >= "3.9.20" and python_version < "4.0"
gunicorn==22.0.0 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
idna==3.10 ; python_full_version >= "3.9.20" and python_version < "4.0"
importlib-metadata==6.11.0 ; python_full_version >= "3.9.20" and python_version < "4.0"
jmespath==1.0.1 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
kubi-ecs-logger==0.1.2 ; python_full_version >= "3.9.20" and python_version < "4"
libsass==0.23.0 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
lxml==4.9.1 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
markupsafe==3.0.2 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
marshmallow==3.19.0 ; python_full_version >= "3.9.20" and python_version < "4"
openpyxl==3.1.5 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
opentelemetry-api==1.22.0 ; python_full_version >= "3.9.20" and python_version < "4.0"
opentelemetry-distro==0.43b0 ; python_full_version >= "3.9.20" and python_version < "4.0"
opentelemetry-exporter-otlp-proto-common==1.22.0 ; python_full_version >= "3.9.20" and python_version < "4.0"
opentelemetry-exporter-otlp-proto-grpc==1.22.0 ; python_full_version >= "3.9.20" and python_version < "4.0"
opentelemetry-exporter-otlp-proto-http==1.22.0 ; python_full_version >= "3.9.20" and python_version < "4.0"
opentelemetry-exporter-otlp==1.22.0 ; python_full_version >= "3.9.20" and python_version < "4.0"
opentelemetry-instrumentation-wsgi==0.43b0 ; python_full_version >= "3.9.20" and python_version < "4.0"
opentelemetry-instrumentation==0.43b0 ; python_full_version >= "3.9.20" and python_version < "4.0"
opentelemetry-propagator-aws-xray==1.0.2 ; python_full_version >= "3.9.20" and python_version < "4.0"
opentelemetry-proto==1.22.0 ; python_full_version >= "3.9.20" and python_version < "4.0"
opentelemetry-sdk-extension-aws==2.0.2 ; python_full_version >= "3.9.20" and python_version < "4.0"
opentelemetry-sdk==1.22.0 ; python_full_version >= "3.9.20" and python_version < "4.0"
opentelemetry-semantic-conventions==0.43b0 ; python_full_version >= "3.9.20" and python_version < "4.0"
opentelemetry-util-http==0.43b0 ; python_full_version >= "3.9.20" and python_version < "4.0"
packaging==24.2 ; python_full_version >= "3.9.20" and python_version < "4"
phonenumbers==8.13.49 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
pikepdf==9.4.0 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
pillow==10.4.0 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
protobuf==4.25.5 ; python_full_version >= "3.9.20" and python_version < "4.0"
psycopg2-binary==2.9.10 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
py==1.11.0 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
pycparser==2.22 ; platform_python_implementation == "CPython" and sys_platform == "win32" and python_full_version >= "3.9.20" and python_full_version < "4.0.0"
pydantic-core==2.23.4 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
pydantic-settings==2.5.2 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
pydantic==2.9.2 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
pyheck==0.1.5 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
python-dateutil==2.9.0.post0 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
python-dotenv==1.0.1 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
pytz==2024.2 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
rcssmin==1.1.2 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
redis==4.4.4 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
requests==2.32.2 ; python_full_version >= "3.9.20" and python_version < "4.0"
rjsmin==1.2.2 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
s3transfer==0.7.0 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
sentry-sdk==2.8.0 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
setuptools==75.4.0 ; python_full_version >= "3.9.20" and python_version < "4.0"
six==1.16.0 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
sqlparse==0.5.0 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
tenacity==9.0.0 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
trade-remedies-client @ git+https://github.com/uktrade/trade-remedies-client.git@b2853f6aade7aed082fa7abc435d470427cc2fcd ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
typing-extensions==4.12.2 ; python_full_version >= "3.9.20" and python_version < "4.0"
tzdata==2024.2 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0" and sys_platform == "win32"
urllib3==1.26.19 ; python_full_version >= "3.9.20" and python_version < "4.0"
v2-api-client @ git+https://github.com/uktrade/trs_v2_api_client.git@f8a766216db0d5b50ac5e433ccf243fcf30fca24 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
werkzeug==3.1.3 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
whitenoise==6.6.0 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
wrapt==1.16.0 ; python_full_version >= "3.9.20" and python_version < "4.0"
zipp==3.21.0 ; python_full_version >= "3.9.20" and python_version < "4.0"
zope-event==5.0 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
zope-interface==7.1.1 ; python_full_version >= "3.9.20" and python_full_version < "4.0.0"
64 changes: 64 additions & 0 deletions trade_remedies_public/config/cf_env.py
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,64 @@
from typing import Optional, Any
from pydantic import BaseModel, ConfigDict, Field
from pydantic_settings import BaseSettings


class VCAPServices(BaseModel):
model_config = ConfigDict(extra="ignore")

postgres: list[dict[str, Any]] = Field(alias="postgres", default=[])
redis: list[dict[str, Any]] = Field(alias="redis", default=[])
aws_s3_bucket: list[dict[str, Any]] = Field(alias="aws-s3-bucket", default=[])
opensearch: list[dict[str, Any]] = Field(alias="opensearch", default=[])


class CloudFoundrySettings(BaseSettings):
SENTRY_DSN: str = ""
SENTRY_ENVIRONMENT: str = "local"
DJANGO_SECRET_KEY: str = ""
DEBUG: bool = True
ALLOWED_HOSTS: str = ""
ORGANISATION_NAME: str = "Organisation name placeholder"
ORGANISATION_INITIALISM: str = "Organisation initialism placeholder"
BASIC_AUTH_USER: bool = False
VCAP_SERVICES: Optional[VCAPServices] = {}
REDIS_DATABASE_NUMBER: int = 2
REDIS_BASE_URL: str = "redis://redis:6379"
SECURE_COOKIE: bool = False
SESSION_LENGTH_MINUTES: int = 30
SECURE_CSRF_COOKIE: bool = False
CSRF_COOKIE_HTTPONLY: bool = False
USE_2FA: bool = True
VERIFY_EMAIL: bool = True
GA_TAG_MANAGER_ID: str = "GTM-XXXXXX"
API_BASE_URL: str = "http://localhost:8000"
HEALTH_CHECK_TOKEN: str = "health-check-token"
ENVIRONMENT_KEY: str = "PUB-ENV"
S3_STORAGE_KEY: Optional[str] = None
S3_STORAGE_SECRET: Optional[str] = None
S3_BUCKET_NAME: Optional[str] = None
AWS_STORAGE_BUCKET_NAME: Optional[str] = None
AWS_REGION: str = "eu-west-1"
CLAM_AV_USERNAME: Optional[str] = None
CLAM_AV_PASSWORD: Optional[str] = None
CLAM_AV_DOMAIN: Optional[str] = None
USE_CLAM_AV: bool = True
ROOT_LOG_LEVEL: str = "INFO"
DJANGO_LOG_LEVEL: str = "INFO"
DJANGO_SERVER_LOG_LEVEL: str = "INFO"
DJANGO_REQUEST_LOG_LEVEL: str = "INFO"
DEFAULT_CHUNK_SIZE: int = 33554432
FILE_MAX_SIZE_BYTES: int = 31457280

def get_allowed_hosts(self) -> list[str]:
return self.ALLOWED_HOSTS.split(",") if self.ALLOWED_HOSTS else ["localhost"]

def get_s3_bucket_config(self) -> dict:
"""Return s3 bucket config that matches keys used in CF"""

return {
"aws_region": self.AWS_REGION,
"bucket_name": self.S3_BUCKET_NAME or self.AWS_STORAGE_BUCKET_NAME or "",
"storage_secret": self.S3_STORAGE_SECRET,
"storage_key": self.S3_STORAGE_KEY,
}
Loading