Migration deploy (#942)

* update codeowners

* Add sample rate to sentry (#981)

* Add sample_rate to sentry setup

* Add new action for govpaas branch

* Bump django from 4.2.13 to 4.2.14 (#982)

Bumps [django](https://github.com/django/django) from 4.2.13 to 4.2.14.
- [Commits](django/django@4.2.13...4.2.14)

---
updated-dependencies:
- dependency-name: django
  dependency-type: direct:production
...

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Add new fields to sentry config (#983)

* Remove migration actions (#984)

* Update Pingdom health check Database, Redis and Celery (#871)

* Change health check end point to /pingdom/ping.xml.
* Extend health check to include Celery and Redis in addition to Database.

* DPM 202 asim formatter logging (#893)

* Use ASIM formatter for logging on production environments
* Upgrade Python version in Docker

* add build config file

* allow for running on gov.uk paas and dbt platform

* allow any python 3.12 version

* allow nodejs buildpack to be included

* set nodejs version to 20.11

* add collect static command to build

* do not use sentry when dsn unset

* add required variables for local build

* DPM 623 Update DB connection environment variable (#915)

* Update database connection environment variable to use DATABASE_CREDENTIALS

* Update image_build for packaging.
Co-Authored-By: Kuds <[email protected]>

* Add extra env vars to image build.

* Adding Env vars
Co-Authored-By: Kuds <[email protected]>

* Update env vars
<[email protected]>

* Adding Env vars

* Added more env vars

* More Env vars

* More Env Vars

* Remove Duplicate Env Var

* Added extra env var

* fix: remove unnecessary start from package.json

* Update image_build_run.sh to include all env vars

* Add setuptools to requirements.txt

* fix: try downgrade of python

* Reset migration for staging deploy

* Apply admin migration before enquiry (#933)

* Revert migration script changes (#941)

* Remove celery check from pingdom (#944)

* Remove celery check from pingdom

* Remove celery pingdom checks

* fix: always call collectstatic
Co-Authored-By: Paul <[email protected]>

* Add saas compilation to image build script

* fix: update celery timezone

* chore: update default timezone

---------

Signed-off-by: dependabot[bot] <[email protected]>
Co-authored-by: Christopher <[email protected]>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Marijn Kampf <[email protected]>
Co-authored-by: Lawrence Goldstien <[email protected]>
Co-authored-by: markwarriner <[email protected]>

.circleci/config.yml

@@ -42,7 +42,7 @@ jobs:
           path: cypress/screenshots
   docs-build:
     docker:
-      - image: python:3.8
+      - image: python:3.12
     steps:
       - checkout
       - dotenv/source:

.copilot/config.yml

@@ -0,0 +1,4 @@
+repository: emt
+builder:
+  name: paketobuildpacks/builder-jammy-full
+  version: 0.3.339

.copilot/image_build_run.sh

@@ -0,0 +1,103 @@
+#!/usr/bin/env bash
+
+# Exit early if something goes wrong
+set -e
+
+# Add commands below to run inside the container after all the other buildpacks have been applied
+
+export DEBUG="True"
+export DJANGO_SECRET_KEY="supersecretkey"
+
+# Database settings
+export DATABASE_CREDENTIALS='{"username": "postgres", "password": "password", "engine": "postgres", "port": 5432, "dbname": "postgres", "host": "db", "dbInstanceIdentifier": "emt-db"}'
+
+# Sentry
+export DJANGO_SENTRY_DSN=""
+
+export ENQUIRIES_PER_PAGE="10"
+export ENQUIRY_RESPONSIVENESS_PERIOD_WEEKS="6"
+
+# DataHub links
+export DATA_HUB_FRONTEND="https://www.datahub.dev.uktrade.io"
+export DATA_HUB_CREATE_COMPANY_PAGE_URL="https://www.datahub.dev.uktrade.io/companies/create"
+
+# DataHub API variables when running Data Hub API locally on port 8000
+export DATA_HUB_METADATA_URL="http://docker.for.mac.localhost:8000/v4/metadsuf"
+export DATA_HUB_COMPANY_SEARCH_URL="http://docker.for.mac.localhost:8000/v4/search/company"
+export DATA_HUB_CONTACT_SEARCH_URL="http://docker.for.mac.localhost:8000/v3/search/contact"
+export DATA_HUB_CONTACT_CREATE_URL="http://docker.for.mac.localhost:8000/v3/contact"
+export DATA_HUB_INVESTMENT_CREATE_URL="http://docker.for.mac.localhost:8000/v3/investment"
+export DATA_HUB_ADVISER_SEARCH_URL="http://docker.for.mac.localhost:8000/adviser/"
+export DATA_HUB_WHOAMI_URL="http://docker.for.mac.localhost:8000/whoami/"
+
+# Hawk
+export DATA_HUB_HAWK_ID="hawk-id"
+export DATA_HUB_HAWK_KEY="hawk-key"
+
+# Celery and Redis
+export ENQUIRY_STATUS_UPDATE_INTERVAL_DAYS="1"
+export ENQUIRY_STATUS_SHOULD_UPDATE="0"
+export REDIS_BASE_URL="redis://redis:6379"
+export CELERY_TIMEZONE="UTC"
+
+# Staff SSO/OAuth2 settings
+export FEATURE_ENFORCE_STAFF_SSO_ENABLED="1"
+
+# Control Consent Service
+export FEATURE_ENFORCE_CONSENT_SERVICE="1"
+
+# The AUTHBROKER_URL needs to be accessible from the browser for the
+# authorization redirect and also from the Docker container for the access token
+# POST request, and the only way to do this is to use the
+# docker.for.mac.localhost host provided by Docker.
+export AUTHBROKER_URL="http://docker.for.mac.localhost:8080"
+export AUTHBROKER_CLIENT_ID="contact-web-ops-for-details"
+export AUTHBROKER_CLIENT_SECRET="contact-web-ops-for-details"
+export AUTHBROKER_TOKEN_SESSION_KEY="_authbroker_token"
+export AUTHBROKER_STAFF_SSO_SCOPE="dummy-scope"
+
+export MOCK_SSO_TOKEN="dummy-token"
+export MOCK_SSO_SCOPE="dummy-scope"
+export MOCK_SSO_USERNAME="testuser"
+export MOCK_SSO_EMAIL_USER_ID="[email protected]"
+export COMPOSE_PROJECT_NAME="enquiry-mgmt-tool"
+
+# Deny http for the OAuth flow
+export OAUTHLIB_INSECURE_TRANSPORT="0"
+
+# Activity Stream
+export ACTIVITY_STREAM_ENQUIRY_POLL_INTERVAL_MINS="1"
+export ACTIVITY_STREAM_KEY_ID="enquiry-mgmt-key-id"
+export ACTIVITY_STREAM_KEY="enquiry-mgmt-secret-key"
+export ACTIVITY_STREAM_SEARCH_URL="https://activity-stream/search"
+export ACTIVITY_STREAM_SEARCH_TARGET_URL="/international/invest/contact/"
+export ACTIVITY_STREAM_ENQUIRY_SEARCH_KEY1="enquiry-key-1"
+export ACTIVITY_STREAM_ENQUIRY_SEARCH_VALUE1="enquiry-value-2"
+export ACTIVITY_STREAM_ENQUIRY_SEARCH_KEY2="enquiry-key-2"
+export ACTIVITY_STREAM_ENQUIRY_SEARCH_VALUE2="enquiry-value-2"
+export ACTIVITY_STREAM_ENQUIRY_DATA_OBJ="enquiry-data-obj"
+
+# Date from which enquiry data is to be fetched in AS
+export ACTIVITY_STREAM_INITIAL_LOAD_DATE="01-January-2020"
+
+# Settings for CSRF and Session cookies
+export CSRF_COOKIE_SECURE="False"
+export CSRF_COOKIE_HTTPONLY="False"
+export SESSION_COOKIE_SECURE="False"
+# SESSION_COOKIE_AGE=36000 # Optional, defaults to 32400 (9 hours)
+
+# Set HSTS headers, only needs to be True in Production
+export SET_HSTS_HEADERS="False"
+
+export NON_RESPONSIVE_ENQUIRY_INITIAL_LOAD_DATE="01-April-2020"
+export ACTIVITY_STREAM_SECOND_QUALIFICATION_SEARCH_NAME="enquiry-data-search-name"
+export ACTIVITY_STREAM_SECOND_QUALIFICATION_SEARCH_VALUE="enquiry-data-search-value"
+
+export ALLOW_TEST_FIXTURE_SETUP="allow"
+
+echo "Running npm i"
+npm i
+echo "Running npm run sass"
+npm run sass
+echo "Running python manage.py collectstatic --noinput --traceback"
+python manage.py collectstatic --noinput

Dockerfile

@@ -1,4 +1,4 @@
-FROM python:3.8
+FROM python:3.12
 
 RUN mkdir /usr/src/app
 WORKDIR /usr/src/app

Procfile

@@ -1,3 +1,3 @@
-web: ./web.sh
+web: bash web.sh
 celery: celery -A app worker -l info
 celerybeat: celery -A app beat -l info

app/pingdom/__init__.py

@@ -0,0 +1 @@
+"""Functionality for monitoring the uptime of the service."""

app/pingdom/services.py

@@ -0,0 +1,54 @@
+from django.conf import settings
+from django.db import DatabaseError
+from redis import Redis
+
+from app.enquiries.celery import app as celery_app
+from app.enquiries.models import Enquiry
+
+
+class CheckDatabase:
+    """Check the database is up and running."""
+
+    name = "database"
+
+    def check(self):
+        """Perform the check."""
+        try:
+            Enquiry.objects.exists()
+            return True, ""
+        except DatabaseError as exception:
+            return False, f"Pingdom check Database: {exception}"
+
+
+class CheckCelery:
+    name = "celery"
+
+    def check(self):
+        try:
+            insp = celery_app.control.inspect()
+            nodes = insp.stats()
+            if not nodes:
+                raise Exception("Celery is not running")
+            return True, ""
+        except Exception as exception:
+            return False, f"Pingdom check Celery: {exception}"
+
+
+class CheckRedis:
+    name = "redis"
+
+    def check(self):
+        try:
+            redis = Redis.from_url(settings.REDIS_BASE_URL)
+            if redis.ping():
+                return True, ""
+            else:
+                return False, "Redis is not connected"
+        except Exception as exception:
+            return False, f"Pingdom check Redis: {exception}"
+
+
+services_to_check = (
+    CheckDatabase,
+    CheckRedis,
+)

app/pingdom/tests/test_services.py

@@ -0,0 +1,75 @@
+from unittest.mock import patch
+
+from django.db import DatabaseError
+from django.test import TestCase
+from redis import RedisError
+
+from app.pingdom.services import CheckCelery, CheckDatabase, CheckRedis
+
+
+class PingdomServicesTestCase(TestCase):
+    def test_check_database_success(self):
+        check_database = CheckDatabase()
+        result = check_database.check()
+        assert check_database.name == "database"
+        assert result == (True, "")
+
+    def test_check_database_failure(self):
+        check_database = CheckDatabase()
+        with patch(
+            'app.enquiries.models.Enquiry.objects.exists',
+            side_effect=DatabaseError('No database'),
+        ):
+            result = check_database.check()
+
+        assert result == (False, "Pingdom check Database: No database")
+
+    def test_check_celery_success(self):
+        check_celery = CheckCelery()
+        with patch(
+            'app.enquiries.celery.app.control.inspect.stats',
+            return_value=[{}]
+        ):
+            result = check_celery.check()
+        assert check_celery.name == "celery"
+        assert result == (True, "")
+
+    def test_check_celery_failure(self):
+        check_celery = CheckCelery()
+        with patch(
+            'app.enquiries.celery.app.control.inspect.stats',
+            return_value=None
+        ):
+            result = check_celery.check()
+
+        assert result == (False, "Pingdom check Celery: Celery is not running")
+
+    def test_check_redis_success(self):
+        check_redis = CheckRedis()
+        with patch(
+            'redis.Redis.ping',
+            return_value=True,
+        ):
+            result = check_redis.check()
+        assert check_redis.name == "redis"
+        assert result == (True, "")
+
+    def test_check_redis_no_ping(self):
+        check_redis = CheckRedis()
+        with patch(
+            'redis.Redis.ping',
+            return_value=False,
+        ):
+            result = check_redis.check()
+        assert check_redis.name == "redis"
+        assert result == (False, "Redis is not connected")
+
+    def test_check_redis_failure(self):
+        check_redis = CheckRedis()
+        with patch(
+            'redis.Redis.ping',
+            side_effect=RedisError("Redis error"),
+        ):
+            result = check_redis.check()
+
+        assert result == (False, "Pingdom check Redis: Redis error")

app/pingdom/tests/test_views.py

@@ -0,0 +1,64 @@
+import pytest
+from unittest.mock import patch
+
+from app.enquiries.tests import test_dh_utils
+from django.db import DatabaseError
+
+from rest_framework import status
+from rest_framework.reverse import reverse
+from redis.exceptions import RedisError
+
+from app.pingdom.services import CheckDatabase, CheckRedis
+from app.pingdom.views import ping
+
+
+pytestmark = pytest.mark.django_db
+
+
+class ServiceHealthCheckPingdomTestCase(test_dh_utils.DataHubUtilsTests):
+    @patch.object(CheckDatabase, 'check')
+    @patch.object(CheckRedis, 'check')
+    def test_ping_success(self, check_database, check_redis):
+        check_database.return_value = (True, "")
+        check_redis.return_value = (True, "")
+
+        response = ping({})
+
+        assert response.status_code == status.HTTP_200_OK
+        assert '<status>OK</status>' in str(response.content)
+        assert response.headers['content-type'] == 'text/xml'
+
+    @patch.object(CheckDatabase, 'check')
+    def test_ping_failure(self, check_database):
+        check_database.return_value = (False, "Error message")
+        response = ping({})
+
+        assert response.status_code == status.HTTP_500_INTERNAL_SERVER_ERROR
+        assert '<status>FALSE</status>' in str(response.content)
+        assert '<!--Error message-->' in str(response.content)
+        assert response.headers['content-type'] == 'text/xml'
+
+    def test_check_database_fail(self):
+        url = reverse('ping')
+
+        with patch(
+            'app.enquiries.models.Enquiry.objects.exists',
+            side_effect=DatabaseError('No database'),
+        ):
+            response = self.client.get(url)
+
+            assert response.status_code == status.HTTP_500_INTERNAL_SERVER_ERROR
+            assert '<status>FALSE</status>' in str(response.content)
+            assert response.headers['content-type'] == 'text/xml'
+
+    def test_check_redis_fail(self):
+        url = reverse('ping')
+        with patch(
+            'redis.Redis.ping',
+            side_effect=RedisError("Redis error"),
+        ):
+            response = self.client.get(url)
+
+            assert response.status_code == status.HTTP_500_INTERNAL_SERVER_ERROR
+            assert '<status>FALSE</status>' in str(response.content)
+            assert response.headers['content-type'] == 'text/xml'

app/pingdom/urls.py

@@ -0,0 +1,8 @@
+from django.urls import path
+
+from .views import pingdom
+
+
+urlpatterns = [
+    path("ping.xml", pingdom, name="pingdom"),
+]