Merge pull request #160 from LandRegistry/develop

2.4.0

CONTRIBUTING.md

@@ -10,7 +10,7 @@ Working on your first Pull Request? You can learn how from this *free* series, [
 
 Please use the `develop` branch as starting point for your own branch, and the target for any pull requests.
 
-If you're adding a commodity, at the very minimum it will need a `docker-compose-fragment.yml` and a README entry. If there is support for extra functionality such as provisioning snippets, then a working example should be placed in the snippets directory and linked to from the README.
+If you're adding a commodity, at the very minimum it will need a `compose-fragment.yml` and a README entry. If there is support for extra functionality such as provisioning snippets, then a working example should be placed in the snippets directory and linked to from the README.
 
 ## Code of Conduct
 

README.md

@@ -125,20 +125,22 @@ This file specifies which commodities the dev-env should create and launch for t
 The list of allowable commodity values is:
 
 1. postgres-13
-2. db2_community
+2. postgres-17
+3. db2_community
 4. elasticsearch5
-5. nginx
-6. rabbitmq
-7. redis
-8. swagger
-9. wiremock
-10. squid
-11. auth
-12. cadence
-13. cadence-web
-14. activemq
-15. ibmmq
-16. localstack
+5. elasticsearch7
+6. nginx
+7. rabbitmq
+8. redis
+9. swagger
+10. wiremock
+11. squid
+12. auth
+13. cadence
+14. cadence-web
+15. activemq
+16. ibmmq
+17. localstack
 
 The file may optionally also indicate that one or more services are resource intensive ("expensive") when starting up. The dev env will start those containers seperately - 3 at a time - and wait until each are declared healthy (or crash and get restarted 10 times) before starting any more.
 
@@ -166,7 +168,7 @@ If you want to spatially enable your database see the following example:
 
 [Example - Spatial](snippets/spatial_postgres-init-fragment.sql)
 
-The default Postgres port 5432 will be available for connections from other containers. Port 5434 is exposed for external connections from the host.
+The default Postgres port 5432 will be available for connections from other containers, hostname `postgres-13` or `postgres-17`. Port `5434` (for PG13) or `5435` (for PG17) is exposed for external connections from the host.
 
 **`/manage.py`**
 
@@ -192,15 +194,29 @@ The ports 9300 and 9302 are exposed on the host.
 
 [Example](snippets/elasticsearch5-fragment.sh)
 
+##### ElasticSearch 7
+
+The ports 9207 and 9307 are exposed on the host.
+
+If the ElasticSearch 7 container is returning the follow error log message:
+
+`max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]`
+
+Run the following command in a terminal to set the system's max map count.
+
+```bash
+sysctl -w vm.max_map_count=262144
+```
+
 ##### Nginx
 
 **`/fragments/nginx-fragment.conf`**
 
 This file forms part of an NGINX configration file. It will be merged into the server directive of the main configuration file.
 
-Important - if your app is adding itself as a proxied location{} behind NGINX, NGINX must start AFTER your app, otherwise it will error with a host not found. So your app's docker-compose-fragment.yml must actually specify NGINX as a service and set the depends_on variable with your app's name.
+Important - if your app is adding itself as a proxied location{} behind NGINX, NGINX must start AFTER your app, otherwise it will error with a host not found. So your app's compose-fragment.yml must actually specify NGINX as a service and set the depends_on variable with your app's name.
 
-Compose will automatically merge this with the dev-env's own NGINX fragment. See the end of the [example Compose fragment](snippets/docker-compose-fragment.yml) for the exact code.
+Compose will automatically merge this with the dev-env's own NGINX fragment. See the end of the [example Compose fragment](snippets/compose-fragment.yml) for the exact code.
 
 [Example](snippets/nginx-fragment.conf)
 
@@ -322,8 +338,9 @@ From the host system:
 [Cadence Web](https://github.com/uber/cadence-web) is a web-based user interface which is used to view workflows from Cadence, see what's running, and explore and debug workflow executions. This also comes with a RESTful API that allows us query
 cadence core services.
 
-*Running Cadence web locally*
-- In a web browser enter <http://localhost:5004>
+_Running Cadence web locally_
+
+* In a web browser enter <http://localhost:5004>
 
 ###### Localstack
 

logic.rb

@@ -75,7 +75,7 @@
 
 # Does a version check and self-update if required
 if options['self_update']
-  this_version = '2.3.0'
+  this_version = '2.4.0'
   puts colorize_lightblue("This is a universal dev env (version #{this_version})")
   # Skip version check if not on master (prevents infinite loops if you're in a branch that isn't up to date with the
   # latest release code yet)

scripts/add-aliases.sh

@@ -16,6 +16,7 @@ alias ex="$DC_CMD exec"
 alias status="$DC_CMD ps"
 alias run="$DC_CMD run --rm"
 alias psql13="$DC_CMD exec postgres-13 psql -h postgres-13 -U root -d"
+alias psql17="$DC_CMD exec postgres-17 psql -h postgres-17 -U root -d"
 alias db2co="$DC_CMD exec --user db2inst1 db2_community bash -c '~/sqllib/bin/db2'"
 alias gitlist="bash $DEV_ENV_ROOT_DIR/scripts/git_list.sh"
 alias gitpull="bash $DEV_ENV_ROOT_DIR/scripts/git_pull.sh"
@@ -212,6 +213,7 @@ function devenv-help(){
                                                            if you add -f it will automatically fix issues where possible
                                                            (flags can be combined)
     psql13 <name of database>                        -     run psql in the postgres-13 container
+    psql17 <name of database>                        -     run psql in the postgres-17 container
     db2co                                            -     run db2 command line in the db2_community container
     manage <name of container> <command>             -     run manage.py commands in a container
     alembic <name of container> <command>            -     run an alembic db command in a container, with the appropriate environment variables preset

scripts/commodities.rb

@@ -6,6 +6,7 @@
 require_relative 'provision_db2_community'
 require_relative 'provision_nginx'
 require_relative 'provision_elasticsearch5'
+require_relative 'provision_elasticsearch7'
 require_relative 'provision_wiremock'
 require_relative 'provision_localstack'
 
@@ -133,7 +134,7 @@ def provision_commodities(root_loc, new_containers)
   # If you later modify .commodities to allow this to run again (e.g. if you've added new apps to your group),
   # you'll need to delete the postgres container and it's volume else you'll get errors.
   # Do a fullreset, or docker-compose rm -v -f postgres-13
-  ['13'].each do |postgres_version|
+  %w[13 17].each do |postgres_version|
     provision_postgres(root_loc, new_containers, postgres_version)
     # Alembic, too
     provision_alembic(root_loc, postgres_version)
@@ -145,6 +146,8 @@ def provision_commodities(root_loc, new_containers)
   provision_nginx(root_loc, new_containers)
   # Elasticsearch5
   provision_elasticsearch5(root_loc)
+  # Elasticsearch7
+  provision_elasticsearch7(root_loc)
   # Auth
   provision_auth(root_loc, new_containers)
   # Wiremock mappings

scripts/docker/db2_community/Dockerfile

@@ -1,4 +1,4 @@
-FROM docker.io/hmlandregistry/db2-cgroupaware:11.5.8.0
+FROM docker.io/hmlandregistry/db2-cgroupaware:11.5.9.0
 
 EXPOSE 50000 55000
 

scripts/docker/elasticsearch5/Dockerfile

@@ -1,4 +1,4 @@
-FROM docker.io/elasticsearch:5.2.1
+FROM docker.io/elasticsearch:5.6.16
 
 ENV ES_JAVA_OPTS -Xms1024m -Xmx1024m
 
@@ -7,4 +7,4 @@ RUN echo "bootstrap.memory_lock: true" >> /usr/share/elasticsearch/config/elasti
  echo "indices.fielddata.cache.size: 50%" >> /usr/share/elasticsearch/config/elasticsearch.yml && \
  echo "indices.memory.index_buffer_size: 50%" >> /usr/share/elasticsearch/config/elasticsearch.yml && \
  sed -i -e 's/.*Xms2g.*//' /etc/elasticsearch/jvm.options && \
- sed -i -e 's/.*Xmx2g.*//' /etc/elasticsearch/jvm.options
+ sed -i -e 's/.*Xmx2g.*//' /etc/elasticsearch/jvm.options

scripts/docker/elasticsearch7/Dockerfile

@@ -0,0 +1,9 @@
+FROM docker.elastic.co/elasticsearch/elasticsearch:7.17.24
+
+ENV ES_JAVA_OPTS -Xms1024m -Xmx1024m
+ENV discovery.type single-node
+
+# Remove default heap size and add low-memory optimisations
+RUN echo "bootstrap.memory_lock: true" >> /usr/share/elasticsearch/config/elasticsearch.yml && \
+ echo "indices.fielddata.cache.size: 50%" >> /usr/share/elasticsearch/config/elasticsearch.yml && \
+ echo "indices.memory.index_buffer_size: 50%" >> /usr/share/elasticsearch/config/elasticsearch.yml

scripts/docker/elasticsearch7/compose-fragment.yml

@@ -0,0 +1,18 @@
+services:
+  elasticsearch7:
+    container_name: elasticsearch7
+    build: ../scripts/docker/elasticsearch7
+    ports:
+      - "9207:9200"
+      - "9307:9300"
+    # restart: on-failure
+    platform: "linux/amd64"
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1
+      nofile:
+        soft: 65536
+        hard: 65536
+    cap_add:
+      - IPC_LOCK

scripts/docker/elasticsearch7/docker-compose-fragment.3.7.yml

@@ -0,0 +1,18 @@
+version: '3.7'
+services:
+  elasticsearch7:
+    container_name: elasticsearch7
+    build: ../scripts/docker/elasticsearch7
+    ports:
+      - "9207:9200"
+      - "9307:9300"
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1
+      nofile:
+        soft: 65536
+        hard: 65536
+    cap_add:
+      - IPC_LOCK
+    restart: on-failure

scripts/docker/elasticsearch7/docker-compose-fragment.yml

@@ -0,0 +1,19 @@
+version: '2'
+services:
+  elasticsearch7:
+    container_name: elasticsearch7
+    build: ../scripts/docker/elasticsearch7
+    ports:
+      - "9207:9200"
+      - "9307:9300"
+    ulimits:
+      memlock:
+        soft: -1
+        hard: -1
+      nofile:
+        soft: 65536
+        hard: 65536
+    mem_limit: 2048m
+    cap_add:
+      - IPC_LOCK
+    restart: on-failure

scripts/docker/ibmmq/Dockerfile

@@ -1,4 +1,4 @@
-FROM icr.io/ibm-messaging/mq:9.3.3.0-r2
+FROM icr.io/ibm-messaging/mq:9.3.5.1-r2
 
 # Auto-accept the license
 # Create default users and channels
@@ -7,13 +7,5 @@ ENV LICENSE=accept \
   MQ_ENABLE_METRICS=false \
   MQ_QMGR_NAME=LOCAL_QM
 
-# Add a special run script that stops the web server manually when stop signal received
-# Due to https://github.com/ibm-messaging/mq-container/issues/523
-COPY tini run.sh /tmp/
-USER 0
-RUN chmod +x /tmp/tini && chmod +x /tmp/run.sh
-USER 1001
-ENTRYPOINT ["/tmp/tini", "-g", "--", "/tmp/run.sh"]
-
 # For persistence
 VOLUME /mnt/mqm

scripts/docker/nginx/Dockerfile

@@ -1,13 +1,12 @@
-FROM docker.io/nginx:1.22
+FROM docker.io/nginx:1.26
 
-RUN apt-get update && apt-get install openssl && \
-  rm /etc/nginx/conf.d/default.conf && \
+RUN rm /etc/nginx/conf.d/default.conf && \
   mkdir /etc/nginx/ssl && \
   mkdir /etc/nginx/configs && \
-  openssl req -x509 -nodes -days 365 -newkey rsa:2048 -keyout /etc/nginx/ssl/ssl.key -out /etc/nginx/ssl/ssl.crt -subj "/C=GB/ST=devon/L=plymouth/O=land_registry/OU=seaton_court/CN=testsslkey/[email protected]"
+  openssl req -x509 -noenc -newkey rsa:2048 -keyout /etc/nginx/ssl/key.pem -out /etc/nginx/ssl/req.pem -days 90 -subj "/C=GB/ST=Devon/L=Plymouth/O=HM Land Registry/OU=DDaT/CN=localhost"
 
 # The base server config, which sets up the SSL etc.
-COPY server.conf /etc/nginx/conf.d/server.conf
+COPY nginx.conf /etc/nginx/conf.d
 
 # To ensure our changes and config fragments we copy in are actually persisted in between container recreates.
 # This has to go at the end as the data is frozen once the volume is declared

scripts/docker/nginx/nginx.conf

@@ -0,0 +1,29 @@
+# generated 2024-06-17, Mozilla Guideline v5.7, nginx 1.26.1, OpenSSL 3.0.11, modern configuration, no OCSP
+# https://ssl-config.mozilla.org/#server=nginx&version=1.26.1&config=modern&openssl=3.0.11&ocsp=false&guideline=5.7
+server {
+  listen 80 default_server;
+  listen [::]:80 default_server;
+
+  location / {
+      return 301 https://$host$request_uri;
+  }
+}
+
+server {
+  listen 443 ssl;
+  listen [::]:443 ssl;
+  http2 on;
+
+  ssl_certificate /etc/nginx/ssl/req.pem;
+  ssl_certificate_key /etc/nginx/ssl/key.pem;
+  ssl_session_timeout 1d;
+  ssl_session_cache shared:MozSSL:10m;  # about 40000 sessions
+  ssl_session_tickets off;
+
+  # modern configuration
+  ssl_protocols TLSv1.3;
+  ssl_prefer_server_ciphers off;
+
+  # As part of the provisioning process, files containing apps /location directives will be placed in here
+  include /etc/nginx/configs/*.conf;
+}

scripts/docker/postgres-17/Dockerfile

@@ -0,0 +1,17 @@
+FROM docker.io/postgres:17
+
+# Install the PostGIS extension
+RUN apt-get update && \
+	apt-get install -y postgresql-17-postgis-3 && \
+	apt-get clean && \
+	rm -rf /var/lib/apt/lists/* /tmp/* /var/tmp/*
+
+# This user will be created as the superuser
+# PG* are for dev use - while in the container psql will just work (useful for the provisioning)
+ENV POSTGRES_USER=root \
+ POSTGRES_PASSWORD=superroot \
+ PGUSER=root \
+ PGPASSWORD=superroot
+
+ HEALTHCHECK --interval=10s --retries=20  \
+  CMD pg_isready -h localhost || exit 1

scripts/docker/postgres-17/compose-fragment.yml

@@ -0,0 +1,6 @@
+services:
+  postgres-17:
+    container_name: postgres-17
+    build: ../scripts/docker/postgres-17/
+    ports:
+      - "5435:5432"

scripts/docker/postgres-17/docker-compose-fragment.3.7.yml

@@ -0,0 +1,7 @@
+version: '3.7'
+services:
+  postgres-17:
+    container_name: postgres-17
+    build: ../scripts/docker/postgres-17/
+    ports:
+      - "5435:5432"

scripts/docker/postgres-17/docker-compose-fragment.yml

@@ -0,0 +1,7 @@
+version: '2'
+services:
+  postgres-17:
+    container_name: postgres-17
+    build: ../scripts/docker/postgres-17/
+    ports:
+      - "5435:5432"