Updating based on PR comments

alembic-autogenerate.py

@@ -2,15 +2,14 @@
 
 #
 # How to use this script:
-# 1. Load the LSST environment and setup sdm_schemas and felis.
-#        source loadLSST.bash or activate your eups setup
-#        setup felis
-#        setup -r /path/to/sdm_schemas -- SDM_SCHEMAS_DIR should work
+# 1. Install Felis and sdm_schemas, set environment variables:
+#        pip install lsst-felis testing.postgresql alembic sqlalchemy pyyaml
+#        git clone https://github.com/lsst/sdm-schemas
+#        cd sdm_schemas
+#        export SDM_SCHEMAS_DIR=`pwd```
 # 2. From the root of the consdb git repo, invoke the script. Supply a
 #    revision message as the command line argument:
-#        python alembic-autogenerate.py this is my revision message "\n" \
-#            the message can span multiple lines "\n" \
-#            if desired
+#        python alembic-autogenerate.py DM-12345
 # 3. Heed the message at the end to revise your auto-generated code as needed.
 # 4. Remove the autogenerated creation of sql views (visit1, ccdvisit1).
 #
@@ -44,7 +43,6 @@
 # Loop over each of the instruments
 pattern = os.environ["SDM_SCHEMAS_DIR"] + "/yml/cdb_*.yaml"
 instruments = [re.search(r"cdb_(.+)\.yaml$", file).group(1) for file in glob.glob(pattern)]
-instruments = []  # you can just hard code the few we want
 for instrument in instruments:
     # Set up a temporary PostgreSQL instance using testing.postgresql
     with setup_postgres_test_db() as instance:

doc/.gitignore

@@ -8,3 +8,5 @@ doxygen.conf
 # Sphinx products
 _build
 py-api
+
+*.DS_Store

doc/operator-guide/deployment.rst

@@ -2,37 +2,43 @@
 Deployment
 ###########
 
-* Database
+Database
+========
 
     Deployments of the Consolidated Database are currently located at
 
       -  Summit
-      -  USDF (+ dev)
+      -  USDF (+ dev, use the same underlying database, a replication of Summit)
       -  Base Test Stand (BTS)
       -  Tucson Test Stand (TTS)
 
     Updates to these deployments may be needed when there are edits to the schema for any of the cdb_* tables defined in <link to> sdm_schemas.
 
     Tools:
+    ------
 
       - Argo-CD
       - LOVE
       - felis
 
     Repositories:
+    -------------
 
         - Phalanx
         - sdm_schemas
         - consdb
 
     Access needed:
+    --------------
 
         - NOIRLab VPN
         - Summit VPN
-        - USDF??
+        - USDF
 
     Process:
+    --------
 
 
 
-* REST API Server
+REST API Server
+===============

doc/operator-guide/monitoring.rst

@@ -2,24 +2,28 @@
 Monitoring
 ###########
 
-* Reporting channels
+Reporting channels
+==================
 
   - Users of ConsDB, ConsDBClient (``pqserver``) will usually report via #consolidated-database in rubin-obs.slack.com when they are having issues.
   - ConsDB operators should monitor this channel and #ops-usdf, #ops-usdf-alerts for issues and outages reported, as well as escalate verified database issues.
 
-* Database
+Database
+========
 
   - The ConsDB team is responsible for verifying whether or not the database is up when issues are reported
     - They can check the method reported by the users, check using ``psql``/ ``pgcli``, and check in the #ops-usdf slack channel for currently reported issues.
 
   - Once the ConsDB team has confirmed there is an issue with the database, they should notify #ops-usdf slack channel and USDF DBAs should be responsible for fixing/restarting.
 
-* REST API Server
+REST API Server
+===============
 
-- If we suspect the API server died, the ConsDB team should be responsible for checking and restarting
+  - If we suspect the API server died, the ConsDB team should be responsible for checking and restarting
   - Use the appropriate argo-cd deployment graph to check deployment logs, and potentially restart the service.
 
 
-- Other issues
+  Other issues
+  ------------
   - K8s infrastructure died The ConsDB team can verify that that is the problem, but there are likely to be wider issues seen
   - USDF or Summit K8s/IT support should be responsible for fixing.

doc/operator-guide/schema-migration-process.rst

@@ -2,21 +2,16 @@
 Schema Migration Process
 ########################
 
-* How to add columns to sdm_schemas
+Add columns to sdm_schemas
+==========================
 
-  - Edit the repository at https://sdm-schemas.lsst.io/ to apply your schema changes to any of the ``cdb_*.yml`` schemas.
-  - Check that valid SQL tables can be created with ``felis`` (https://felis.lsst.io/user-guide/cli.html#felis-validate) using:
+  - Add the requested database additions, justifications, and where they are generated to (https://rubinobs.atlassian.net/wiki/spaces/DM/pages/246644760/Consolidated+Database+Non-EFD+Entries)
+  - Create a ticket and edit the repository at https://sdm-schemas.lsst.io/ to apply your schema changes to any of the ``cdb_*.yml`` schemas.
+  - If your ``sdm_schemas`` PR has issues, check that the schema conforms to Felis's data model and valid SQL tables can be created with ``felis validate/create`` (https://felis.lsst.io/user-guide/cli.html#felis-validate) using:
+  - Alembic migrations should be automatically created by a git workflow after your ``sdm_schemas`` pull request completes.
 
-  .. code-block:: bash
-
-    pip install lsst-felis
-
-    felis validate [options] [edited cdb_*.yml files]
-
-  - Create a Jira ticket and assign it to Valerie to create a schema migration. (Is there a separate migration request process?)
-
-
-* Create an Alembic Migration
+Create an Alembic Migration (manually)
+======================================
 
   - Alembic (https://alembic.sqlalchemy.org/en/latest/front.html) keeps track of versioning by autogenerated migrations to sync the test stands and summit databases.
   - Versioning our database schema changes allows us to apply edits and move the database’s state forward or backward as needed.
@@ -38,14 +33,16 @@ Schema Migration Process
        - Ensure constraints and renamed columns are correct.
 
 
-* Test alembic migration and code to populate the new columns/tables at TTS/BTS if Summit schema is changing
+Test alembic migration
+======================
 
-   - You will need to rest the migration on your ``consdb`` branch on TTS before merging or applying the migration to the Summit.
+   - You will need to test the version migrating the TTS db using your ``consdb`` branch on before merging or applying the migration to the Summit.
+   - This testing should include testing any code that populates the new columns/tables at TTS/BTS if Summit schema is changing.
 
    1. Update the deployment on the test stand:
 
      - Choose the appropriate test stand (TTS, BTS)
-     - Create a branch in ``phalanx`` and edit the corresponding test stand environment file ``phalanx/applications/consdb/values-<test stand>.yaml`` to point to your branch.
+     - Create a branch in ``phalanx`` and edit the corresponding test stand environment file ``phalanx/applications/consdb/values-<test stand>.yaml`` to point to your branch's built docker image (tickets-DM-###).
      - Coordinate and announce in the appropriate slack channel that you will begin testing your migrations.
      - Update the consdb deployment in ``<url.to.teststand>/argo-cd`` to use your ``phalanx`` branch in the ``Target Revision``. Refresh and check pod logs.
      - Verify the tables that you will be upgrading exist using ``psql``
@@ -75,12 +72,14 @@ Schema Migration Process
   - Update your existing phalanx branch to point the environment based deployments to this consdb tag. You are able to retest on the test stand at this point, hopefully there were no changes to your consdb pull request so this step is trivial.
 
 
-* Deploy migration in synchrony at Summit (if necessary), USDF, and Prompt Release (if necessary)
+Deploy migration in synchrony at Summit (if necessary), USDF, and Prompt Release (if necessary)
+-----------------------------------------------------------------------------------------------
 
       - What is prompt release?
 
 
-* Deploy code to populate at Summit and/or USDF
+Deploy code to populate db at Summit and/or USDF
+------------------------------------------------
 
      - Follow the testing steps above for testing alembic migration and code at TTS/BTS, before the you consider deploying at the summit.
 

doc/user-guide/schemas.rst

@@ -2,22 +2,31 @@
 Schemas
 ########
 
-* Types of schemas
-    * Summit for observers and Summit systems
+Types of schemas
+================
+    Summit for observers and Summit systems
+    ---------------------------------------
         * Smallest, contains primary key information from HeaderService and additional information from other Summit systems, including experimental and engineering data
-    * USDF for staff and analytical uses
+
+    USDF for staff and analytical uses
+    ----------------------------------
         * Largest, contains a full replica of the Summit plus additional information from USDF systems including Prompt Processing and Data Release Production, possibly Calibration Products Production, and human annotations from processing campaigns
-    * Release for science users
+
+    Release for science users
+    -------------------------
         * Near-real-time "prompt" ConsDB replicates a subset of the USDF version
         * Data Release ConsDB is a snapshot of a subset of the USDF version with data pertaining to the exposures/visits in the DR
-* Schema browser
 
-* Versioning
+Schema browser
+==============
+
+Versioning
+==========
 
-https://rubin-obs.slack.com/archives/C07QJMQ7L4A/p1730482605167509
+(https://rubin-obs.slack.com/archives/C07QJMQ7L4A/p1730482605167509)
 
-- schemas are using semantic versioning
-- should be consistent across all schemas, not just ConsDB
+- Schemas are using semantic versioning
+- Should be consistent across all schemas, not just ConsDB
 
     major: backward incompatible changes to the database objects (adding a table, deleting a column)
 
@@ -43,4 +52,4 @@ https://rubin-obs.slack.com/archives/C07QJMQ7L4A/p1730482605167509
     What do users see, how does TAP play into this, do the schema need this type of micro versioning?
 
 
-Services/cosndb repo versioning strategy - services of monthly YY.0M.DD
+- Services/cosndb repo versioning strategy - services of monthly YY.0M.DD