diff --git a/alembic-autogenerate.py b/alembic-autogenerate.py index 23853423..252b0196 100755 --- a/alembic-autogenerate.py +++ b/alembic-autogenerate.py @@ -2,14 +2,17 @@ # # How to use this script: -# 1. Install Felis and sdm_schemas, set environment variables: -# pip install lsst-felis testing.postgresql alembic sqlalchemy pyyaml -# git clone https://github.com/lsst/sdm-schemas +# 1. Install required packages and sdm_schemas, set environment variables: +# pip install lsst-felis testing.postgresql alembic sqlalchemy pyyaml \ +# black psycopg2-binary +# git clone https://github.com/lsst/sdm_schemas # cd sdm_schemas # export SDM_SCHEMAS_DIR=`pwd``` # 2. From the root of the consdb git repo, invoke the script. Supply a # revision message as the command line argument: -# python alembic-autogenerate.py DM-12345 +# python alembic-autogenerate.py this is my revision message "\n" \ +# the message can span multiple lines "\n" \ +# if desired # 3. Revise your auto-generated code as needed. # 4. Remove the autogenerated creation of sql views (visit1, ccdvisit1). # diff --git a/doc/contributor-guide/adding-columns.rst b/doc/contributor-guide/adding-columns.rst index a52fb7fe..bc8f410f 100644 --- a/doc/contributor-guide/adding-columns.rst +++ b/doc/contributor-guide/adding-columns.rst @@ -7,7 +7,7 @@ Structure - ConsDB content must relate to exposures or visits or observations structured like exposures. General time series should go in the Engineering and Facilities Database (EFD). - ConsDB content should generally be scalar values. Large amounts of data, especially arrays or images or cubes, should generally go into the Large File Annex (LFA). -- Avoid arrays expressed as individual columns (e.g. ``something0``, ``something1``, ``something2``) where possible, as this increases the number of columns drastically (and there is `a limit `_), makes it hard to query (``SELECT`` clauses need to list all of these individually, and ``WHERE`` clauses may need to include large ``OR`` or ``AND`` conditions), and potentially requires a lot of database storage space. +- Avoid arrays expressed as individual columns (e.g. ``something0``, ``something1``, ``something2``) where possible, as this increases the number of columns drastically (and there is `a limit `__), makes it hard to query (``SELECT`` clauses need to list all of these individually, and ``WHERE`` clauses may need to include large ``OR`` or ``AND`` conditions), and potentially requires a lot of database storage space. - Columns should be named in all lowercase with underscore (``_``) separators, also known as "snake_case". Data sources diff --git a/doc/index.rst b/doc/index.rst index eae2c8ac..bb642a45 100644 --- a/doc/index.rst +++ b/doc/index.rst @@ -3,8 +3,8 @@ ConsDB ====== -``lsst.consdb`` is developed at https://github.com/lsst-dm/consdb. -You can find Jira issues for this module under the `consdb `_ component. +``lsst.consdb`` is developed at `https://github.com/lsst-dm/consdb `__. +You can find Jira issues for this module under the `ConsDB `__ component. ############# Documentation diff --git a/doc/operator-guide/deployment.rst b/doc/operator-guide/deployment.rst index e234c8ee..49d8b332 100644 --- a/doc/operator-guide/deployment.rst +++ b/doc/operator-guide/deployment.rst @@ -5,14 +5,14 @@ Deployment Database ======== -- Deployments of the Consolidated Database are currently located at +Deployments of the Consolidated Database are currently located at - Summit - USDF (+ dev, use the same underlying database, a replication of Summit) - Base Test Stand (BTS) - Tucson Test Stand (TTS) -- Updates to these deployments may be needed when there are edits to the schema for any of the cdb_* tables defined in sdm_schemas. +Updates to these deployments may be needed when there are edits to the schema for any of the cdb_* tables defined in sdm_schemas. Tools: ------ @@ -24,9 +24,9 @@ Tools: Repositories: ------------- -- ``phalanx`` (https://github.com/lsst-sqre/phalanx) -- ``sdm_schemas`` (https://github.com/lsst/sdm_schemas) -- ``consdb`` (https://github.com/lsst-dm/consdb) +- `phalanx `__ +- `sdm_schemas `__ +- `consdb `__ Access needed: -------------- @@ -39,6 +39,49 @@ Process: -------- +Deploy code to populate db at Summit and/or USDF +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +Follow the testing steps above for testing alembic migration and code at TTS/BTS, before the you consider deploying at the summit. + +The steps to deploy at the summit mirror the steps to test on a test stand with coordination and permission from the observers and site teams. +Access to argo-cd deployments is available via the Summit OpenVPN. +To coordinate your deployment update on the summit, you must attend Coordination Activities Planning (CAP) meeting on Tuesday mornings and announce your request. + +Add it to the CAP SITCOM confluence agenda `here `__ + +The CAP members may tell you a time frame that is acceptable for you to perform these changes. + +They may also tell you specific people to coordinate with to help you take images to test LATISS and LSSTCOMCAMSIM tables. There will be more tables to test eventually. +Channels to note: #rubinobs-test-planning; #summit-announce; #summit-auxtel, and `channel usage guide `__. + +When you get your final approval and designated time to perform the changes to ConsDB, announce on #summit-announce, and follow similar steps as test stand procedure above. + +USDF Deployment Steps +^^^^^^^^^^^^^^^^^^^^^ + +Must happen in synchrony with Summit deployment + +1. Disable (pause) SUBSCRIPTION at USDF. (in what table) +2. Perform the migration at the summit with the steps below before step 3. +3. Connect to the USDF database via psql and perform the alembic migration. +4. Check or test as agreed upon with the ConsDB team. +5. Enable and Refresh Subscription at USDF. + +If there is no impact or coordination with Summit needed: Run alembic migration at USDF, and test as appropriate. + +Summit Deployment Steps +^^^^^^^^^^^^^^^^^^^^^^^ + +1. Use a branch in ``phalanx`` to point to the ConsDB tag for deployment. +2. Set the Argo-CD application ``consdb's`` target revision to your ``phalanx`` branch. +3. Refresh the ConsDB application and review pod logs. +4. Connect to the summit database via psql and perform the alembic migration. +5. Have an image taken with the observing team, then verify database entries with a SQL query or Jupyter notebook. +6. Check your new entries in the database using a jupyter notebook or SQL query in RSP showing your new image has been inserted to the database as expected. + +Once deployment succeeds, set the ``Target Revision`` in Argo-CD back to ``main`` and complete the ``phalanx`` PR for the tested ConsDB tag. + REST API Server -=============== +=============== \ No newline at end of file diff --git a/doc/operator-guide/monitoring.rst b/doc/operator-guide/monitoring.rst index d19c5538..8765ce26 100644 --- a/doc/operator-guide/monitoring.rst +++ b/doc/operator-guide/monitoring.rst @@ -5,26 +5,26 @@ Monitoring Reporting channels ================== -- Users of ConsDB, ConsDBClient (``pqserver``) will usually report via #consolidated-database in rubin-obs.slack.com when they are having issues. -- ConsDB operators should monitor this channel and #ops-usdf, #ops-usdf-alerts for issues and outages reported, as well as escalate verified database issues. +Users of ConsDB, ConsDBClient (``pqserver``) will usually report via #consolidated-database in rubin-obs.slack.com when they are having issues. +ConsDB operators should monitor this channel and #ops-usdf, #ops-usdf-alerts for issues and outages reported, as well as escalate verified database issues. Database ======== -- The ConsDB team is responsible for verifying whether or not the database is up when issues are reported - - They can check the method reported by the users, check using ``psql``/ ``pgcli``, and check in the #ops-usdf slack channel for currently reported issues. +The ConsDB team is responsible for verifying whether or not the database is up when issues are reported +They can check the method reported by the users, check using ``psql``/ ``pgcli``, and check in the #ops-usdf slack channel for currently reported issues. -- Once the ConsDB team has confirmed there is an issue with the database, they should notify #ops-usdf slack channel and USDF DBAs should be responsible for fixing/restarting. +Once the ConsDB team has confirmed there is an issue with the database, they should notify #ops-usdf slack channel and USDF DBAs should be responsible for fixing/restarting. REST API Server =============== -- If we suspect the API server died, the ConsDB team should be responsible for checking and restarting -- Use the appropriate argo-cd deployment graph to check deployment logs, and potentially restart the service. +If we suspect the API server died, the ConsDB team should be responsible for checking and restarting +Use the appropriate argo-cd deployment graph to check deployment logs, and potentially restart the service. Other issues ------------ -- K8s infrastructure died The ConsDB team can verify that that is the problem, but there are likely to be wider issues seen -- USDF or Summit K8s/IT support should be responsible for fixing. +K8s infrastructure died The ConsDB team can verify that that is the problem, but there are likely to be wider issues seen +USDF or Summit K8s/IT support should be responsible for fixing. diff --git a/doc/operator-guide/runbook.rst b/doc/operator-guide/runbook.rst index c88e5aca..a5034267 100644 --- a/doc/operator-guide/runbook.rst +++ b/doc/operator-guide/runbook.rst @@ -2,7 +2,7 @@ RunBook ######## -https://rubinobs.atlassian.net/wiki/spaces/LSSTOps/pages/45665320/Consolidated+Database+ConsDB+Runbook+draft+incomplete +Confluence runbook initial incomplete version `here `__ Overview ======== @@ -82,9 +82,9 @@ Periodic tasks Documentation and Training ========================== -- Links to documentation and training resources +Links to documentation and training resources -- Here: (https://consdb.lsst.io) +LSST io page at `consdb.lsst.io `__ Support ======= diff --git a/doc/user-guide/description-and-goals.rst b/doc/user-guide/description-and-goals.rst index f31bb04d..63ae5e24 100644 --- a/doc/user-guide/description-and-goals.rst +++ b/doc/user-guide/description-and-goals.rst @@ -2,4 +2,4 @@ Description and Goals ###################### -https://dmtn-227.lsst.io/ +`Our data management technote (227) `__