Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

SIMSBIOHUB-627: Migrate BCTW Feature Branch #1437

Merged
merged 62 commits into from
Dec 17, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
62 commits
Select commit Hold shift + click to select a range
0bba169
WIP BCTW Migration
NickPhura Oct 9, 2024
7b3f4fe
WIP BCTW Migration
NickPhura Oct 9, 2024
7d1953c
feat: created migration for ats,lotek and vectronic BCTW telemetry ta…
MacQSL Oct 9, 2024
29571ff
Merge remote-tracking branch 'origin/SIMSBIOHUB-627-md-telemetry-tabl…
NickPhura Oct 9, 2024
125fe40
fix: added triggers
MacQSL Oct 9, 2024
2547f2c
WIP BCTW Migration
NickPhura Oct 9, 2024
1f69023
Merge branch 'SIMSBIOHUB-627' of https://github.com/bcgov/biohubbc in…
MacQSL Oct 9, 2024
84f95c3
WIP BCTW Migration fixes
NickPhura Oct 10, 2024
3c26a02
fix: changed file name
MacQSL Oct 10, 2024
9708f91
Merge branch 'SIMSBIOHUB-627' of https://github.com/bcgov/biohubbc in…
MacQSL Oct 10, 2024
52d59b0
Add untested constraints to deployment table
NickPhura Oct 10, 2024
ee7abb0
chore: updating migration
MacQSL Oct 10, 2024
a49ae51
Merge branch 'SIMSBIOHUB-627' of https://github.com/bcgov/biohubbc in…
MacQSL Oct 10, 2024
113277e
Add constraint to lotek credential table
NickPhura Oct 10, 2024
0d2902a
Fixes
NickPhura Oct 10, 2024
5231393
fix: added new indexes
MacQSL Oct 10, 2024
692592e
Merge branch 'SIMSBIOHUB-627' of https://github.com/bcgov/biohubbc in…
MacQSL Oct 10, 2024
553d8e6
fix: updated indexes and primary keys
MacQSL Oct 10, 2024
001087d
fix: migration tweaks
MacQSL Oct 10, 2024
3a01525
Add deployment/device triggers
NickPhura Oct 10, 2024
52cce73
Merge branch 'SIMSBIOHUB-627' of https://github.com/bcgov/biohubbc in…
NickPhura Oct 10, 2024
995e4e2
chore: updated lookup descriptions and notes
MacQSL Oct 10, 2024
5edd176
Fixes
NickPhura Oct 10, 2024
7eba706
Merge branch 'SIMSBIOHUB-627' of https://github.com/bcgov/biohubbc in…
NickPhura Oct 10, 2024
d2afdd5
fix: updated comments
MacQSL Oct 10, 2024
652798c
BCTW Data Migration: Device CRUD (#1394)
MacQSL Oct 11, 2024
3f8b369
Merge branch 'dev' of https://github.com/bcgov/biohubbc into SIMSBIOH…
MacQSL Oct 16, 2024
5cc3796
SIMSBIOHUB-627 Telemetry (#1397)
MacQSL Oct 17, 2024
b400d57
SIMSBIOHUB-627 - Deployments (#1398)
NickPhura Oct 21, 2024
bd8b6bb
SIMSBIOHUB-627: Rename Deployment Files (#1402)
NickPhura Oct 21, 2024
4e915d6
SIMSBIOHUB-627: Manual Telemetry CRUD (#1403)
MacQSL Oct 21, 2024
28b2c99
SIMSBIOHUB-627: Get Telemetry endpoints (#1405)
MacQSL Oct 22, 2024
8b4059f
SIMSBIOHUB-627: Device Endpoints (#1404)
NickPhura Oct 23, 2024
c797538
SIMSBIOHUB-627: Manual Telemetry Endpoints (#1407)
MacQSL Oct 23, 2024
324cf0b
SIMSBIOHUB-627: Import Telemetry CSV (#1410)
MacQSL Oct 24, 2024
7bc4d06
SIMSBIOHUB-627: Deprecate BCTW Device Service (#1411)
MacQSL Oct 25, 2024
6e74f1f
SIMSBIOHUB-627: API Updates (#1412)
NickPhura Oct 25, 2024
8ef66ac
SIMSBIOHUB-627: Deprecate 2 (#1415)
MacQSL Oct 28, 2024
7026dfa
Merge remote-tracking branch 'origin/dev' into SIMSBIOHUB-627
NickPhura Oct 30, 2024
8f8b52f
SIMSBIOHUB-627: Seed (#1417)
MacQSL Oct 30, 2024
3c39184
SIMSBIOHUB-627: API Changes (for Frontend Changes) (#1419)
NickPhura Nov 2, 2024
2703d83
SIMSBIOHUB-627: Frontend Changes (#1420)
NickPhura Nov 8, 2024
9e1cd73
Merge remote-tracking branch 'origin/dev' into SIMSBIOHUB-627
NickPhura Nov 13, 2024
6571610
Merge fixes
NickPhura Nov 13, 2024
f31f752
Merge branch 'dev' of https://github.com/bcgov/biohubbc into SIMSBIOH…
MacQSL Nov 14, 2024
18c121c
fix: merge conflicts with origin/dev
MacQSL Nov 20, 2024
94b2293
fix: dropped unused prop from date/time fields formikProps
MacQSL Nov 20, 2024
181d5c9
Merge remote-tracking branch 'origin/dev' into SIMSBIOHUB-627
NickPhura Nov 22, 2024
f6e7d4c
Merge remote-tracking branch 'origin/dev' into SIMSBIOHUB-627
NickPhura Nov 22, 2024
502999a
Merge remote-tracking branch 'origin/dev' into SIMSBIOHUB-627
NickPhura Nov 25, 2024
085199b
Merge remote-tracking branch 'origin/dev' into SIMSBIOHUB-627
NickPhura Nov 26, 2024
5bf290e
Cronjob Deployment Configs (#1433)
MacQSL Nov 27, 2024
9f7d521
SIMSBIOHUB-627: API - Telemetry Cronjob (#1424)
MacQSL Nov 28, 2024
fdc21f7
SIMSBIOHUB-627: Find Telemetry (#1434)
NickPhura Nov 28, 2024
14053b1
SIMSBIOHUB-646: Add Device Keys Button to Manage Deployments & Misc U…
mauberti-bc Dec 6, 2024
8ef3a0b
Merge remote-tracking branch 'origin/dev' into SIMSBIOHUB-627
NickPhura Dec 10, 2024
25a1fc3
Merge remote-tracking branch 'origin/dev' into SIMSBIOHUB-627
NickPhura Dec 11, 2024
227667d
ignore-skip
NickPhura Dec 11, 2024
19c4239
SIMSBIOHUB-627: Fix Manage Telemetry Page (#1452)
NickPhura Dec 16, 2024
ebf2c96
SIMSBIOHUB-627: Fix Survey Spatial Telemetry Components (#1453)
NickPhura Dec 17, 2024
7747ada
ignore-skip
NickPhura Dec 17, 2024
77e9a3a
Minor Fixes
NickPhura Dec 17, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 33 additions & 18 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -19,28 +19,39 @@ export $(shell sed 's/=.*//' .env)
# 2. Edit the `.env` file as needed to update variables and secrets
# 3. Run `make web`

env: | setup ## Copies the default ./env_config/env.docker to ./.env
setup: | setup-env check-env ## Copies the default ./env_config/env.docker to ./.env
env: | setup-env check-env ## Copies the default ./env_config/env.docker to ./.env

postgres: | close build-postgres run-postgres ## Performs all commands necessary to run the postgres project (db) in docker
backend: | close build-backend run-backend ## Performs all commands necessary to run all backend projects (db, api) in docker
web: | close build-web check-env run-web ## Performs all commands necessary to run all backend+web projects (db, api, app) in docker
postgres: | close check-env build-postgres run-postgres ## Performs all commands necessary to run the postgres project (db) in docker
backend: | close check-env build-backend run-backend ## Performs all commands necessary to run all backend projects (db, api) in docker
web: | close check-env build-web run-web ## Performs all commands necessary to run all backend+web projects (db, api, app) in docker

db-setup: | build-db-setup run-db-setup ## Performs all commands necessary to run the database migrations and seeding
db-setup: | check-env build-db-setup run-db-setup ## Performs all commands necessary to run the database migrations and seeding

clamav: | build-clamav run-clamav ## Performs all commands necessary to run clamav
clamav: | check-env build-clamav run-clamav ## Performs all commands necessary to run clamav

fix: | lint-fix format-fix ## Performs both lint-fix and format-fix commands

## ------------------------------------------------------------------------------
## Setup/Cleanup Commands
## Setup Commands
## ------------------------------------------------------------------------------

setup: ## Prepares the environment variables used by all project docker containers
setup-env: ## Prepares the environment variables used by all project docker containers, by copying the sample 'env.docker' to '.env'. Note: Some variables may need to be updated, like secrets
@echo "==============================================="
@echo "Make: setup - copying env.docker to .env"
@echo "Make: setup-env - copying env.docker to .env"
@echo "==============================================="
@cp -i env_config/env.docker .env

check-env: ## Logs any env vars that are missing or have no value, in the '.env' file
@echo "==============================================="
@echo "Make: check-env - checking for missing env vars"
@echo "==============================================="
@awk -F '=' 'NR==FNR && !/^#/ && NF {a[$$1]; next} !/^#/ && NF && !($$1 in a)' .env env_config/env.docker | while read -r line; do echo "Warning: Missing value for $$line in .env"; done

## ------------------------------------------------------------------------------
## Cleanup Commands
## ------------------------------------------------------------------------------

close: ## Closes all project containers
@echo "==============================================="
@echo "Make: close - closing Docker containers"
Expand All @@ -61,12 +72,6 @@ prune: ## Deletes ALL docker artifacts (even those not associated to this projec
@docker system prune --all --volumes -f
@docker volume prune --all -f

check-env: ## Check for missing env vars
@echo "==============================================="
@echo "Make: check-env - checking for missing env vars"
@echo "==============================================="
@awk -F '=' 'NR==FNR && !/^#/ && NF {a[$$1]; next} !/^#/ && NF && !($$1 in a)' .env env_config/env.docker

## ------------------------------------------------------------------------------
## Build/Run Postgres DB Commands
## - Builds all of the SIMS postgres db projects (db, db_setup)
Expand Down Expand Up @@ -142,6 +147,16 @@ api-container: ## Executes into the api container.
@docker compose exec api bash

## ------------------------------------------------------------------------------
## Cronjob commands
## - You can include additional CLI arguments by appending the `args` param
## - Ex: `make telemetry-cronjob args="--concurrently 100 --batchSize 1000"`
## ------------------------------------------------------------------------------
telemetry-cronjob: ## Run the telemetry cronjob
@echo "==============================================="
@echo "Telemetry Cronjob"
@echo "==============================================="
@docker compose exec api npm run telemetry-cronjob -- $(args)
## ------------------------------------------------------------------------------
## Database migration commands
## ------------------------------------------------------------------------------

Expand All @@ -158,7 +173,7 @@ run-db-setup: ## Run the database migrations and seeding
@docker compose up db_setup

## ------------------------------------------------------------------------------
## clamav commands
## Clamav commands
## ------------------------------------------------------------------------------

build-clamav: ## Build the clamav image
Expand Down Expand Up @@ -324,13 +339,13 @@ log-db-setup: ## Runs `docker logs <container> -f` for the database setup contai
## Runs ts-trace to find typescript compilation issues and hotspots
## Docs: https://github.com/microsoft/typescript-analyze-trace
## ------------------------------------------------------------------------------
trace-app:
trace-app: ## Runs ts-trace to find typescript compilation issues and hotspots in the app
@echo "==============================================="
@echo "Typscript trace - searching App hotspots"
@echo "==============================================="
@cd app && npx tsc -p ./tsconfig.json --generateTrace ts-traces || npx @typescript/analyze-trace --skipMillis 100 --forceMillis 300 --expandTypes ts-traces

trace-api:
trace-api: ## Runs ts-trace to find typescript compilation issues and hotspots in the api
@echo "==============================================="
@echo "Typscript trace - searching for Api hotspots"
@echo "==============================================="
Expand Down
12 changes: 8 additions & 4 deletions api/.pipeline/config.js
Original file line number Diff line number Diff line change
Expand Up @@ -71,6 +71,8 @@ const phases = {
dbName: `${dbName}`,
phase: 'dev',
changeId: deployChangeId,
telemetryCronjobSchedule: '0 0 * * *', // Daily at midnight
telemetryCronjobDisabled: !isStaticDeployment,
suffix: `-dev-${deployChangeId}`,
instance: `${name}-dev-${deployChangeId}`,
version: `${deployChangeId}-${changeId}`,
Expand All @@ -83,7 +85,6 @@ const phases = {
backboneArtifactIntakePath: '/api/artifact/intake',
biohubTaxonPath: '/api/taxonomy/taxon',
biohubTaxonTsnPath: '/api/taxonomy/taxon/tsn',
bctwApiHost: 'https://moe-bctw-api-dev.apps.silver.devops.gov.bc.ca',
critterbaseApiHost: 'https://moe-critterbase-api-dev.apps.silver.devops.gov.bc.ca/api',
nodeEnv: 'development',
s3KeyPrefix: (isStaticDeployment && 'sims') || `local/${deployChangeId}/sims`,
Expand Down Expand Up @@ -114,6 +115,8 @@ const phases = {
dbName: `${dbName}`,
phase: 'test',
changeId: deployChangeId,
telemetryCronjobSchedule: '0 0 * * *', // Daily at midnight
telemetryCronjobDisabled: !isStaticDeployment,
suffix: `-test`,
instance: `${name}-test`,
version: `${version}`,
Expand All @@ -126,7 +129,6 @@ const phases = {
backboneArtifactIntakePath: '/api/artifact/intake',
biohubTaxonPath: '/api/taxonomy/taxon',
biohubTaxonTsnPath: '/api/taxonomy/taxon/tsn',
bctwApiHost: 'https://moe-bctw-api-test.apps.silver.devops.gov.bc.ca',
critterbaseApiHost: 'https://moe-critterbase-api-test.apps.silver.devops.gov.bc.ca/api',
nodeEnv: 'production',
s3KeyPrefix: 'sims',
Expand Down Expand Up @@ -157,6 +159,8 @@ const phases = {
dbName: `${dbName}-spi`,
phase: 'test-spi',
changeId: deployChangeId,
telemetryCronjobSchedule: '0 0 * * *', // Daily at midnight
telemetryCronjobDisabled: !isStaticDeployment,
suffix: `-test-spi`,
instance: `${name}-spi-test-spi`,
version: `${version}`,
Expand All @@ -169,7 +173,6 @@ const phases = {
backboneArtifactIntakePath: '/api/artifact/intake',
biohubTaxonPath: '/api/taxonomy/taxon',
biohubTaxonTsnPath: '/api/taxonomy/taxon/tsn',
bctwApiHost: 'https://moe-bctw-api-test.apps.silver.devops.gov.bc.ca',
critterbaseApiHost: 'https://moe-critterbase-api-test.apps.silver.devops.gov.bc.ca/api',
nodeEnv: 'production',
s3KeyPrefix: 'sims',
Expand Down Expand Up @@ -200,6 +203,8 @@ const phases = {
dbName: `${dbName}`,
phase: 'prod',
changeId: deployChangeId,
telemetryCronjobSchedule: '0 0 * * *', // Daily at midnight
telemetryCronjobDisabled: !isStaticDeployment,
suffix: `-prod`,
instance: `${name}-prod`,
version: `${version}`,
Expand All @@ -212,7 +217,6 @@ const phases = {
backboneArtifactIntakePath: '/api/artifact/intake',
biohubTaxonPath: '/api/taxonomy/taxon',
biohubTaxonTsnPath: '/api/taxonomy/taxon/tsn',
bctwApiHost: 'https://moe-bctw-api-prod.apps.silver.devops.gov.bc.ca',
critterbaseApiHost: 'https://moe-critterbase-api-prod.apps.silver.devops.gov.bc.ca/api',
nodeEnv: 'production',
s3KeyPrefix: 'sims',
Expand Down
7 changes: 5 additions & 2 deletions api/.pipeline/lib/api.deploy.js
Original file line number Diff line number Diff line change
Expand Up @@ -25,12 +25,16 @@ const apiDeploy = async (settings) => {
objects.push(
...oc.processDeploymentTemplate(`${templatesLocalBaseUrl}/api.dc.yaml`, {
param: {
NAMESPACE: phases[phase].namespace,
NAME: phases[phase].name,
SUFFIX: phases[phase].suffix,
VERSION: phases[phase].tag,
HOST: phases[phase].host,
APP_HOST: phases[phase].appHost,
CHANGE_ID: phases.build.changeId || changeId,
// Cronjobs
TELEMETRY_CRONJOB_SCHEDULE: phases[phase].telemetryCronjobSchedule,
TELEMETRY_CRONJOB_DISABLED: phases[phase].telemetryCronjobDisabled,
// Node
NODE_ENV: phases[phase].nodeEnv,
NODE_OPTIONS: phases[phase].nodeOptions,
Expand All @@ -42,8 +46,7 @@ const apiDeploy = async (settings) => {
BACKBONE_ARTIFACT_INTAKE_PATH: phases[phase].backboneArtifactIntakePath,
BIOHUB_TAXON_PATH: phases[phase].biohubTaxonPath,
BIOHUB_TAXON_TSN_PATH: phases[phase].biohubTaxonTsnPath,
// BCTW / Critterbase
BCTW_API_HOST: phases[phase].bctwApiHost,
// Critterbase
CB_API_HOST: phases[phase].critterbaseApiHost,
// S3
S3_KEY_PREFIX: phases[phase].s3KeyPrefix,
Expand Down
2 changes: 1 addition & 1 deletion api/.pipeline/lib/clean.js
Original file line number Diff line number Diff line change
Expand Up @@ -67,7 +67,7 @@ const clean = (settings) => {
namespace: phaseObj.namespace
});

oc.raw('delete', ['all,pvc,secrets,Secrets,secret,configmap,endpoints,Endpoints'], {
oc.raw('delete', ['all,pvc,secrets,Secrets,secret,configmap,endpoints,Endpoints,cronjobs,Cronjobs'], {
selector: `app=${phaseObj.instance},env-id=${phaseObj.changeId},!shared,github-repo=${oc.git.repository},github-owner=${oc.git.owner}`,
wait: 'true',
namespace: phaseObj.namespace
Expand Down
6 changes: 6 additions & 0 deletions api/.pipeline/templates/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -10,3 +10,9 @@ The pipeline code builds and deploys all pods/images/storage/etc needed to deplo
- Create ObjectStore Secret

The included templates under `prereqs` can be imported via the "Import YAML" page in OpenShift.

## Telemetry Cronjob

How to manually trigger cronjob?

- `oc create job --from=cronjob/biohubbc-telemetry-cronjob-<suffix> <name of job>`
Loading
Loading