diff --git a/README.md b/README.md index 16307bb..6239425 100644 --- a/README.md +++ b/README.md @@ -1,32 +1,70 @@ -# Workstation Housekeeping v1.11 +## Workstation Cleaner (wscleaner) -Scripts to manage data on the NGS workstation +Workstation Cleaner (wscleaner) deletes local directories that have been uploaded to the DNAnexus cloud storage service. +When executed, Runfolders in the input (root) directory are deleted based on the following criteria: -## Workstation Cleaner (wscleaner) +* A single DNAnexus project is found matching the runfolder name +* All local FASTQ files are uploaded and in a 'closed' state +* X logfiles are present in the DNA Nexus project /Logfiles directory (NB X can be added as a command line argument - default is 5) + +or if the run is identified as a TSO500 run, based on: + * the bcl2fastq2_output.log file created by the automated scripts + AND + * Presence of `_TSO` in the human readable DNANexus project name + +A DNAnexus API key must be cached locally using the `--set-key` option. + +## Workstation Environment +The directory `env/` in this repository contains conda environment scripts for the workstation. These remove conflicts in the PYTHONPATH environment variable by editing the variable when conda is activated. The conda documentation describes where to place these scripts under ['saving environment variables'](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#macos-and-linux). -Delete local directories that have been uploaded to the DNAnexus cloud storage service. -See wscleaner readme for more info +## Install +As descibed above, on the workstation 2 environments exist - wscleaner and wscleaner_test (for development work). +You need to activate these environment before installing with pip (as below). -## ngrok_start.sh -Allow SSH access to the system by running ngrok as a background process. As of v1.11 supports dockerised ngrok instance. +```bash +git clone https://github.com/moka-guys/workstation_housekeeping.git +pip install workstation_housekeeping/wscleaner +wscleaner --version # Print version number +``` -### Installation +## Automated usage +The script `wscleaner_command.sh` is called by the crontab. This activates the enviroment and passes the logfile path (and any other non-default arguments). +A development command script `wscleaner_command_dev.sh` can be used to call the test environment and provide testing arguments, eg --dry-run -See knowledge base article for ngrok installation. -### Usage +## Manual Usage -Non-dockerised ngrok: +``` +usage: wscleaner [-h] [--auth AUTH] [--dry-run] [--logfile LOGFILE] + [--min-age MIN_AGE] [--logfile-count LOGFILE_COUNT] + [--version] + root -`sudo bash ngrok_start.sh` +positional arguments: + root A directory containing runfolders to process -Dockerised ngrok: +optional arguments: + -h, --help show this help message and exit + --auth AUTH A text file containing the DNANexus authentication + token + --dry-run Perform a dry run without deleting files + --logfile LOGFILE A path for the application logfile + --min-age MIN_AGE The age (days) a runfolder must be to be deleted + --logfile-count LOGFILE_COUNT + The number of logfiles a runfolder must have in + /Logfiles + --version Print version +``` -`sudo bash ngrok_start.sh docker` +## Test -### output +```bash +# Run from the cloned repo directory after installation +pytest . --auth_token DNA_NEXUS_KEY +``` -The script will output the ngrok connection details +## License +Developed by Viapath Genome Informatics diff --git a/ngrok_start.sh b/ngrok_start.sh deleted file mode 100755 index 54490fb..0000000 --- a/ngrok_start.sh +++ /dev/null @@ -1,37 +0,0 @@ -#!/bin/bash -# ngrok_start.sh - A script to allow SSH access to the system by running ngrok as a background process -# Prints SSH details if ngrok is already running. -# Note: The ngrok process can be closed at anytime with `kill $(pidof ngrok)` - -# Get the process ID of ngrok if it is already running on the system -EXISTING_PROCESS=$(pidof ngrok) -ngrok_instance=$1 -# If ngrok is not running, start as a background process and print the url for SSH access -if [ -z $EXISTING_PROCESS ] ;then - if [[ $ngrok_instance == "docker" ]]; then - # cat the ngrok password into a variable so it can be passed as a environment argument to docker (-e) - # run docker container in detached mode (-d) - # name the instance - # set --net=host attaches the container to the host network, rather than the docker network - this means ports don't need to be remapped. - # and use ngrok/ngrok:latest tcp --region eu 22 : Open an connection to ngrok server on port 22 - # &> /dev/null : Discard stdout and stderr to empty output stream - ngrok_token=$(cat /usr/local/src/mokaguys/.ngrok) - docker run -d --name NGROK --net=host -it -e NGROK_AUTHTOKEN=$ngrok_token ngrok/ngrok:latest tcp 22 --region eu &> /dev/null - else - # nohup [command] : Keep the process running the command even after you quit the session - # ngrok tcp --region eu 22 : Open an connection to ngrok server on port 22 - # &> /dev/null : Discard stdout and stderr to empty output stream - # & : Run as a background process - nohup ngrok tcp --region eu 22 &> /dev/null & - fi - # Pause for a few seconds to allow the connection to complete. - sleep 3 - # Write the ngrok public url for SSH access to the syslog. - # Triggers alert in slack with ssh url details and writes to stderr. - NGROK_URL=$(curl http://localhost:4040/api/tunnels 2>/dev/null | jq ".tunnels[0].public_url") - logger -s "ngrok_start - new workstation host - $NGROK_URL" -else - # If ngrok is already running, print the public url for SSH access to stderr - NGROK_URL=$(curl http://localhost:4040/api/tunnels 2>/dev/null | jq ".tunnels[0].public_url") - echo "ngrok_start - $NGROK_URL" 1>&2 -fi diff --git a/wscleaner/README.md b/wscleaner/README.md deleted file mode 100644 index 9322b0d..0000000 --- a/wscleaner/README.md +++ /dev/null @@ -1,70 +0,0 @@ -# Workstation Cleaner - -Workstation Cleaner (wscleaner) deletes local directories that have been uploaded to the DNAnexus cloud storage service. - -When executed, Runfolders in the input (root) directory are deleted based on the following criteria: - -* A single DNAnexus project is found matching the runfolder name -* All local FASTQ files are uploaded and in a 'closed' state -* X logfiles are present in the DNA Nexus project /Logfiles directory (NB X can be added as a command line argument - default is 5) - -or if the run is identified as a TSO500 run, based on: - * the bcl2fastq2_output.log file created by the automated scripts - AND - * Presence of `_TSO` in the human readable DNANexus project name - -A DNAnexus API key must be cached locally using the `--set-key` option. - -## Workstation Environment -The directory `env/` in this repository contains conda environment scripts for the workstation. These remove conflicts in the PYTHONPATH environment variable by editing the variable when conda is activated. The conda documentation describes where to place these scripts under ['saving environment variables'](https://conda.io/projects/conda/en/latest/user-guide/tasks/manage-environments.html#macos-and-linux). - -## Install -As descibed above, on the workstation 2 environments exist - wscleaner and wscleaner_test (for development work). -You need to activate these environment before installing with pip (as below). - - -```bash -git clone https://github.com/moka-guys/workstation_housekeeping.git -pip install workstation_housekeeping/wscleaner -wscleaner --version # Print version number -``` - -## Automated usage -The script `wscleaner_command.sh` is called by the crontab. This activates the enviroment and passes the logfile path (and any other non-default arguments). -A development command script `wscleaner_command_dev.sh` can be used to call the test environment and provide testing arguments, eg --dry-run - - -## Manual Usage - -``` -usage: wscleaner [-h] [--auth AUTH] [--dry-run] [--logfile LOGFILE] - [--min-age MIN_AGE] [--logfile-count LOGFILE_COUNT] - [--version] - root - -positional arguments: - root A directory containing runfolders to process - -optional arguments: - -h, --help show this help message and exit - --auth AUTH A text file containing the DNANexus authentication - token - --dry-run Perform a dry run without deleting files - --logfile LOGFILE A path for the application logfile - --min-age MIN_AGE The age (days) a runfolder must be to be deleted - --logfile-count LOGFILE_COUNT - The number of logfiles a runfolder must have in - /Logfiles - --version Print version -``` - -## Test - -```bash -# Run from the cloned repo directory after installation -pytest . --auth_token DNA_NEXUS_KEY -``` - -## License - -Developed by Viapath Genome Informatics