Experiencing difficulties with batching

[jackstrophysics]

Hi there! I am currently attempting to use Blast in batch mode, but am experiencing some difficulties and was hoping to get some help. I have followed the instructions on how to run Blast locally, up to and including the 'Setup environment file' section, and have setup the input and output files and additional environment variables as directed. However when I run the bash run/blastctl batch up command from the base Blast directory, I receive the following:

+ docker compose --file docker/docker-compose.yml --env-file env/.env.default --env-file env/.env.dev --profile batch up --build --remove-orphans --abort-on-container-exit
[+] Running 8/8
 ✔ Container blast-base-database-1       Created                                                                                                                                                        0.0s 
 ✔ Container blast-base-batch-1          Created                                                                                                                                                        0.0s 
 ✔ Container blast-base-rabbitmq-1       Created                                                                                                                                                        0.0s 
 ✔ Container blast-base-celery-beat-1    Recreated                                                                                                                                                      0.2s 
 ✔ Container blast-base-flower-1         Recreated                                                                                                                                                      0.1s 
 ✔ Container blast-base-app-1            Recreated                                                                                                                                                      0.2s 
 ✔ Container blast-base-celery-worker-1  Recreated                                                                                                                                                      0.2s 
 ✔ Container blast-base-celery-worker-2  Recreated                                                                                                                                                      0.2s 
Attaching to app-1, batch-1, celery-beat-1, celery-worker-1, celery-worker-2, database-1, flower-1, rabbitmq-1
batch-1          | Error: you need to provide a host and port to test.
batch-1          | Usage:
batch-1          |     wait-for-it.sh host:port [-s] [-t timeout] [-- command args]
batch-1          |     -h HOST | --host=HOST       Host or IP under test
batch-1          |     -p PORT | --port=PORT       TCP port under test
batch-1          |                                 Alternatively, you specify the host and port as host:port
batch-1          |     -s | --strict               Only execute subcommand if the test succeeds
batch-1          |     -q | --quiet                Don't output any status messages
batch-1          |     -t TIMEOUT | --timeout=TIMEOUT
batch-1          |                                 Timeout in seconds, zero for no timeout
batch-1          |     -- COMMAND ARGS             Execute command with args after the test finishes
database-1       | 2025-01-06 16:14:07+00:00 [Note] [Entrypoint]: Entrypoint script for MariaDB Server 1:10.11.10+maria~ubu2204 started.
app-1            | "/.initializing_data" not found. Running initialization script...
celery-worker-2  | Error: you need to provide a host and port to test.
batch-1 exited with code 1
Aborting on container exit...
[+] Stopping 8/8 Desktop   o View Config   w Enable Watch
 ✔ Container blast-base-celery-worker-1  Stopped                                                                                                                                                        0.2s 
 ✔ Container blast-base-flower-1         Stopped                                                                                                                                                        0.1s 
 ✔ Container blast-base-celery-beat-1    Stopped                                                                                                                                                        0.1s 
 ✔ Container blast-base-rabbitmq-1       Stopped                                                                                                                                                        0.3s 
 ✔ Container blast-base-database-1       Stopped                                                                                                                                                        1.0s 
 ✔ Container blast-base-batch-1          Stopped                                                                                                                                                        0.0s 
 ✔ Container blast-base-app-1            Stopped                                                                                                                                                        0.1s 
 ✔ Container blast-base-celery-worker-2  Stopped  

It seems like wait-for-it.sh is not being provided with a host or port, however docker-entrypoint.batch.sh contains the line bash wait-for-it.sh 0.0.0.0:${WEB_APP_PORT} --timeout=0 which, to my understanding, should be providing precisely these. However, in checking that the environment variables are set, echo ${WEB_APP_PORT} returns none, so my guess is that the .env.default script is not being run first?

In case it is of relevance to Docker installations, I am working on a MacBook Air M3 2024.

Any advice would be greatly appreciated!

[manning-ncsa]

I admit that this is the mode I understood the least when I created the new launcher/wrapper script and the one tested the least 😬 My test of the launch command matches your output. I will investigate and let you know what I discover.

[manning-ncsa]

I created a new issue for this and pushed a fix to this branch. Can you checkout that branch and see if it at least gets you past the launch stage? On my end I am testing with the example input CSV in the batch folder but no results are being output, so I'm investigating that.

[manning-ncsa]

On my end I am testing with the example input CSV in the batch folder but no results are being output, so I'm investigating that.

There were more bugs that I fixed, and in the process maybe got carried away and ended up changing where the CSV data file is generated. Instead of specifying an output file path, you now specify an output directory, in which timestamped CSV files are generated of the form blast_results.%Y%m%d%H%M%S.csv (e.g. blast_results.20251306192340.csv). This approach reduces the risk that a costly batch run is wasted by accidentally overwriting an existing results file, and it eliminates the need to proactively create a file prior to launching the batch run.

As of this commit, the batch mode runs successfully with (the new) default values.

I'll also note that you can completely reset the Blast batch mode environment using

bash run/blastctl batch purge-db

if for example you started a batch run that aborted and then receive an error 429 Conflict when trying to repeat the run because the named transient already exists in the database.

[manning-ncsa]

A segmentation fault from the RabbitMQ instance? That is bizarre. Try the purge-db command as I showed above before running it again and see if the behavior is repeatable.

[jackstrophysics]

The seg fault persists even after the purge-db command.

[manning-ncsa]

We already discussed this offline, but for reference: Perhaps RabbitMQ was failing before but you did not notice because of the batch container failure. A cursory search yielded this discussion about needing an image built for the M1/2/3 chip architecture.

[djones1040]

you might try my docker/docker-compose.dev.yaml. I specify different platforms for my M1 mac. Unfortunately have not integrated this into the main repo, but maybe Andrew has some ideas.

name: blast-dev

networks:
  internal:
    external: false

volumes:
  blast-db: {}
  django-static: {}
  blast-data:
    name: "blast-data"

services:

  app:
    platform: linux/amd64
    extends:
      file: docker-compose.yml
      service: app
    build:
      context: ../app/
    ports:
      - 127.0.0.1:8000:${WEB_APP_PORT}
    environment:
      - "DEV_MODE=1"
    env_file:
      - ../env/.env.default
      - ../env/.env.dev
    volumes:
      - blast-data:/mnt/data
      - django-static:/app/static
      - ./dustmapsrc:/tmp/.dustmapsrc:ro
      - ../app:/app

  database:
    platform: linux/arm64
    extends:
      file: docker-compose.yml
      service: database
    env_file:
      - ../env/.env.default
      - ../env/.env.dev

  celery-worker:
    platform: linux/amd64
    extends:
      file: docker-compose.yml
      service: celery-worker
    build:
      context: ../app/
    env_file:
      - ../env/.env.default
      - ../env/.env.dev
    volumes:
      - blast-data:/mnt/data
      - django-static:/app/static
      - ./dustmapsrc:/tmp/.dustmapsrc:ro
      - ../app:/app

  celery-beat:
    platform: linux/amd64
    extends:
      file: docker-compose.yml
      service: celery-beat
    build:
      context: ../app/
    env_file:
      - ../env/.env.default
      - ../env/.env.dev
    volumes:
      - blast-data:/mnt/data
      - django-static:/app/static
      - ./dustmapsrc:/tmp/.dustmapsrc:ro
      - ../app:/app

  rabbitmq:
    platform: linux/arm64
    extends:
      file: docker-compose.yml
      service: rabbitmq
    env_file:
      - ../env/.env.default
      - ../env/.env.dev

  flower:
    platform: linux/amd64
    extends:
      file: docker-compose.yml
      service: flower
    build:
      context: ../app/
    env_file:
      - ../env/.env.default
      - ../env/.env.dev
    volumes:
      - blast-data:/mnt/data
      - django-static:/app/static
      - ./dustmapsrc:/tmp/.dustmapsrc:ro
      - ../app:/app

  batch:
    platform: linux/amd64
    extends:
      file: docker-compose.yml
      service: batch
    env_file:
      - ../env/.env.default
      - ../env/.env.dev

[djones1040]

note delicate balance between arm64 and amd64

[djones1040]

honestly I think batch mode might be a bit deprecated. I encourage running local installs, but a simpler work flow might be adding transients using the transient uploads URL: http://127.0.0.1:8000/transient_uploads and then grabbing data with the API following instructions from https://blast.readthedocs.io/en/latest/usage/web_api.html. On my mac I have also had some trouble with the SED fitting stage taking a very long time, so please let me know if you encounter similar issues. I've also had to make some small edits to my docker compose files, so let's keep in touch and please let us know what issues you encounter.