Nomad port collision issue with alloc from different groups

[valodzka]

Nomad version

Nomad v1.8.4 (Build: 22ab32e, Date: 2024-09-17T20:18:34Z)

Environment

Debian GNU/Linux 12

Issue

Port collision occurring during deployment when Nomad allocates the same port (30943) to a new allocation while the previous allocation is still shutting down.

Reproduction steps

1. Initial allocation (692e5352) using port 30943
2. New allocation (dc4c7ccc) assigned same port 30943 (allocations belong to same job but to different groups)
3. Timeline:
   • 09:21:19 - Old allocation receives stop signal
   • 09:21:27 - New allocation attempts to start
   • 09:21:29 - New allocation fails with port collision
   • 09:21:38 - Old allocation finally terminates

Degailed logs

nomad logs:

2025-01-20T09:21:19.057Z [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=692e5352-46b0-d1f2-b8df-caac60e26a22 task=java type=Killing msg="Sent interrupt. Waiting 3m0s before force killing" failed=false
2025-01-20T09:21:19.137Z [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=692e5352-46b0-d1f2-b8df-caac60e26a22 task=java type="Waiting for shutdown delay" msg="Waiting for shutdown_delay of 5s before killing the task." failed=false
2025-01-20T09:21:19.159Z [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=dc4c7ccc-156e-b889-8e77-3910d223b48a task=java type=Received msg="Task received by client" failed=false
2025-01-20T09:21:27.171Z [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=dc4c7ccc-156e-b889-8e77-3910d223b48a task=java type="Task Setup" msg="Building Task Directory" failed=false
2025-01-20T09:21:27.575Z [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=dc4c7ccc-156e-b889-8e77-3910d223b48a task=java type=Started msg="Task started by client" failed=false
2025-01-20T09:21:38.103Z [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=692e5352-46b0-d1f2-b8df-caac60e26a22 task=java type=Terminated msg="Exit Code: 143, Exit Message: \"Docker container exited with non-zero exit code: 143\"" failed=false
2025-01-20T09:21:38.126Z [INFO]  client.alloc_runner.task_runner: Task event: alloc_id=692e5352-46b0-d1f2-b8df-caac60e26a22 task=java type=Killed msg="Task successfully killed" failed=false
2025-01-20T09:21:38.138Z [INFO]  client.alloc_runner.task_runner.task_hook.logmon: plugin process exited: alloc_id=692e5352-46b0-d1f2-b8df-caac60e26a22 task=java plugin=/usr/bin/nomad id=642473
2025-01-20T09:21:38.138Z [INFO]  client.gc: marking allocation for GC: alloc_id=692e5352-46b0-d1f2-b8df-caac60e26a22

new alloc app logs:

Jan 20 09:21:28 15f9cf722360[2894512]: + exec java ... port=30943
Jan 20 09:21:29 15f9cf722360[2894512]: Could not start: java.net.BindException: Address already in use

old alloc app logs:

Jan 20 07:03:15 ac3bf2eca298[2894512]: + exec java ... port=30943 
Jan 20 09:21:24 ac3bf2eca298[2894512]: Shutdown called

Expected Result

Nomad should prevent port collisions by ensuring previous allocation fully releases the port before allowing reuse.

[Juanadelacuesta]

Hello @valodzka!
Thank you for taking the time to report this, in order to understand your issue better and find a solution it would be very helpful if you can provide some more information: Is this a recurrent problem? Can you share your jobspec?

[valodzka]

This is a recurring but infrequent issue. The bug triggers when two randomly selected ports from 12000 Nomad ports coincide. For a single port, chance is 1/12000. With multiple ports per allocation, probability increases significantly - with 1000 servers, 2 allocations per server, and 10 ports per allocation, it reaches ~50% per deployment (our deployment is smaller though).

The job I'm using is quite sensitive so I can't share it, but this should be reproducible with any job having:

• multiple groups
• many reserved ports
• running multiple allocs on same server
• allocs having stop time a few dozens of seconds

[Juanadelacuesta]

Hi @valodzka!
Is the port collision happening when you upgrade your tasks? If this is the case, I would recommend to set restart on the job spec long enough to give the old allocation time to die before restarting the replacement.
If on the contrary the collision is not happening on job upgrades and its not between an alloc and its replacement, it is indeed a bug. Can you confirm what is your case?

[valodzka]

Is the port collision happening when you upgrade your tasks? If this is the case, I would recommend to set restart on the job spec long enough to give the old allocation time to die before restarting the replacement.

Yes, it happens during job deployment. Do I understand correctly that this is expected Nomad behavior:

• Two tasks (A and B) from one job are updated on the same server (from version 0 to 1)
• B0 and A1 are assigned the same port
• Task A0 is stopped, then A1 tries to start before B0 stops and gets a port conflict
• Task A1 fails and the only solution is to configure restart parameters - it's impossible to avoid the conflict
  ?

[Juanadelacuesta]

As far as I understand, it is expected behaviour if you have the port statically configured. Nomad strives for service, so the new allocation will start without stoping the old one to ensure there is always at least one allocation serving at all times. If the port is not static, getting the same port randomly twice should be a very uncommon scenario so there is no mechanism in place for it.
Im asking around in the team if anyone has seen a corner case here where we do get the same random port twice, but it should not happen regularly.

[valodzka]

If the port is not static, getting the same port randomly twice should be a very uncommon scenario

While this is true for one port, as the number of ports increases, the probability of collisions grows very rapidly (see the birthday problem for a similar case with unintuitive probability), and this also increases with the number of servers.

However, if this is the "Nomad way" of handling it, I can think of a workaround, such as adding a startup pre-check script that verifies if the port is free and delays the start if it isn't.

For anyone looking for workaround, if you have startup bash script you can add something like this to it:

ensure_port_free() {
  local -r PORT="$1"
  local -r WAIT=3
  local -r ATTEMPTS=6
  local CHECK_OUT

  for ((i=0; i<ATTEMPTS; i++)); do
    CHECK_OUT=$(ss --listening --tcp --udp --numeric --no-header "( sport = :$PORT )")

    if [[ ! "$CHECK_OUT" =~ ^[[:space:]]*$ ]]; then
      echo "Check $i/$ATTEMPTS: Port $PORT is in use. Waiting $WAIT seconds..."
      sleep $WAIT
    else
      return 0
    fi
  done

  echo "Port $PORT is still in use after $ATTEMPTS checks, aborting startup"
  exit 1
}

ensure_port_free $NOMAD_PORT_port1
ensure_port_free $NOMAD_PORT_port2
ensure_port_free $NOMAD_PORT_port3

...