Sleep command ignored

[johnaohara]

I have a script with a number of sleep commands that are being ignored. testing sleep in isolation, it is working, however in my benchmarks the script is not pausing for the required duration, e.g.

Script:

  run-benchmark:
  - sh:
      command: sdk install jbang 0.122.0
      prompt:
        "set as default? (Y/n): ": "Y"  
  - sh: sleep 20s
  - signal: BENCHMARK_START
  - sh: jbang wrk2@hyperfoil -R ${{INJECTION_RATE}} -d ${{ BENCHMARK_DURATION }} -c ${{BENCHMARK_CONNECTIONS}} -t ${{BENCHMARK_THREADS}} http://0.0.0.0:8080/hello/greeting/john
  - signal: BENCHMARK_DONE

Output:

�[m�[37m08:01:29.444 run-benchmark:2356@mwperf-server05: run-benchmark
�[m�[37m08:01:30.110 run-benchmark:2356@mwperf-server05: sdk install jbang 0.122.0
jbang 0.122.0 is already installed.
�[m�[37m08:01:50.111 run-benchmark:2356@mwperf-server05: sleep: 20s
�[m�[37m08:01:50.112 reached BENCHMARK_START
�[m�[37m08:01:50.112 capture-prometheus-stats:2358@mwperf-server05: wait-for: BENCHMARK_START

Script:

  setup-kepler:
  - script: install-kepler-bare-metal
  - script: start-kepler-bare-metal
  - sh: sleep 2s

Output:

�[m�[37m08:00:27.673 run-setup:2271@mwperf-server05: sudo systemctl enable container-kepler --now
Created symlink /etc/systemd/system/multi-user.target.wants/container-kepler.service → /usr/lib/systemd/system/container-kepler.service.
Created symlink /etc/systemd/system/default.target.wants/container-kepler.service → /usr/lib/systemd/system/container-kepler.service.
�[m�[37m08:00:29.676 run-setup:2271@mwperf-server05: sleep: 2s
�[m�[37m08:00:29.677 run-setup:2271@mwperf-server05: script-cmd: checkout-application
�[m�[37m08:00:29.678 run-setup:2271@mwperf-server05: checkout-application
�[m�[37m08:00:30.086 run-setup:2271@mwperf-server05: cd /tmp/

[johnaohara]

There appears to be a race cond, effecting the sleep command or using sh: sleep. If I attach a debugger, it works as expected;

08:47:02.802 run-setup:520@mwperf-server05: sudo systemctl enable container-kepler --now
Created symlink /etc/systemd/system/multi-user.target.wants/container-kepler.service → /usr/lib/systemd/system/container-kepler.service.
Created symlink /etc/systemd/system/default.target.wants/container-kepler.service → /usr/lib/systemd/system/container-kepler.service.
08:47:03.631 run-setup:520@mwperf-server05: echo preSleep
preSleep
08:47:27.648 run-setup:520@mwperf-server05: sleep 20

[johnaohara]

Changing the sh command to echo something after sleep works;

- sh: sleep 20; echo done-sleeping #hack to get sleep to work in lab :(

[willr3]

In the first example

  run-benchmark:
  - sh:
      command: sdk install jbang 0.122.0
      prompt:
        "set as default? (Y/n): ": "Y"  
  - sh: sleep 20s
  - signal: BENCHMARK_START
  - sh: jbang wrk2@hyperfoil -R ${{INJECTION_RATE}} -d ${{ BENCHMARK_DURATION }} -c ${{BENCHMARK_CONNECTIONS}} -t ${{BENCHMARK_THREADS}} http://0.0.0.0:8080/hello/greeting/john
  - signal: BENCHMARK_DONE

with output

�[m�[37m08:01:29.444 run-benchmark:2356@mwperf-server05: run-benchmark
�[m�[37m08:01:30.110 run-benchmark:2356@mwperf-server05: sdk install jbang 0.122.0
jbang 0.122.0 is already installed.
�[m�[37m08:01:50.111 run-benchmark:2356@mwperf-server05: sleep: 20s
�[m�[37m08:01:50.112 reached BENCHMARK_START
�[m�[37m08:01:50.112 capture-prometheus-stats:2358@mwperf-server05: wait-for: BENCHMARK_START

it looks like the sleep: 20s was the output of the sleep command in qDup and not sleep in the remote shell. If that is the case then something is amiss with the Scheduling. If it was sh: sleep 20s then we have some output stream tracing to do :)

[willr3]

hmm... created an example and I think the issue is that we both read the logs expecting to see an N second timestamp interval between the sleep N message and the following message but qDup logs the command when it finishes so the interval is between the preceding message and the sleep N

�[m�[37m08:01:30.110 run-benchmark:2356@mwperf-server05: sdk install jbang 0.122.0
jbang 0.122.0 is already installed.
�[m�[37m08:01:50.111 run-benchmark:2356@mwperf-server05: sleep: 20s

and

�[m�[37m08:00:27.673 run-setup:2271@mwperf-server05: sudo systemctl enable container-kepler --now
Created symlink /etc/systemd/system/multi-user.target.wants/container-kepler.service → /usr/lib/systemd/system/container-kepler.service.
Created symlink /etc/systemd/system/default.target.wants/container-kepler.service → /usr/lib/systemd/system/container-kepler.service.
�[m�[37m08:00:29.676 run-setup:2271@mwperf-server05: sleep: 2s

Please let me know if there is something else you are seeing that I missed.