Is there a set of versions of Jupyterhub, wrapspawner, batchspawner, and traitlets that are known to work together?

[dstndstn]

Hi,

I'm trying to update my Jupyterhub installation again, and seemingly facing new incarnations of the same bugs I've encountered previously.

Long story short, I have a python3.9 setup with default pip versions of most things, specifically,

jupyterhub                3.1.0
jupyterlab                3.5.2
jupyter_core              5.1.2
traitlets                 5.8.0
batchspawner              1.2.0        (+ manually-applied patch)
wrapspawner               1.0.2.dev0      (git)

and I'm getting the common behavior that batchspawner (Slurm, in my case) works, but once wrapped in ProfilesSpawner, it fails; the user's jupyter server starts up, but never seems to contact the jupyterhub server.

I have read through and tried the things mentioned in #54, #41, #35, and my own ancient #24, but no dice.

I'll give some details on what I'm seeing below, but I guess my question is are there known versions of these packages that work together, and would it be possible to please document them in the README?

Brief Jhub logs:

Jan 06 12:10:12 mn003 start.sh[20715]: [I 2023-01-06 12:10:12.834 JupyterHub log:186] 200 POST /hub/api/batchspawner ([email protected]) 23.27ms
Jan 06 12:10:13 mn003 start.sh[20715]: [I 2023-01-06 12:10:13.022 JupyterHub batchspawner:463] Notebook server job 354117 started at cn002:58213
...
Jan 06 12:10:18 mn003 start.sh[20715]: [W 2023-01-06 12:10:18.810 JupyterHub base:1104] User dlang is slow to become responsive (timeout=10)
Jan 06 12:10:18 mn003 start.sh[20715]: [D 2023-01-06 12:10:18.810 JupyterHub base:1109] Expecting server for dlang at: http://mn003:0/user/dlang/
...
Jan 06 12:12:02 mn003 start.sh[20715]: [W 2023-01-06 12:12:02.744 JupyterHub user:881] dlang's server never showed up at http://mn003:0/user/dlang/ after 120 seconds. Giving up.
Jan 06 12:12:02 mn003 start.sh[20715]:
Jan 06 12:12:02 mn003 start.sh[20715]:     Common causes of this timeout, and debugging tips:
Jan 06 12:12:02 mn003 start.sh[20715]:
Jan 06 12:12:02 mn003 start.sh[20715]:     1. The server didn't finish starting,
Jan 06 12:12:02 mn003 start.sh[20715]:        or it crashed due to a configuration issue.
Jan 06 12:12:02 mn003 start.sh[20715]:        Check the single-user server's logs for hints at what needs fixing.
Jan 06 12:12:02 mn003 start.sh[20715]:     2. The server started, but is not accessible at the specified URL.
Jan 06 12:12:02 mn003 start.sh[20715]:        This may be a configuration issue specific to your chosen Spawner.
Jan 06 12:12:02 mn003 start.sh[20715]:        Check the single-user server logs and resource to make sure the URL
Jan 06 12:12:02 mn003 start.sh[20715]:        is correct and accessible from the Hub.
Jan 06 12:12:02 mn003 start.sh[20715]:     3. (unlikely) Everything is working, but the server took too long to respond.
Jan 06 12:12:02 mn003 start.sh[20715]:        To fix: increase `Spawner.http_timeout` configuration
Jan 06 12:12:02 mn003 start.sh[20715]:        to a number of seconds that is enough for servers to become responsive.
Jan 06 12:12:02 mn003 start.sh[20715]:
Jan 06 12:12:02 mn003 start.sh[20715]: [D 2023-01-06 12:12:02.745 JupyterHub user:930] Stopping dlang

This log message seems to point directly at the issue:

Expecting server for dlang at: http://mn003:0/user/dlang/

where mn003 is the name of my front-end (jhub) server; it looks like the .server attribute isn't getting set, which sounds deeply familiar. On the second line, it looks like batchspawner is reporting the URL of the server (cn002:58213).

Thanks for any suggestions!

[rcthomas]

I think you shouldn't use wrapspawner from git right now, try the latest release instead? See #54. There's some confusion about what version that fix actually worked for, it must be a very old one. I think we ought to revert that or at least move it off to some other branch for "study."

[rcthomas]

Actually I guess you said you tried that @dstndstn. Well, I'm kind of in the same situation as you, I want to re-rationalize my deployment but I'm kind of stuck on the same batchspawner issue that's been addressed multiple times by different people, and then this thing here.

I'm still running off my fork at https://github.com/rcthomas/wrapspawner.git@vivify-server, with a mismatched JupyterHub (2.x on the remote side, 3.x on the hub), batchspawner 1.2.0.

[dstndstn]

Thanks for your reply! I will check out these leads -- much appreciated! -- and, if I manage to get it working, fingers crossed!, try to put in a PR that lists what my setup is in the README (or maybe some other file).

[satary]

Hi!
Did you manage to make it work?
It is painful for me to still use an ancient hub version =(

[dstndstn]

I ran out of time to work on it without getting it working :(

[satary]

Hm.
I got it working eventually (slurm part).
I had to use the current version of batchspawner from git.
I still can not run systemdspawner for some reason and have no idea how to debug it, as I can not understand where to search for logs.

Package                                 Version
--------------------------------------- -----------
...
batchspawner                            1.2.0 as of https://github.com/jupyterhub/batchspawner/tree/d5f9a0bbaa92748267bb1dd2cc4d4a0436670482
ipykernel                               6.22.0
ipympl                                  0.9.3
ipython                                 7.34.0
jupyter_client                          8.1.0
jupyter_core                            5.3.0
jupyter-events                          0.6.3
jupyter-jsmol                           2022.1.0
jupyter_server                          2.5.0
jupyter-telemetry                       0.1.0
jupyterhub                              3.1.1
jupyterhub-systemdspawner               0.17.0
jupyterlab                              3.5.0
jupyterlab_server                       2.21.0
notebook                                6.5.3
traitlets                               5.9.0
wrapspawner                             1.0.1 (pip release)

EDIT
It turned out, that I just got an old systemd version (we are proudly running Ubuntu 18.04 because reasons) and rolling back to systemdspawner 0.15 did the trick. So for now everything is working.