Unbound JUPYTERHUB_SERVICE_PREFIX leads to 404

[Jon-Lillis]

Hi all, I’m in the process of setting up JupyterHub (with wrapspawner and batchspawner) to better utilise Jupyter on a Slurm cluster, but I’m running into the following issue.

On a successful spawn of the single user session on a cluster node (as indicated by the Slurm output), the user is redirected to '/hostname/user/username/' as expected, but the page 404s.

The issue seems to be the JUPYTERHUB_SERVICE_PREFIX environment variable that’s passed to the single-user session via the job submission script being unbound, instead of ‘/user/username/’.

Following this problem back through the code, I found myself in the construct_child() function of wrapspawner.

def construct_child(self):
        if self.child_spawner is None:
            self.child_spawner = self.child_class(
                user = self.user,
                db   = self.db,
                hub = self.hub,
                authenticator = self.authenticator,
                oauth_client_id = self.oauth_client_id,
                server = self._server,
                config = self.config,
                **self.child_config
                )

When construct_child() is called, self._server contains the correct value. However, following the snippet of code above, self.child_spawner.server - which is later used to define JUPYTERHUB_SERVICE_PREFIX - is None, rather than equal to self._server as I would expect. All other values passed into self.child_class are maintained in self.child_spawner.

What’s odd is that this self.child_spawner.server variable behaves very strangely. Doing absolutely anything to it in the lines that follow the snippet above will solve the 404 problem. If I manually set self.child_spawner.server to self._server straight after self.child_class is created and applied to self.child_spawner, JUPYTERHUB_SERVICE_PREFIX gets populated correctly and the 404 issue goes away. However, both setting the value to None and simply printing it via a self.log.debug(str(self.child_spawner.server)) statement also fixes it.

I’ve been working through the spawner code to try to make sense of this, but I’m not having much luck.
Can anyone explain this strange behaviour?

Could this be related to the patch I had to implement here: jupyterhub/batchspawner#127 (comment)?

If not, what causes the server value passed to the child_class to not be set correctly, and how can this safely be fixed in a more permanent way?

Conda environment

Jupyterhub config file

Jupyterhub output log for run with debug line patch in place
Slurm log for run with patch in place

Jupyterhub output log for run without debug line patch in place
Slurm log for run without patch in place

Thanks in advance.

[welcome]

Thank you for opening your first issue in this project! Engagement like this is essential for open source projects! 🤗

If you haven't done so already, check out Jupyter's Code of Conduct. Also, please try to follow the issue template as it helps other other community members to contribute more effectively.

You can meet the other Jovyans by joining our Discourse forum. There is also an intro thread there where you can stop by and say Hi! 👋

Welcome to the Jupyter community! 🎉

[dstndstn]

ZOMG I can't believe this works! @Jon-Lillis thank you so much for this! Finally got my new Jhub / ProfilesSpawner / SlurmSpawner / Jlab setup working! +1 on the "WHAT IS GOING ON HERE" sentiment. Just logging the None value of client.server is enough to somehow mysteriously make it work?!

For a second I thought maybe touching the 'server' attribute would somehow initialize the traitlet and then it would get linked, but no, that is not happening.

[rcthomas]

Cf. jupyterhub/jupyterhub#3170

I was taking a stab at #41 today and I think I at least cured the reproducer of its traitlets 4/5 problem. I'll probably try this fix myself tomorrow and report back. My sshspawners are fine but my batchspawners aren't, and it's because of this no server thing. I have a feeling something got fixed in traitlets and the result is an oops here.

[rcthomas]

The server argument being passed to child_class() isn't actually getting set, it's different from all the other arguments, and I think it's just getting dropped. The difference is that it's a property with a getter and a setter, not a traitlet that gets set when you call the initializer.

See in JupyterHub Spawner https://github.com/jupyterhub/jupyterhub/blob/master/jupyterhub/spawner.py#L187-L205

The reason it gets populated when you read it is that orm_spawner is a traitlet that gets linked later on. So @dstndstn was close to the answer.

The fix may be to just drop server = self._server to clarify the call to the initializer since it's not doing anything anyway, and then just prod the server getter as in the workaround here. I am not 100% clear on whether it's safe to assume orm_spawner always exists?

[rcthomas]

Or is it safe to just set _server. Doesn't fix the None problem, too bad that would have been one character!

[lalalabox]

I met the same problem. I am trying to use wrapspawner + batchspawner on JupyterHub. Our cluster use Torque(PBS).

wrapspawner: 1.0.0
batchspawner: 1.1.0
python: 3.9.6
jupyterhub : 1.4.2
jupyter core : 4.7.1
jupyter-notebook : 6.4.0
ipython : 7.25.0
ipykernel : 6.0.2
jupyter client : 6.1.12
jupyter lab : 3.0.16
traitlets : 5.0.5

1. If I only use batchspawner.TorqueSpawner following batchspawner example, it works well!

2. However, when it comes to wrapspawner + batchspawner following wrapspawner + batchspawner example and choose local server, I will get 404 error, the same as yours.

[W 2021-07-19 15:44:08.059 JupyterHub log:189] 404 GET /hub/static/style/bootstrap.min.css?v=0e8a7fbd6de23ad6b27ab95802a0a0915af6693af612bc304d83af445529ce5d95842309ca3405d10f538d45c8a3a261b8cff78b4bd512dd9effb4109a71d0ab (@::ffff:127.0.0.1) 0.52ms
[W 2021-07-19 15:44:08.145 JupyterHub log:189] 404 GET /hub/static/style/bootstrap-theme.min.css?v=8b2f045cb5b4d5ad346f6e816aa2566829a4f5f2783ec31d80d46a57de8ac0c3d21fe6e53bcd8e1f38ac17fcd06d12088bc9b43e23b5d1da52d10c6b717b22b3 (@::ffff:127.0.0.1) 0.41ms
[W 2021-07-19 15:44:08.559 JupyterHub log:189] 404 GET /hub/static/style/index.css?v=06e1f33518235bf36a1673b70c8f205bb706470f69d8ec46149d883cf17c53c9719e60341439e57b4057b468037216d7320fcc8afb920fdb6216b37aed5277f6 (@::ffff:127.0.0.1) 0.42ms

3. However, when I add environment variable JUPYTERHUB_SERVICE_PREFIX in jupyterhub_config.py, such as:

c.Spawner.environment = {
    'JUPYTERHUB_SERVICE_PREFIX': '/user/my_username',
}

I can launch and open local server successfully.

Therefore, there must be something wrong with JUPYTERHUB_SERVICE_PREFIX

Thanks to @Jon-Lillis instruction:

What’s odd is that this self.child_spawner.server variable behaves very strangely. Doing absolutely anything to it in the lines that follow the snippet above will solve the 404 problem. If I manually set self.child_spawner.server to self._server straight after self.child_class is created and applied to self.child_spawner, JUPYTERHUB_SERVICE_PREFIX gets populated correctly and the 404 issue goes away. However, both setting the value to None and simply printing it via a self.log.debug(str(self.child_spawner.server)) statement also fixes it.

I manually add a line in wrapspawner.py, such as:

def construct_child(self):
        if self.child_spawner is None:
            self.child_spawner = self.child_class(
                user = self.user,
                db   = self.db,
                hub  = self.hub,
                authenticator = self.authenticator,
                oauth_client_id = self.oauth_client_id,
                server = self._server,
                config = self.config,
                **self.child_config
                )
            # add here!!! to fix JUPYTERHUB_SERVICE_PREFIX  problem
            self.child_spawner.server = self._server

Then, everything works fine!🥳

By the way, in order to use wrapspawner.ProfilesSpawner + batchspawner.TorqueSpawner successfully, I use patch for batchspawner at here #24 . Thanks again! @Jon-Lillis

[rcthomas]

Just doing this was enough for me: _ = self.child_spawner.server right before returning the child_spawner. Kinda weird.

[jbeal-work]

The fix here was useful for me ( Running the ui from a container in front of a LSF system ).