Problems with starting user servers, which lead to corrupt database: BlockingIOError: [Errno 11] Resource temporarily unavailable

[nfx]

Sometimes it's not possible for users to login into hub and they get 500 error. Digging deep, we see

  yield gen.with_timeout(timedelta(seconds=spawner.start_timeout), f)
File "..../site-packages/sudospawner/spawner.py", line 92, in start
  reply = yield self.do(action='spawn', args=self.get_args(), env=self.get_env())
File "..../site-packages/sudospawner/spawner.py", line 59, in do
  p = Subprocess(cmd, stdin=Subprocess.STREAM, stdout=Subprocess.STREAM, stderr=Subprocess.STREAM)
File ".../site-packages/tornado/process.py", line 244, in __init__
  self.proc = subprocess.Popen(*args, **kwargs)
...
File ".../subprocess.py", line 1490, in _execute_child
   restore_signals, start_new_session, preexec_fn)
BlockingIOError: [Errno 11] Resource temporarily unavailable

more than that, db gets corrupt, as servers table in database holds the entry with port that is not running. Maybe it's another issue of Jupyterhub not to validate if entries in that table are actually a valid ones, but it's really annoying.

examining https://github.com/jupyterhub/jupyterhub/blob/master/jupyterhub/user.py we see lots of .commit() calls, but not a single .rollback().

[minrk]

There's an unconditional rollback on every request. It would appear that this is not sufficient to recover the session from whatever failure you are encountering. But I don't see any evidence here that a database rollback would be related to any the issues here.

It's not true that the db is corrupt (or at least, there isn't an example of it described here). It is 100% fine if there's a Server entry in the table that isn't in use yet, though it should get cleaned up eventually.