Using Pydev with embedded sub-interpreters - Crash at Py_EndInterpreter()

[car-bianco]

Hello everyone!

I have been tasked with integrating debugging via Pydev into an application consisting of multiple Python scripts being executed within a C++ process. Every script runs in its own Python interpreter instance. I am using PyDev 9.3.0 on Linux.

When the process enters a certain "shutdown" phase, all interpreters should be terminated by calling Py_EndInterpreter(). When Pydev is enabled, the process is aborted with the following error traceback:

Fatal Python error: Py_EndInterpreter: not the last thread
Python runtime state: initialized

Thread [...] (most recent call first):
  File "[...]/pysrc/pydevd.py", line 2250 in __wait_for_threads_to_finish
  File "[...]/pysrc/pydevd.py", line 2315 in dispose_and_kill_all_pydevd_threads
  File "[...]/pysrc/pydevd.py", line 1768 in process_internal_commands
  File "[...]/pysrc/pydevd.py", line 210 in _on_run
  File "[...]/pysrc/_pydevd_bundle/pydevd_daemon_thread.py", line 49 in run
  File "[...]/python3/lib/python3.10/threading.py", line 1009 in _bootstrap_inner
  File "[...]/python3/lib/python3.10/threading.py", line 966 in _bootstrap

Thread [...] (most recent call first):
  File "[...]/pysrc/_pydevd_bundle/pydevd_comm.py", line 219 in _read_line
  File "[...]/pysrc/_pydevd_bundle/pydevd_comm.py", line 238 in _on_run
  File "[...]/pysrc/_pydevd_bundle/pydevd_daemon_thread.py", line 49 in run
  File "[...]/python3/lib/python3.10/threading.py", line 1009 in _bootstrap_inner
  File "[...]/python3/lib/python3.10/threading.py", line 966 in _bootstrap

Though dispose_and_kill_all_pydevd_threads() is being executed, it looks like the ReaderThread and the PyDBCommandThread are not being closed. They are daemon threads - so they are only ended when the program exits.

Are there any workarounds in order to "gracefully shutdown" those threads anyway, or suggestions as to how this issue can be solved?

[fabioz]

If you execute only in pure-python mode, does this still happen?

i.e.: if you set an environment variable as: PYDEVD_USE_CYTHON=NO

[car-bianco]

If you execute only in pure-python mode, does this still happen?

i.e.: if you set an environment variable as: PYDEVD_USE_CYTHON=NO

This does not seem to make any difference, sadly.

Any suggestion as to what else we should try?

[car-bianco]

Hey @fabioz , Sorry for the double-reply.

After the following two steps, it looks like I can terminate the program successfully.

• Calling pydevd.stoptrace() at the beginning of the "shutdown" phase
• Restoring the commented-out code in

  PyDev.Debugger/_pydevd_bundle/pydevd_comm.py

  Lines 164 to 171 in 2686481

  	# try:
  	# self.sock.shutdown(SHUT_RD)
  	# except:
  	# pass
  	# try:
  	# self.sock.close()
  	# except:
  	# pass

  and reverting the while True: to while not _kill_received:

Would you consider this to be a viable solution? If so, I'd find it reasonable to add the option to explicitly shutdown the reader thread.