Kernel Hangs in OSS-Code Jupyter Plugin when using progress_apply to apply np.array

[jbhanks]

Applies To

• Notebooks (.ipynb files)
• Interactive Window and/or Cell Scripts (.py files with #%% markers)

What happened?

Using the tqdm progress bar with pandas.DataFrame.progress_apply in the Jupyter plugin for OSS-Code results in a kernel hang after execution. The issue does not occur when running the same code in the JupyterLab web UI with the same environment.

Environment info

• OSS-Code Version: 1.96.3
• Jupyter Plugin Version: ms-toolsai.jupyter, 2024.11.0
• Python Version: 3.12.8
• tqdm Version: 4.67.1
• Numpy Version: 2.0.1
• pandas Version: 2.2.3
• Operating System: EndeavourOS, kernel 6.6.70-1-lts

Steps to Reproduce

1. Install the required libraries:

   pip install pandas tqdm ipywidgets

2. Open a Jupyter notebook in OSS-Code.

3. Run the following minimal reproducible example:

   import pandas as pd
   import numpy as np
   import threading
   from tqdm.auto import tqdm
   
   tqdm.pandas()
   
   # Test Data
   data = {'id': [1, 2, 3, 4, 5], 'value': [10, 20, 30, 40, 50]}
   df = pd.DataFrame(data)
   
   print("Threads before progress_apply:", threading.enumerate())
   r = df.progress_apply(np.array)
   print("Threads after progress_apply:", threading.enumerate())

4. Attempt to run another cell, such as:

   print("hello")

Expected Behavior

• The progress_apply operation completes successfully.
• Subsequent cells execute without any issues.

Observed Behavior

• The progress_apply operation completes, but an additional thread remains active.
• Subsequent cells hang indefinitely and do not complete.
• Example thread state output:

  Threads before progress_apply: [<_MainThread(MainThread, started 129631681640000)>, <Thread(IOPub, started daemon 129631616812736)>, <Heartbeat(Heartbeat, started daemon 129631608420032)>, <Thread(Thread-1 (_watch_pipe_fd), started daemon 129631374014144)>, <Thread(Thread-2 (_watch_pipe_fd), started daemon 129631365621440)>, <ControlThread(Control, started daemon 129631357228736)>, <HistorySavingThread(IPythonHistorySavingThread, started 129631348836032)>]
  Threads after progress_apply: [<_MainThread(MainThread, started 129631681640000)>, <Thread(IOPub, started daemon 129631616812736)>, <Heartbeat(Heartbeat, started daemon 129631608420032)>, <Thread(Thread-1 (_watch_pipe_fd), started daemon 129631374014144)>, <Thread(Thread-2 (_watch_pipe_fd), started daemon 129631365621440)>, <ControlThread(Control, started daemon 129631357228736)>, <HistorySavingThread(IPythonHistorySavingThread, started 129631348836032)>, <TMonitor(Thread-24, started daemon 129630451197632)>]
  

Additional Observations

• Running the same code in the JupyterLab web UI does not result in any hanging behavior.
• The issue appears to be specific to how the Jupyter plugin in OSS-Code interacts with threads spawned by tqdm.
• Using regular apply instead of progress_apply resolves the issue, but progress bars are a desired feature.

Steps Taken

• Verified the issue persists across different notebooks and kernel restarts.
• Verified the same code runs without issues in JupyterLab web UI.
• Confirmed that the environment is identical in both cases (Python, tqdm, and pandas versions).

---

VS Code Version

1.96.3

Jupyter Extension Version

2024.11.0

Jupyter logs

Code - OSS (1.96.3, undefined, desktop)
Jupyter Extension Version: 2024.11.0.
Python Extension Version: 2024.23.0-dev.
Pylance Extension not installed.
Platform: linux (x64).
Temp Storage folder ~/.config/Code - OSS/User/globalStorage/ms-toolsai.jupyter/version-2024.11.0
No workspace folder opened.
02:26:43.891 [warn] No interpreter with path ~/Gits/nyc-lien-properties/myenv/bin/python found in Python API, will convert Uri path to string as Id ~/Gits/nyc-lien-properties/myenv/bin/python
02:26:44.438 [warn] Disposing old controller startUsingPythonInterpreter:'.jvsc74a57bd03870672a13622dd6e253fee2e169f3c25df9ac7ec1c680c3e027325f5a55bc64./home/~/Venvs/py4E/python./home/~/Venvs/py4E/python.-m#ipykernel_launcher' for view = 'jupyter-notebook'
02:26:44.438 [warn] Disposing old controller startUsingPythonInterpreter:'.jvsc74a57bd03870672a13622dd6e253fee2e169f3c25df9ac7ec1c680c3e027325f5a55bc64./home/~/Venvs/py4E/python./home/~/Venvs/py4E/python.-m#ipykernel_launcher (Interactive)' for view = 'interactive'
02:26:46.026 [warn] Kernel Spec for 'MyEnv Kernel' (~/.local/share/jupyter/kernels/myenv/kernel.json) hidden, as we cannot find a matching interpreter argv = '/home/~/Gits/nyc-lien-properties/myenv/bin/python'. To resolve this, please change '/home/~/Gits/nyc-lien-properties/myenv/bin/python' to point to the fully qualified Python executable.
02:26:46.029 [warn] The following kernels use interpreters that are no longer valid or not recognized by Python extension, Kernels startUsingLocalKernelSpec:'id=.undefined.-xfrozen_modules=off./home/~/Venvs/py4E/python./home/~/Venvs/py4E/python.-xfrozen_modules=off#-m#ipykernel_launcher'(interpreterId='~/Venvs/py4E/bin/python') and valid interpreter ids include /bin/python, /sbin/python, /usr/bin/python, /usr/sbin/python, /home/~/miniconda3/bin/python, /home/~/miniconda3/envs/cfiber/bin/python, /home/~/miniconda3/envs/cfiber2/bin/python
02:26:46.029 [warn] Disposing old controller startUsingLocalKernelSpec:'.undefined.-xfrozen_modules=off./home/~/Venvs/py4E/python./home/~/Venvs/py4E/python.-xfrozen_modules=off#-m#ipykernel_launcher' for view = 'jupyter-notebook'
02:26:46.029 [warn] Disposing old controller startUsingLocalKernelSpec:'.undefined.-xfrozen_modules=off./home/~/Venvs/py4E/python./home/~/Venvs/py4E/python.-xfrozen_modules=off#-m#ipykernel_launcher (Interactive)' for view = 'interactive'
02:26:49.449 [info] Starting Kernel (Python Path: ~/miniconda3/envs/cfiber/bin/python, Conda, 3.12.8) for '~/Gits/fibermagic-notebooks/test2.ipynb' (disableUI=true)
02:26:50.073 [info] Process Execution: ~/miniconda3/envs/cfiber/bin/python -c "import ipykernel; print(ipykernel.__version__); print("5dc3a68c-e34e-4080-9c3e-2a532b2ccb4d"); print(ipykernel.__file__)"
02:26:50.079 [info] Process Execution: ~/miniconda3/envs/cfiber/bin/python -Xfrozen_modules=off -m ipykernel_launcher --f=/home/~/.local/share/jupyter/runtime/kernel-v3bb9146d31fffc0521bcf9ac2946bfa995f3b6d7b.json
    > cwd: ~/Gits/fibermagic-notebooks
02:26:50.089 [info] Process Execution: ~/miniconda3/envs/cfiber/bin/python -m pip list
02:26:50.581 [info] Kernel successfully started
02:26:50.581 [warn] No nbextensions folder found for kernel .undefined.-xfrozen_modules=off./home/~/miniconda3/envs/cfiber/python./home/~/miniconda3/envs/cfiber/python.-xfrozen_modules=off#-m#ipykernel_launcher

Coding Language and Runtime Version

Python 3.12.8, also 3.10

Language Extension Version (if applicable)

No response

Anaconda Version (if applicable)

conda 24.11.2

Running Jupyter locally or remotely?

Local

Edit: It seems that ipywidgets must be installed for the issue to occur.
Edit2: It also seems that there is an extra thread hanging around in the Lab webUI, but it does not prevent execution of further cells.