Extensive multiprocessing causes pydevd to crash Python 3.12 (but not 3.11) #269

Nirusu · 2023-12-12T12:42:58Z

Issue

My current setup is:

PyCharm 2023.3 (Professional Edition, #PY-233.11799.259, built on December 3, 2023)
pydev debugger (build 233.11799.259)
CPython 3.12.0, installed via brew install [email protected]
macOS 14.1.2 on a Macbook Pro (2020, M1, 16GB RAM)

Using the following reproducing script:

import time
from tqdm.contrib.concurrent import process_map


def make_request(url: str) -> str:
    time.sleep(60)
    return f"Requesting {url}"


def main():
    requests_to_make = ["https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost", "https://localhost"]
    responses = process_map(make_request, requests_to_make, chunksize=1)
    print(responses)


if __name__ == "__main__":
    main()

I create a new project in PyCharm, create a fresh venv with Python 3.12.0 and install tqdm.

When I now run the Script in Debug mode in PyCharm, then (most of the time) the underlying Python process will crash with EXC_BAD_ACCESS and a similar looking crash log (usually the address at KERN_INVALID_ACCESS varies).

If I reduce the number of items in the list requests_to_make (<30 entries), pydevd usually won't crash.
Crashes seem to start occurring between 30-40 entries, going higher makes the crash more reliable.

When I switch to a venv with Python 3.11.6, this issue does not occur and pydevd works correctly even with a large amount of items in requests_to_make.

Crash log (truncated, full log attached)

crash_log.txt

-------------------------------------
Translated Report (Full Report Below)
-------------------------------------

Process:               Python [75372]
Path:                  /opt/homebrew/*/Python.framework/Versions/3.12/Resources/Python.app/Contents/MacOS/Python
Identifier:            org.python.python
Version:               3.12.0 (3.12.0)
Code Type:             ARM-64 (Native)
Parent Process:        pycharm [71816]
Responsible:           pycharm [71816]
User ID:               503

Date/Time:             2023-12-12 13:31:27.5514 +0100
OS Version:            macOS 14.1.2 (23B92)
Report Version:        12
Anonymous UUID:        43948B48-2A90-9D43-EE02-BD74C35F2DA9

Sleep/Wake UUID:       391E73ED-5781-4072-B943-D273E7FB8613

Time Awake Since Boot: 47000 seconds
Time Since Wake:       2493 seconds

System Integrity Protection: enabled

Crashed Thread:        0  Dispatch queue: com.apple.main-thread

Exception Type:        EXC_BAD_ACCESS (SIGSEGV)
Exception Codes:       KERN_INVALID_ADDRESS at 0x0000808080808110 -> 0x0000008080808110 (possible pointer authentication failure)
Exception Codes:       0x0000000000000001, 0x0000808080808110

Termination Reason:    Namespace SIGNAL, Code 11 Segmentation fault: 11
Terminating Process:   exc handler [75372]

VM Region Info: 0x8080808110 is not in any region.  Bytes after previous region: 70875382033  Bytes before following region: 105001204547312
      REGION TYPE                    START - END         [ VSIZE] PRT/MAX SHRMOD  REGION DETAIL
      commpage (reserved)        1000000000-7000000000   [384.0G] ---/--- SM=NUL  ...(unallocated)
--->  GAP OF 0x5f9000000000 BYTES
      MALLOC_NANO              600000000000-600020000000 [512.0M] rw-/rwx SM=PRV  

Thread 0 Crashed::  Dispatch queue: com.apple.main-thread
0   pydevd_pep_669_tracing_cython.cpython-312-darwin.so	       0x105af9ec0 __pyx_pf_14_pydevd_bundle_29pydevd_pep_669_tracing_cython_15PyStartCallback_2__call__ + 4356
1   pydevd_pep_669_tracing_cython.cpython-312-darwin.so	       0x105af8d3c __pyx_pw_14_pydevd_bundle_29pydevd_pep_669_tracing_cython_15PyStartCallback_3__call__ + 788
2   Python                        	       0x104ea5a98 _PyObject_FastCallDictTstate + 96
3   Python                        	       0x104eb187c _PyObject_Call_Prepend + 136
4   Python                        	       0x104e42014 slot_tp_call + 144
5   Python                        	       0x104d99720 _PyObject_MakeTpCall + 128
6   Python                        	       0x104d4f8b4 call_one_instrument.llvm.7388746509351177983 + 240
7   Python                        	       0x104d4d8a8 call_instrumentation_vector.llvm.7388746509351177983 + 288
8   Python                        	       0x104dc21f8 _PyEval_EvalFrameDefault + 1352
9   Python                        	       0x104ea5a98 _PyObject_FastCallDictTstate + 96
10  Python                        	       0x104e44974 slot_tp_init + 212
11  Python                        	       0x104d6576c type_call + 148
12  Python                        	       0x104d99720 _PyObject_MakeTpCall + 128
13  Python                        	       0x104dcc934 _PyEval_EvalFrameDefault + 44164
14  Python                        	       0x104dae9f8 method_vectorcall.llvm.15811550574524407082 + 184
15  Python                        	       0x104dce3a8 _PyEval_EvalFrameDefault + 50936
16  Python                        	       0x104daea74 method_vectorcall.llvm.15811550574524407082 + 308
17  Python                        	       0x104eac2cc _PyVectorcall_Call.llvm.10235175985831010521 + 152
18  Python                        	       0x104dce3a8 _PyEval_EvalFrameDefault + 50936
19  Python                        	       0x104ce7f08 PyEval_EvalCode + 184
20  Python                        	       0x104d32ef8 builtin_exec + 416
21  Python                        	       0x104df4050 cfunction_vectorcall_FASTCALL_KEYWORDS.llvm.2921711504878963487 + 92
22  Python                        	       0x104dcc7e0 _PyEval_EvalFrameDefault + 43824
23  Python                        	       0x104ce7f08 PyEval_EvalCode + 184
24  Python                        	       0x104c96800 run_eval_code_obj.llvm.11164132066462609367 + 88
25  Python                        	       0x104cb4770 run_mod.llvm.11164132066462609367 + 132
26  Python                        	       0x104cf7b24 pyrun_file + 148
27  Python                        	       0x104d57ae0 _PyRun_SimpleFileObject + 288
28  Python                        	       0x104e1a730 _PyRun_AnyFileObject + 232
29  Python                        	       0x104d6b9a8 pymain_run_file_obj + 220
30  Python                        	       0x104d6b7e4 pymain_run_file + 72
31  Python                        	       0x104e42a30 Py_RunMain + 728
32  Python                        	       0x104e42ef4 Py_BytesMain + 40
33  dyld                          	       0x187a8d0e0 start + 2360

Thread 1:
0   libsystem_kernel.dylib        	       0x187dcc0ac __psynch_cvwait + 8
1   libsystem_pthread.dylib       	       0x187e095fc _pthread_cond_wait + 1228
2   Python                        	       0x104d62c00 PyThread_acquire_lock_timed + 284
3   Python                        	       0x104d1e87c acquire_timed + 292
4   Python                        	       0x104cde61c lock_PyThread_acquire_lock + 56
5   Python                        	       0x104d45340 method_vectorcall_VARARGS_KEYWORDS.llvm.14063008687374067480 + 148
6   Python                        	       0x104dcc7e0 _PyEval_EvalFrameDefault + 43824
7   Python                        	       0x104daeab4 method_vectorcall.llvm.15811550574524407082 + 372
8   Python                        	       0x104d99ed8 thread_run + 92
9   Python                        	       0x104cff3dc pythread_wrapper.llvm.15843660109015549643 + 48
10  libsystem_pthread.dylib       	       0x187e09034 _pthread_start + 136
11  libsystem_pthread.dylib       	       0x187e03e3c thread_start + 8

Thread 2:
0   libsystem_kernel.dylib        	       0x187dcce40 __recvfrom + 8
1   _socket.cpython-312-darwin.so 	       0x1045dceb4 sock_recv_impl + 36
2   _socket.cpython-312-darwin.so 	       0x1045dc450 sock_call_ex + 192
3   _socket.cpython-312-darwin.so 	       0x1045dce6c sock_recv_guts + 56
4   _socket.cpython-312-darwin.so 	       0x1045db104 sock_recv + 108
5   Python                        	       0x104deaf28 method_vectorcall_VARARGS.llvm.14063008687374067480 + 128
6   Python                        	       0x104dcc7e0 _PyEval_EvalFrameDefault + 43824
7   Python                        	       0x104daeab4 method_vectorcall.llvm.15811550574524407082 + 372
8   Python                        	       0x104d99ed8 thread_run + 92
9   Python                        	       0x104cff3dc pythread_wrapper.llvm.15843660109015549643 + 48
10  libsystem_pthread.dylib       	       0x187e09034 _pthread_start + 136
11  libsystem_pthread.dylib       	       0x187e03e3c thread_start + 8

Thread 3:
0   libsystem_kernel.dylib        	       0x187dcbea8 __semwait_signal + 8
1   libsystem_c.dylib             	       0x187caf2f8 nanosleep + 220
2   Python                        	       0x104d1ee90 time_sleep + 124
3   Python                        	       0x104df4534 cfunction_vectorcall_O.llvm.2921711504878963487 + 108
4   Python                        	       0x104dcc7e0 _PyEval_EvalFrameDefault + 43824
5   Python                        	       0x104daeab4 method_vectorcall.llvm.15811550574524407082 + 372
6   Python                        	       0x104d99ed8 thread_run + 92
7   Python                        	       0x104cff3dc pythread_wrapper.llvm.15843660109015549643 + 48
8   libsystem_pthread.dylib       	       0x187e09034 _pthread_start + 136
9   libsystem_pthread.dylib       	       0x187e03e3c thread_start + 8

Thread 4:
0   libsystem_kernel.dylib        	       0x187dcc0ac __psynch_cvwait + 8
1   libsystem_pthread.dylib       	       0x187e095fc _pthread_cond_wait + 1228
2   Python                        	       0x104d62bdc PyThread_acquire_lock_timed + 248
3   Python                        	       0x104d1e87c acquire_timed + 292
4   Python                        	       0x104cde61c lock_PyThread_acquire_lock + 56
5   Python                        	       0x104df52d0 cfunction_call + 72
6   Python                        	       0x104d99720 _PyObject_MakeTpCall + 128
7   Python                        	       0x104dcc194 _PyEval_EvalFrameDefault + 42212
8   Python                        	       0x104daeab4 method_vectorcall.llvm.15811550574524407082 + 372
9   Python                        	       0x104dce3a8 _PyEval_EvalFrameDefault + 50936
10  Python                        	       0x104ea5a98 _PyObject_FastCallDictTstate + 96
11  Python                        	       0x104eb187c _PyObject_Call_Prepend + 136
12  Python                        	       0x104e42014 slot_tp_call + 144
13  Python                        	       0x104ead92c _PyObject_Call + 124
14  Python                        	       0x104d99ed8 thread_run + 92
15  Python                        	       0x104cff3dc pythread_wrapper.llvm.15843660109015549643 + 48
16  libsystem_pthread.dylib       	       0x187e09034 _pthread_start + 136
17  libsystem_pthread.dylib       	       0x187e03e3c thread_start + 8

Thread 5:
0   libsystem_kernel.dylib        	       0x187dcc0ac __psynch_cvwait + 8
1   libsystem_pthread.dylib       	       0x187e095fc _pthread_cond_wait + 1228
2   Python                        	       0x104d62bdc PyThread_acquire_lock_timed + 248
3   Python                        	       0x104d1e87c acquire_timed + 292
4   Python                        	       0x104cde61c lock_PyThread_acquire_lock + 56
5   Python                        	       0x104d45340 method_vectorcall_VARARGS_KEYWORDS.llvm.14063008687374067480 + 148
6   Python                        	       0x104dcc7e0 _PyEval_EvalFrameDefault + 43824
7   Python                        	       0x104daeab4 method_vectorcall.llvm.15811550574524407082 + 372
8   Python                        	       0x104dce3a8 _PyEval_EvalFrameDefault + 50936
9   Python                        	       0x104ea5a98 _PyObject_FastCallDictTstate + 96
10  Python                        	       0x104eb187c _PyObject_Call_Prepend + 136
11  Python                        	       0x104e42014 slot_tp_call + 144
12  Python                        	       0x104ead92c _PyObject_Call + 124
13  Python                        	       0x104d99ed8 thread_run + 92
14  Python                        	       0x104cff3dc pythread_wrapper.llvm.15843660109015549643 + 48
15  libsystem_pthread.dylib       	       0x187e09034 _pthread_start + 136
16  libsystem_pthread.dylib       	       0x187e03e3c thread_start + 8

The text was updated successfully, but these errors were encountered:

Nirusu changed the title ~~Extensive multiprocessing causes pydevd to crash Python on Python 3.12~~ Extensive multiprocessing causes pydevd to crash Python 3.12 (but not 3.11) Dec 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Extensive multiprocessing causes pydevd to crash Python 3.12 (but not 3.11) #269

Extensive multiprocessing causes pydevd to crash Python 3.12 (but not 3.11) #269

Nirusu commented Dec 12, 2023 •

edited

Loading

Extensive multiprocessing causes pydevd to crash Python 3.12 (but not 3.11) #269

Extensive multiprocessing causes pydevd to crash Python 3.12 (but not 3.11) #269

Comments

Nirusu commented Dec 12, 2023 • edited Loading

Issue

Crash log (truncated, full log attached)

Nirusu commented Dec 12, 2023 •

edited

Loading