Nuitka & Python 3.12: GC Segmentation Fault
Hey folks! Have you ever run into a weird issue where your Python code just... crashes? Well, I recently wrestled with a nasty one involving Nuitka, Python 3.12, and a pesky segmentation fault during garbage collection (GC). It's a doozy, but let's break it down.
The Problem: SIGSEGV in the GC
So, what exactly went wrong? My project, a Twisted application, was chugging along fine. Then, after a few hours of running under load, boom – a segmentation fault (SIGSEGV). This means the program tried to access memory it wasn't allowed to, leading to a crash. The root cause? It was happening during garbage collection, that automatic process Python uses to clean up unused memory. Specifically, the crash occurred inside Nuitka's handling of the GC.
The Setup: Twisted, Nuitka, and Ubuntu
Let's talk about the environment. I'm using:
- Twisted: A popular event-driven networking framework for Python.
- Nuitka: A compiler that translates Python code into C code, then compiles that C code into an executable.
- Python 3.12: The latest and greatest (at the time) Python version.
- Ubuntu 24.04 (Noble): My Linux distribution, the latest version.
- Pip and Virtual Environments: Everything is installed into a virtual environment using pip, ensuring that all package versions are well-defined and reproducible.
I built my project using a pyproject.toml file with Nuitka, using its default settings. I built it in a builder image, and the compiled wheels were installed into other images with the same Ubuntu and Python versions.
The Mystery of the Reproducibility
Here's the kicker: reproducing the issue was a bit of a gamble. It would take anywhere from 15 minutes to 5 hours of runtime before the crash happened. Talk about frustrating! This made debugging a real challenge. I am using a lot of concurrent tasks using the Twisted framework. This concurrency probably puts more pressure on the GC.
The Culprits: Python 3.12 and Nuitka
Now, here's where things get interesting. After some detective work, I pinpointed the issue to a combination of Python 3.12 and Nuitka. The problem vanished when I:
- Used Python 3.10
- Disabled Nuitka compilation.
This led me to believe there's a compatibility issue between Nuitka and Python 3.12, specifically in how they interact with the garbage collector.
The Evidence: Core Dumps and Debug Flags
I managed to get a core dump (a snapshot of the program's memory at the time of the crash) with the --debug flag enabled in Nuitka. This provided a crucial stack trace, which is a record of the function calls leading up to the crash.
Here's a snippet from the core dump:
#0 _PyGCHead_PREV (gc=<optimized out>) at /usr/include/python3.12/internal/pycore_gc.h:68
#1 Nuitka_gc_list_move (node=0x7d8a00007d8a, list=0x7ffc435e1ed0) at /opt/buildAgent/temp/buildTmp/build-env-__eo58iu/lib/python3.12/site-packages/nuitka/build/static_src/HelpersAllocator.c:309
#2 0x00007d8a7daa6f41 in Nuitka_finalize_garbage (tstate=0xba5748 <_PyRuntime+459656>, collectable=0x7ffc435e1f60) at /opt/buildAgent/temp/buildTmp/build-env-__eo58iu/lib/python3.12/site-packages/nuitka/build/static_src/HelpersAllocator.c:366
#3 0x00007d8a7daa76d7 in Nuitka_gc_collect_main (tstate=0xba5748 <_PyRuntime+459656>, generation=1, n_collected=0x0, n_uncollectable=0x7d8a798e64c0) at /opt/buildAgent/temp/buildTmp/build-env-__eo58iu/lib/python3.12/site-packages/nuitka/build/static_src/HelpersAllocator.c:581
#4 0x00007d8a7daa78a6 in Nuitka_gc_collect_with_callback (tstate=0xba5748 <_PyRuntime+459656>, generation=1) at /opt/buildAgent/temp/buildTmp/build-env-__eo58iu/lib/python3.12/site-packages/nuitka/build/static_src/HelpersAllocator.c:620
#5 0x00007d8a7daa7981 in Nuitka_gc_collect_generations (tstate=0x7d8a00007d8a) at /opt/buildAgent/temp/buildTmp/build-env-__eo58iu/lib/python3.12/site-packages/nuitka/build/static_src/HelpersAllocator.c:637
#6 0x00007d8a7daaaa68 in Nuitka_PyObject_GC_Link (op=0x7d8a78d014e0) at /opt/buildAgent/temp/buildTmp/build-env-__eo58iu/lib/python3.12/site-packages/nuitka/build/static_src/HelpersAllocator.c:671
#7 0x00007d8a7da92194 in Nuitka_PyType_AllocNoTrack (type=0xa3b7e0 <PyDictItems_Type>) at /opt/buildAgent/temp/buildTmp/build-env-__eo58iu/lib/python3.12/site-packages/nuitka/build/include/nuitka/allocator.h:292
#8 0x00007d8a7da922cc in Nuitka_GC_New (type=0x7d8a00007d8a) at /opt/buildAgent/temp/buildTmp/build-env-__eo58iu/lib/python3.12/site-packages/nuitka/build/include/nuitka/allocator.h:357
#9 0x00007d8a7daa7b01 in _MAKE_DICT_ITERATOR (tstate=0x7d8a00007d8a, dict=0x7d8a78bf3b80, type=0x7d8a00007d8a, is_iteritems=192) at /opt/buildAgent/temp/buildTmp/build-env-__eo58iu/lib/python3.12/site-packages/nuitka/build/static_src/HelpersDictionaries.c:950
#10 0x00007d8a7daab1a1 in DICT_ITERITEMS (tstate=0x7d8a00007d8a, dict=0x7ffc435e1ed0) at /opt/buildAgent/temp/buildTmp/build-env-__eo58iu/lib/python3.12/site-packages/nuitka/build/static_src/HelpersDictionaries.c:974
#11 0x00007d8a7d64fdaf in impl_bobby_tables$36$$36$$36$helper_function_complex_call_helper_pos_keywords_star_dict (tstate=0xba5748 <_PyRuntime+459656>, python_pars=0x7ffc435e1ed0) at /opt/buildAgent/temp/buildTmp/build-via-sdist-22p54vpk/bobby_tables-1.0.dev1/build/lib/bobby_tables.build/module.bobby_tables.c:2893
#12 0x00007d8a7d992379 in impl_bobby_tables$lunch$36$$36$$36$function__9__eat (tstate=0xba5748 <_PyRuntime+459656>, self=0x7ffc435e1ed0, python_pars=0x0) at /opt/buildAgent/temp/buildTmp/build-via-sdist-22p54vpk/bobby_tables-1.0.dev1/build/lib/bobby_tables.build/module.bobby_tables.lunch.c:4550
#13 0x00007d8a7da9773b in Nuitka_CallFunctionPosArgsKwArgs (tstate=0xba5748 <_PyRuntime+459656>, function=0x7d8a7dd09be0, args=0x0, args_size=2, kw=0x7d8a78c56cc0) at static_src/CompiledFunctionType.c:2908
#14 0x00007d8a7da97831 in Nuitka_CallMethodFunctionPosArgsKwArgs (tstate=0xba5748 <_PyRuntime+459656>, function=0x7d8a7dd09be0, object=0x7d8a00007d8a, args=0x7ffc435e1ed0, args_size=888832, kw=0x7d8a78c56cc0) at static_src/CompiledFunctionType.c:2957
#15 0x00007d8a7da99f73 in Nuitka_Method_tp_call (method=0x7d8a79b645e0, args=0x7d8a7974db70, kw=0x7d8a78c56cc0) at /opt/buildAgent/temp/buildTmp/build-env-__eo58iu/lib/python3.12/site-packages/nuitka/build/static_src/CompiledMethodType.c:243
#16 0x00007d8a7d64a2ec in CALL_FUNCTION (tstate=0xba5748 <_PyRuntime+459656>, function_object=0x7d8a79b645e0, positional_args=0x7ffc435e1ed0, named_args=0x0) at /opt/buildAgent/temp/buildTmp/build-env-__eo58iu/lib/python3.12/site-packages/nuitka/build/include/nuitka/calling.h:73
#17 0x00007d8a7d65112c in impl_bobby_tables$36$$36$$36$helper_function_complex_call_helper_pos_keywords_star_dict (tstate=0xba5748 <_PyRuntime+459656>, python_pars=0x7ffc435e1ed0) at /opt/buildAgent/temp/buildTmp/build-via-sdist-22p54vpk/bobby_tables-1.0.dev1/build/lib/bobby_tables.build/module.bobby_tables.c:3186
#18 0x00007d8a7d98e539 in impl_bobby_tables$lunch$36$$36$$36$function__8_heat (tstate=0xba5748 <_PyRuntime+459656>, self=0x7ffc435e1ed0, python_pars=0x0) at /opt/buildAgent/temp/buildTmp/build-via-sdist-22p54vpk/bobby_tables-1.0.dev1/build/lib/bobby_tables.build/module.bobby_tables.lunch.c:3722
#19 0x00007d8a7da9773b in Nuitka_CallFunctionPosArgsKwArgs (tstate=0xba5748 <_PyRuntime+459656>, function=0x7d8a7dcd2340, args=0x0, args_size=4, kw=0x7d8a78bf0500) at static_src/CompiledFunctionType.c:2908
#20 0x00007d8a7da97831 in Nuitka_CallMethodFunctionPosArgsKwArgs (tstate=0xba5748 <_PyRuntime+459656>, function=0x7d8a7dcd2340, object=0x7d8a00007d8a, args=0x7ffc435e1ed0, args_size=888832, kw=0x7d8a78bf0500) at static_src/CompiledFunctionType.c:2957
#21 0x00007d8a7da99f73 in Nuitka_Method_tp_call (method=0x7d8a79b3be70, args=0x7d8a79701dc0, kw=0x7d8a78bf0500) at /opt/buildAgent/temp/buildTmp/build-env-__eo58iu/lib/python3.12/site-packages/nuitka/build/static_src/CompiledMethodType.c:243
#22 0x00007d8a7d64a2ec in CALL_FUNCTION (tstate=0xba5748 <_PyRuntime+459656>, function_object=0x7d8a79b3be70, positional_args=0x7ffc435e1ed0, named_args=0x0) at /opt/buildAgent/temp/buildTmp/build-env-__eo58iu/lib/python3.12/site-packages/nuitka/build/include/nuitka/calling.h:73
#23 0x00007d8a7d64d157 in impl_bobby_tables$36$$36$$36$helper_function_complex_call_helper_pos_star_list_star_dict (tstate=0xba5748 <_PyRuntime+459656>, python_pars=0x7ffc435e1ed0) at /opt/buildAgent/temp/buildTmp/build-via-sdist-22p54vpk/bobby_tables-1.0.dev1/build/lib/bobby_tables.build/module.bobby_tables.c:1213
#24 0x00007d98b02e in impl_bobby_tables$lunch$36$$36$$36$function__3_prep (tstate=0xba5748 <_PyRuntime+459656>, self=0x7ffc435e1ed0, python_pars=0x0) at /opt/buildAgent/temp/buildTmp/build-via-sdist-22p54vpk/bobby_tables-1.0.dev1/build/lib/bobby_tables.build/module.bobby_tables.lunch.c:2296
#25 0x00007da9a244 in Nuitka_CallFunctionVectorcall (tstate=0xba5748 <_PyRuntime+459656>, function=0x7dcd3e40, args=0x0, args_size=138033698333888, kw_names=0x7dccf9d8, kw_size=3) at static_src/CompiledFunctionType.c:3236
#26 0x00007da9a72e in Nuitka_Method_tp_vectorcall (method=0x7dcd3e40, stack=0x7ffc435e1ed0, nargsf=6, kw_names=0x7dccf9d8) at /opt/buildAgent/temp/buildTmp/build-env-__eo58iu/lib/python3.12/site-packages/nuitka/build/static_src/CompiledMethodType.c:188
#27 0x00007dabc19a in CALL_FUNCTION_WITH_POS_ARGS2_KW_SPLIT (tstate=0x7d8a00007d8a, called=0x7ffc435e1ed0, pos_args=0x0, kw_values=0x7ffc435e2920, kw_names=0x7dccf9c0) at /opt/buildAgent/temp/buildTmp/build-env-__eo58iu/lib/python3.12/site-packages/nuitka/build/static_src/HelpersCallingGenerated.c:9386
#28 0x00007d9bb0bd in impl_bobby_tables$drop$36$$36$$36$function__14_getStudents (tstate=0xba5748 <_PyRuntime+459656>, self=0x7ffc435e1ed0, python_pars=0x0) at /opt/buildAgent/temp/buildTmp/build-via-sdist-22p54vpk/bobby_tables-1.0.dev1/build/lib/bobby_tables.build/module.bobby_tables.drop.c:5955
#29 0x00007da9a244 in Nuitka_CallFunctionVectorcall (tstate=0xba5748 <_PyRuntime+459656>, function=0x7d06b840, args=0x0, args_size=138033698333888, kw_names=0x0, kw_size=0) at static_src/CompiledFunctionType.c:3236
#30 0x00007da9a2eb in Nuitka_Function_tp_vectorcall (function=0x7d06b840, stack=0x7d8a7fe31a68, nargsf=1, kw_names=0x0) at static_src/CompiledFunctionType.c:3294
#31 0x0000000000549825 in PyObject_Vectorcall ()
#32 0x00000000005d71d9 in _PyEval_EvalFrameDefault ()
#33 0x000000000054c96d in ?? ()
#34 0x00000000005db2ca in _PyEval_EvalFrameDefault ()
#35 0x000000000054c96d in ?? ()
#36 0x00000000005db2ca in _PyEval_EvalFrameDefault ()
#37 0x000000000054c96d in ?? ()
#38 0x00000000005db2ca in _PyEval_EvalFrameDefault ()
#39 0x00000000005d571b in PyEval_EvalCode ()
#40 0x00000000006084c2 in ?? ()
#41 0x00000000006b44f3 in ?? ()
#42 0x00000000006b425a in _PyRun_SimpleFileObject ()
#43 0x00000000006b408f in _PyRun_AnyFileObject ()
#44 0x00000000006bc0f5 in Py_RunMain ()
#45 0x00000000006bbbdd in Py_BytesMain ()
#46 0x00007d8a800231ca in __libc_start_call_main (main=main@entry=0x518830, argc=argc@entry=5, argv=argv@entry=0x7ffc435e3648) at ../sysdeps/nptl/libc_start_call_main.h:58
#47 0x00007d8a8002328b in __libc_start_main_impl (main=0x518830, argc=5, argv=0x7ffc435e3648, init=<optimized out>, fini=<optimized out>, rtld_fini=<optimized out>, stack_end=0x7ffc435e3638) at ../csu/libc-start.c:360
#48 0x0000000000657005 in _start ()
This trace points to the Nuitka_gc_list_move function in HelpersAllocator.c, which is part of Nuitka's memory management. It also shows calls related to finalizing garbage and collecting garbage.
More Version Details:
- Nuitka Version: 2.8.8 (the one I was using for this run, but I've tried newer versions over the past month).
- Python Version: 3.12.3 (Debian flavor).
- Ubuntu Version: 24.04.3 (Noble).
- GCC Version: 13.3.0.
The Hypothesis: Nuitka's GC Integration in Python 3.12
Based on the evidence, my best guess is that there's a problem with how Nuitka interacts with Python 3.12's garbage collector. Specifically, it seems like something goes wrong when Nuitka manages memory, potentially leading to corrupted data structures and, eventually, the segmentation fault during a GC cycle. This is a very complex interaction, and it takes time to determine where the issue is.
The Workarounds and Next Steps
So, what can you do if you encounter something similar?
- Downgrade Python (if possible): If you're able to, try using Python 3.10. It seems to work fine with Nuitka.
- Disable Nuitka: If you can't downgrade, temporarily disable Nuitka compilation. This is less ideal, but it will prevent the crashes.
- Keep Nuitka Updated: I've been using the latest Nuitka versions, and there is a possibility this is resolved. Make sure to keep Nuitka updated to see if the issue has been fixed.
- Provide Detailed Reports: If you see this error, and you want to contribute, provide as much detail as possible (versions, core dumps, etc.) when reporting the issue, so developers can have enough information to resolve the problem.
Conclusion: A Tricky Bug
This SIGSEGV during GC is a frustrating issue. The core of the problem appears to lie in the integration of Nuitka's memory management with Python 3.12's garbage collection. Hopefully, this detailed look at my problem helps others who have encountered this issue. I'm hoping for a fix, and I will update this if there is any progress.