UBSan: runtime undefined behaviours when using -fsanitize=undefined -fno-sanitize-recover #132097
Description
Bug report
Bug description:
We finally managed to fix gh-111178, but this is assuming that Python is compiled with -fsanitize=function. When compiling Python as follows (with clang 18.1.8):
./configure \
-q --with-pydebug \
--with-undefined-behavior-sanitizer \
CC=clang LD=clang \
CFLAGS="-fsanitize=undefined -fno-sanitize-recover" \
LDFLAGS="-fsanitize=undefined -fno-sanitize-recover"we still have some tests failing:
- gh-132097: allow AC to disable fastcall convention to avoid UBSan failures #131605 (comment)
- gh-132097: allow AC to disable fastcall convention to avoid UBSan failures #131605 (comment) (cc @vstinner as the original author to gh-111178: Fix function signature in pyexpat.c #131674).
Others related to the second item above:
0:00:56 load avg: 32.43 [448/486/3] test_xxtestfuzz worker non-zero exit code (Exit code 1) -- running (2): test.test_multiprocessing_spawn.test_processes (54.6 sec), test.test_multiprocessing_forkserver.test_processes (55.2 sec)
Modules/expat/xmlparse.c:6779:5: runtime error: call to function expat_default_handler through pointer to incorrect function type 'void (*)(void *, const char *, int)'
/$HOME/lib/python/cpython/./Modules/_elementtree.c:3212: note: expat_default_handler defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Modules/expat/xmlparse.c:6779:5
Others on fautlhandler, but I'm not sure whether this is the UB that is being reported or something else:
#0 0x7ff4cf25f76b in raise (/lib64/libpthread.so.0+0x1676b) (BuildId: 938e42b7e407d175ee3ef9a89c038168101d330c)
#1 0x55776b2dc6f2 in faulthandler_raise_sigsegv /$HOME/lib/python/cpython/./Modules/faulthandler.c:1053:5
#2 0x55776b2dc6f2 in faulthandler_sigsegv /$HOME/lib/python/cpython/./Modules/faulthandler.c:1069:9
#3 0x55776b01e13d in cfunction_call /$HOME/lib/python/cpython/Objects/methodobject.c:562:18
#4 0x55776af71824 in _PyObject_MakeTpCall /$HOME/lib/python/cpython/Objects/call.c:242:18
#5 0x55776b1a4db0 in _PyEval_EvalFrameDefault /$HOME/lib/python/cpython/Python/generated_cases.c.h:1434:35
#6 0x55776b174755 in _PyEval_Vector /$HOME/lib/python/cpython/Python/ceval.c:1902:12
#7 0x55776b1744d9 in PyEval_EvalCode /$HOME/lib/python/cpython/Python/ceval.c:829:21
#8 0x55776b2956b3 in run_mod /$HOME/lib/python/cpython/Python/pythonrun.c:1436:19
#9 0x55776b292931 in _PyRun_StringFlagsWithName /$HOME/lib/python/cpython/Python/pythonrun.c:1259:15
#10 0x55776b292931 in _PyRun_SimpleStringFlagsWithName /$HOME/lib/python/cpython/Python/pythonrun.c:578:15
#11 0x55776b2d70e5 in pymain_run_command /$HOME/lib/python/cpython/Modules/main.c:255:11
#12 0x55776b2d70e5 in pymain_run_python /$HOME/lib/python/cpython/Modules/main.c:671:21
#13 0x55776b2d70e5 in Py_RunMain /$HOME/lib/python/cpython/Modules/main.c:761:5
#14 0x55776b2d7775 in pymain_main /$HOME/lib/python/cpython/Modules/main.c:791:12
#15 0x55776b2d7802 in Py_BytesMain /$HOME/lib/python/cpython/Modules/main.c:815:12
#16 0x7ff4cf03624c in __libc_start_main (/lib64/libc.so.6+0x3524c) (BuildId: 74f77bf013a66413c77197c121955e029c32d259)
#17 0x55776ae8dd49 in _start /home/abuild/rpmbuild/BUILD/glibc-2.31/csu/../sysdeps/x86_64/start.S:120
UndefinedBehaviorSanitizer can not provide additional info.
SUMMARY: UndefinedBehaviorSanitizer: SEGV (/lib64/libpthread.so.0+0x1676b) (BuildId: 938e42b7e407d175ee3ef9a89c038168101d330c) in raise
==9369==ABORTINGFinally, test_regrtest also fails but the test suite just hangs after crashing (but the UB seems to happen in a test related to XML so it's probably the same thing):
0:01:16 load avg: 26.11 [485/486/5] test_regrtest failed (1 failure) (38.8 sec) -- running (1): test_socket (35.7 sec)
test test_regrtest failed -- Traceback (most recent call last):
File "/$HOME/lib/python/cpython/Lib/test/test_regrtest.py", line 2289, in test_xml
output = self.run_tests(testname, "--junit-xml", filename,
exitcode=EXITCODE_BAD_TEST)
File "/$HOME/lib/python/cpython/Lib/test/test_regrtest.py", line 927, in run_tests
return self.run_python(cmdargs, **kw)
~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^
File "/$HOME/lib/python/cpython/Lib/test/test_regrtest.py", line 777, in run_python
proc = self.run_command(args, **kw)
File "/$HOME/lib/python/cpython/Lib/test/test_regrtest.py", line 768, in run_command
self.fail(msg)
~~~~~~~~~^^^^^
AssertionError: Command ['/$HOME/lib/python/cpython/python', '-X', 'faulthandler', '-I', '-m', 'test', '--testdir=/tmp/test_python_3mwd8cdm/tmpsfkk6fdz', 'test_regrtest_noop51', '--junit-xml', '@test_12395_tmpæ'] failed with exit code 1, but exit code 2 expected!
stdout:
---
Using random seed: 777580575
0:00:00 load avg: 26.11 Run 1 test sequentially in a single process
0:00:00 load avg: 26.11 [1/1] test_regrtest_noop51
test test_regrtest_noop51 failed -- Traceback (most recent call last):
File "/tmp/test_python_3mwd8cdm/tmpsfkk6fdz/test_regrtest_noop51.py", line 8, in test_failed
self.fail()
~~~~~~~~~^^
AssertionError: None
Stdout:
abc ef
Stdout:
abc ef
Modules/expat/xmlparse.c:3162:9: runtime error: call to function expat_start_handler through pointer to incorrect function type 'void (*)(void *, const char *, const char **)'
/$HOME/lib/python/cpython/./Modules/_elementtree.c:3259: note: expat_start_handler defined here
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior Modules/expat/xmlparse.c:3162:9
---
0:01:46 load avg: 16.11 running (1): test_socket (1 min 5 sec)
^C
CPython versions tested on:
CPython main branch
Operating systems tested on:
Linux
Linked PRs
- gh-132097: allow AC to disable fastcall convention to avoid UBSan failures #131605
- gh-132097: remove unnecessary clinic casts to
PyCFunctionand others #131665 - gh-132097: fix runtime UB when calling expat handlers #132265
- gh-132097: skip tests raising an explicit SIGSEV when UB sanitizer is on #132398
- gh-132097: use a macro for semantically casting function pointers #132406