patchcheck.py raises SystemError on main #105820
Description
Bug report
Following #105801, running python Tools/patchcheck/patchcheck.py now results in SystemError being raised.
To reproduce
- Make any change to a
.pyfile, e.g.:
--- a/Lib/test/test_fstring.py
+++ b/Lib/test/test_fstring.py
@@ -19,6 +19,7 @@
a_global = 'global variable'
+- Run
python Tools/patchcheck/patchcheck.py:
C:\Users\alexw\coding\cpython>python Tools/patchcheck/patchcheck.py
Running Release|x64 interpreter...
Getting base branch for PR ... upstream/main
Getting the list of files that have been added/changed ... 1 file
Fixing Python file whitespace ... Traceback (most recent call last):
File "C:\Users\alexw\coding\cpython\Tools\patchcheck\patchcheck.py", line 325, in <module>
main()
File "C:\Users\alexw\coding\cpython\Tools\patchcheck\patchcheck.py", line 293, in main
normalize_whitespace(python_files)
File "C:\Users\alexw\coding\cpython\Tools\patchcheck\patchcheck.py", line 35, in call_fxn
result = fxn(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^
File "C:\Users\alexw\coding\cpython\Tools\patchcheck\patchcheck.py", line 188, in normalize_whitespace
and reindent.check(os.path.join(SRCDIR, path))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "C:\Users\alexw\coding\cpython\Tools\patchcheck\reindent.py", line 138, in check
if r.run():
^^^^^^^
File "C:\Users\alexw\coding\cpython\Tools\patchcheck\reindent.py", line 203, in run
for _token in tokens:
File "C:\Users\alexw\coding\cpython\Lib\tokenize.py", line 537, in _generate_tokens_from_c_tokenizer
for info in it:
SystemError: Negative size passed to PyUnicode_New
This failure was visible on the Azure Pipelines job on the PR FYI: https://dev.azure.com/Python/cpython/_build/results?buildId=130389&view=logs&j=256d7e09-002a-52d7-8661-29ee3960640e&t=3d7276d3-4e8d-5309-55ad-fb0b172d9925
Your environment
Python 3.13.0a0 (heads/main:c5111aec2b, Jun 15 2023, 10:53:37) [MSC v.1932 64 bit (AMD64)] on win32