py/objint_longlong: Fix left shift of negative values. #17734

projectgus · 2025-07-22T01:22:44Z

Summary

Closes #17702, thanks to @jepler for running these additional checks.

The overflow check for 'longlong' representation left shift added in py: Fixes and test coverage for 64-bit big integer representations. #16953 was wrong, and depended on UB. Rewrote to be the same as the small int left shift overflow check.
Skip the viper "boundary" store tests if arbitrary integer support is disabled, these tests depend on generating very long integers and then truncating them.

Note: First version of this PR enabled UBSan for longlong tests, but this surfaced another issue so has been spun out to #17735.

Testing

Re-ran unix port (with UBSan), observed tests pass and no errors.

Trade-offs and Alternatives

The viper "boundary" store tests could probably be rewritten to also work withouth arbitrary precision integers, but relatively fiddly to do so.

projectgus · 2025-07-22T01:28:16Z

Hmm, so running the longlong tests with UBSan I get two failures like this:

 --- /home/runner/work/micropython/micropython/tests/results/extmod_vfs_rom.py.exp	2025-07-22 01:24:29.075299604 +0000
+++ /home/runner/work/micropython/micropython/tests/results/extmod_vfs_rom.py.out	2025-07-22 01:24:29.075299604 +0000
@@ -1 +1,13 @@
-SKIP
+Traceback (most recent call last):
+  File "/home/runner/work/micropython/micropython/tests/extmod/vfs_rom.py", line 489, in <module>
+  File "/home/runner/work/micropython/micropython/lib/micropython-lib/python-stdlib/unittest/unittest/__init__.py", line 464, in main
+  File "/home/runner/work/micropython/micropython/lib/micropython-lib/python-stdlib/unittest/unittest/__init__.py", line 269, in run
+  File "/home/runner/work/micropython/micropython/lib/micropython-lib/python-stdlib/unittest/unittest/__init__.py", line 254, in run
+  File "/home/runner/work/micropython/micropython/lib/micropython-lib/python-stdlib/unittest/unittest/__init__.py", line 420, in _run_suite
+  File "/home/runner/work/micropython/micropython/tests/extmod/vfs_rom.py", line 166, in setUpClass
+  File "/home/runner/work/micropython/micropython/tests/extmod/vfs_rom.py", line 145, in make_romfs
+  File "/home/runner/work/micropython/micropython/tests/extmod/vfs_rom.py", line 136, in _make_romfs
+  File "/home/runner/work/micropython/micropython/tests/extmod/vfs_rom.py", line 115, in mkfile
+  File "/home/runner/work/micropython/micropython/tests/extmod/vfs_rom.py", line 80, in _encode_uint
+RuntimeError: maximum recursion depth exceeded

(I don't see these locally, but I have a different gcc locally.)

This looks a lot like the problem @koxudaxi was describing in #17649...

codecov · 2025-07-22T01:30:07Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.38%. Comparing base (3185bb5) to head (096ff8b).
Report is 2 commits behind head on master.

Additional details and impacted files

@@           Coverage Diff           @@
##           master   #17734   +/-   ##
=======================================
  Coverage   98.38%   98.38%           
=======================================
  Files         171      171           
  Lines       22239    22239           
=======================================
  Hits        21880    21880           
  Misses        359      359

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

github-actions · 2025-07-22T01:31:44Z

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

projectgus · 2025-07-22T01:54:50Z

Have pulled out the CI change to its own PR.

py/objint_longlong.c

tests/micropython/viper_ptr16_store_boundary.py

yoctopuce · 2025-07-22T08:04:53Z

Another option to fix the overflow check would be:

if ((int)rhs_val > 62) {
    overflow = 1;
} else {
    rhs_val = 1ll << (int)rhs_val;
    overflow = mp_mul_ll_overflow(lhs_val, rhs_val, &result);                
}

I am not sure which one would generate the shortest code, given that the compiler might be able to merge the call to mp_mul_ll_overflow with the code in MP_BINARY_OP_MULTIPLY

projectgus · 2025-07-22T23:55:50Z

Another option to fix the overflow check would be:
if ((int)rhs_val > 62) {
    overflow = 1;
} else {
    rhs_val = 1ll << (int)rhs_val;
    overflow = mp_mul_ll_overflow(lhs_val, rhs_val, &result);                
}
I am not sure which one would generate the shortest code, given that the compiler might be able to merge the call to mp_mul_ll_overflow with the code in MP_BINARY_OP_MULTIPLY

Neat idea! I tried it, but unfortunately the code size is still a little bigger:

(Building the Zephyr qemu cortex-m3 board as west build -b qemu_cortex_m3 ~/micropython/ports/zephyr -- -DCONF_FILE=prj_minimal.conf)

master - 114068
this PR - 114120
calling mp_mul_overflow - 114192

I'm guessing either the calls can't be merged, or marshalling the values is still more work than doing the comparisons required in this version.

Previous comment was wrong, left shifting a negative value is UB in C. Use the same approach as small int shifts (from runtime.c). Signed-off-by: Angus Gratton <angus@redyak.com.au>

These tests all depend on generating arbitrarily long (>64-bit) integers. It would be possible to have these tests work in this case I think, as the results are always masked to shorter values. But quite fiddly. So just rename them so they are automatically skipped if the target doesn't have big int support. This work was funded through GitHub Sponsors. Signed-off-by: Angus Gratton <angus@redyak.com.au>

projectgus added this to the release-1.26.0 milestone Jul 22, 2025

projectgus added the py-core Relates to py/ directory in source label Jul 22, 2025

projectgus force-pushed the bugfix/longlong_shift branch from d2517f1 to 09c6e9b Compare July 22, 2025 01:48

projectgus changed the title ~~py/objint_longlong: Fix left shift of negative values, add sanitizer to CI build.~~ py/objint_longlong: Fix left shift of negative values. Jul 22, 2025

projectgus mentioned this pull request Jul 22, 2025

ci: Enable UBSan for 'longlong' builds in CI, add stack margin for sanitizer builds. #17735

Open

dpgeorge reviewed Jul 22, 2025

View reviewed changes

py/objint_longlong.c Outdated Show resolved Hide resolved

dpgeorge reviewed Jul 22, 2025

View reviewed changes

tests/micropython/viper_ptr16_store_boundary.py Outdated Show resolved Hide resolved

projectgus force-pushed the bugfix/longlong_shift branch from 09c6e9b to 977f0a3 Compare July 22, 2025 23:56

dpgeorge approved these changes Jul 23, 2025

View reviewed changes

projectgus force-pushed the bugfix/longlong_shift branch from 977f0a3 to e31ba59 Compare July 23, 2025 04:53

jepler mentioned this pull request Jul 23, 2025

core: Add MICROPY_USE_GCC_MUL_OVERFLOW_INTRINSIC. #17754

Draft

projectgus added 2 commits July 24, 2025 15:50

py/objint_longlong: Fix left shift of negative values.

5d9ef6b

Previous comment was wrong, left shifting a negative value is UB in C. Use the same approach as small int shifts (from runtime.c). Signed-off-by: Angus Gratton <angus@redyak.com.au>

dpgeorge force-pushed the bugfix/longlong_shift branch from e31ba59 to 096ff8b Compare July 24, 2025 05:54

dpgeorge merged commit 096ff8b into micropython:master Jul 24, 2025
70 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Uh oh!

py/objint_longlong: Fix left shift of negative values. #17734

py/objint_longlong: Fix left shift of negative values. #17734

Uh oh!

projectgus commented Jul 22, 2025 •

edited

Loading

Uh oh!

projectgus commented Jul 22, 2025

Uh oh!

codecov bot commented Jul 22, 2025 •

edited

Loading

Uh oh!

github-actions bot commented Jul 22, 2025

Uh oh!

projectgus commented Jul 22, 2025

Uh oh!

Uh oh!

Uh oh!

yoctopuce commented Jul 22, 2025 •

edited

Loading

Uh oh!

projectgus commented Jul 22, 2025

Uh oh!

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Uh oh!

py/objint_longlong: Fix left shift of negative values. #17734

py/objint_longlong: Fix left shift of negative values. #17734

Uh oh!

Conversation

projectgus commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Testing

Trade-offs and Alternatives

Uh oh!

projectgus commented Jul 22, 2025

Uh oh!

codecov bot commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

github-actions bot commented Jul 22, 2025

Uh oh!

projectgus commented Jul 22, 2025

Uh oh!

Uh oh!

Uh oh!

yoctopuce commented Jul 22, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

projectgus commented Jul 22, 2025

Uh oh!

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

projectgus commented Jul 22, 2025 •

edited

Loading

codecov bot commented Jul 22, 2025 •

edited

Loading

yoctopuce commented Jul 22, 2025 •

edited

Loading