Skip to content

py/objint_longlong: Fix left shift of negative values. #17734

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Open
wants to merge 2 commits into
base: master
Choose a base branch
from
Open

py/objint_longlong: Fix left shift of negative values. #17734

wants to merge 2 commits into from
+5 −5

Conversation

projectgus
Copy link
Contributor

@projectgus projectgus commented Jul 22, 2025 •
edited
Loading

Summary

Closes #17702, thanks to @jepler for running these additional checks.

  • The overflow check for 'longlong' representation left shift added in py: Fixes and test coverage for 64-bit big integer representations. #16953 was wrong, and depended on UB. Rewrote to be the same as the small int left shift overflow check.
  • Skip the viper "boundary" store tests if arbitrary integer support is disabled, these tests depend on generating very long integers and then truncating them.

Note: First version of this PR enabled UBSan for longlong tests, but this surfaced another issue so has been spun out to #17735.

Testing

  • Re-ran unix port (with UBSan), observed tests pass and no errors.

Trade-offs and Alternatives

  • The viper "boundary" store tests could probably be rewritten to also work withouth arbitrary precision integers, but relatively fiddly to do so.

@projectgus projectgus added this to the release-1.26.0 milestone Jul 22, 2025
@projectgus projectgus added the py-core Relates to py/ directory in source label Jul 22, 2025
@projectgus
Copy link
Contributor Author

Hmm, so running the longlong tests with UBSan I get two failures like this:

 --- /home/runner/work/micropython/micropython/tests/results/extmod_vfs_rom.py.exp	2025-07-22 01:24:29.075299604 +0000
+++ /home/runner/work/micropython/micropython/tests/results/extmod_vfs_rom.py.out	2025-07-22 01:24:29.075299604 +0000
@@ -1 +1,13 @@
-SKIP
+Traceback (most recent call last):
+  File "/home/runner/work/micropython/micropython/tests/extmod/vfs_rom.py", line 489, in <module>
+  File "/home/runner/work/micropython/micropython/lib/micropython-lib/python-stdlib/unittest/unittest/__init__.py", line 464, in main
+  File "/home/runner/work/micropython/micropython/lib/micropython-lib/python-stdlib/unittest/unittest/__init__.py", line 269, in run
+  File "/home/runner/work/micropython/micropython/lib/micropython-lib/python-stdlib/unittest/unittest/__init__.py", line 254, in run
+  File "/home/runner/work/micropython/micropython/lib/micropython-lib/python-stdlib/unittest/unittest/__init__.py", line 420, in _run_suite
+  File "/home/runner/work/micropython/micropython/tests/extmod/vfs_rom.py", line 166, in setUpClass
+  File "/home/runner/work/micropython/micropython/tests/extmod/vfs_rom.py", line 145, in make_romfs
+  File "/home/runner/work/micropython/micropython/tests/extmod/vfs_rom.py", line 136, in _make_romfs
+  File "/home/runner/work/micropython/micropython/tests/extmod/vfs_rom.py", line 115, in mkfile
+  File "/home/runner/work/micropython/micropython/tests/extmod/vfs_rom.py", line 80, in _encode_uint
+RuntimeError: maximum recursion depth exceeded

(I don't see these locally, but I have a different gcc locally.)

This looks a lot like the problem @koxudaxi was describing in #17649...

@codecov Codecov
Copy link

codecov bot commented Jul 22, 2025 •
edited
Loading

Codecov Report

All modified and coverable lines are covered by tests ✅

Project coverage is 98.37%. Comparing base (e993f53) to head (e31ba59).
Report is 20 commits behind head on master.

Additional details and impacted files 
@@            Coverage Diff             @@
##           master   #17734      +/-   ##
==========================================
- Coverage   98.41%   98.37%   -0.04%     
==========================================
  Files         171      171              
  Lines       22210    22210              
==========================================
- Hits        21857    21849       -8     
- Misses        353      361       +8

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions GitHub Actions
Copy link

Code size report:

   bare-arm:    +0 +0.000% 
minimal x86:    +0 +0.000% 
   unix x64:    +0 +0.000% standard
      stm32:    +0 +0.000% PYBV10
     mimxrt:    +0 +0.000% TEENSY40
        rp2:    +0 +0.000% RPI_PICO_W
       samd:    +0 +0.000% ADAFRUIT_ITSYBITSY_M4_EXPRESS
  qemu rv32:    +0 +0.000% VIRT_RV32

@projectgus projectgus force-pushed the bugfix/longlong_shift branch from d2517f1 to 09c6e9b Compare July 22, 2025 01:48
@projectgus projectgus changed the title py/objint_longlong: Fix left shift of negative values, add sanitizer to CI build. py/objint_longlong: Fix left shift of negative values. Jul 22, 2025
@projectgus projectgus mentioned this pull request Jul 22, 2025
Open
@projectgus
Copy link
Contributor Author

Have pulled out the CI change to its own PR.

dpgeorge
dpgeorge reviewed Jul 22, 2025
View reviewed changes
dpgeorge
dpgeorge reviewed Jul 22, 2025
View reviewed changes
@yoctopuce
Copy link
Contributor

yoctopuce commented Jul 22, 2025 •
edited
Loading

Another option to fix the overflow check would be:

if ((int)rhs_val > 62) {
    overflow = 1;
} else {
    rhs_val = 1ll << (int)rhs_val;
    overflow = mp_mul_ll_overflow(lhs_val, rhs_val, &result);                
}

I am not sure which one would generate the shortest code, given that the compiler might be able to merge the call to mp_mul_ll_overflow with the code in MP_BINARY_OP_MULTIPLY

@projectgus
Copy link
Contributor Author

Another option to fix the overflow check would be:

if ((int)rhs_val > 62) {
    overflow = 1;
} else {
    rhs_val = 1ll << (int)rhs_val;
    overflow = mp_mul_ll_overflow(lhs_val, rhs_val, &result);                
}

I am not sure which one would generate the shortest code, given that the compiler might be able to merge the call to mp_mul_ll_overflow with the code in MP_BINARY_OP_MULTIPLY

Neat idea! I tried it, but unfortunately the code size is still a little bigger:

(Building the Zephyr qemu cortex-m3 board as west build -b qemu_cortex_m3 ~/micropython/ports/zephyr -- -DCONF_FILE=prj_minimal.conf)

  • master - 114068
  • this PR - 114120
  • calling mp_mul_overflow - 114192

I'm guessing either the calls can't be merged, or marshalling the values is still more work than doing the comparisons required in this version.

@projectgus projectgus force-pushed the bugfix/longlong_shift branch from 09c6e9b to 977f0a3 Compare July 22, 2025 23:56
@projectgus
py/objint_longlong: Fix left shift of negative values.
cdcc4ab 
Previous comment was wrong, left shifting a negative value is UB in C. Use
the same approach as small int shifts (from runtime.c).

Signed-off-by: Angus Gratton <angus@redyak.com.au>
@projectgus
tests/micropython/viper: Boundary value tests depend on 'intbig'.
e31ba59 
These tests all depend on generating arbitrarily long (>64-bit)
integers.

It would be possible to have these tests work in this case I think,
as the results are always masked to shorter values. But quite fiddly.

This work was funded through GitHub Sponsors.

Signed-off-by: Angus Gratton <angus@redyak.com.au>
@projectgus projectgus force-pushed the bugfix/longlong_shift branch from 977f0a3 to e31ba59 Compare July 23, 2025 04:53
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@dpgeorge dpgeorge dpgeorge approved these changes

Assignees
No one assigned
Labels
py-core Relates to py/ directory in source
Projects
None yet
Milestone
release-1.26.0
Development

Successfully merging this pull request may close these issues.

-fsanitizer=undefined diagnostics with longlong
3 participants
@projectgus @yoctopuce @dpgeorge
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy