gc.c: Ensure a gap of one byte before the finaliser table. #7722

jepler · 2021-08-27T14:29:30Z

.. or, for !MICROPY_ENABLE_FINALISER, before the first block of the pool.

I tested this with a version of #7716 (comment) and there were no problems detected. ~~I haven't tested it on our real HW yet.~~ it also fixes the original problem on our real hw

When you wish to use the valgrind memory analysis tool on micropython, you can arrange to define MICROPY_DEBUG_VALGRIND to enable use of special valgrind macros. For now, this only fixes `gc_get_ptr` so that it never emits the diagnostic "Conditional jump or move depends on uninitialised value(s)". Signed-off-by: Jeff Epler <jepler@gmail.com>

.. or, for !MICROPY_ENABLE_FINALISER, before the first block of the pool. Closes: adafruit#5021 Closes: micropython#7116 Signed-off-by: Jeff Epler <jepler@gmail.com>

codecov-commenter · 2021-08-27T14:40:51Z

Codecov Report

Merging #7722 (c9bd52f) into master (8c4ba57) will decrease coverage by 0.02%.
The diff coverage is 100.00%.

@@            Coverage Diff             @@
##           master    #7722      +/-   ##
==========================================
- Coverage   98.27%   98.24%   -0.03%     
==========================================
  Files         154      154              
  Lines       20077    20042      -35     
==========================================
- Hits        19730    19690      -40     
- Misses        347      352       +5

Impacted Files	Coverage Δ
py/gc.c	`99.07% <100.00%> (ø)`
py/obj.c	`96.82% <0.00%> (-1.20%)`	⬇️
py/bc.c	`88.65% <0.00%> (-1.04%)`	⬇️
py/objlist.c	`99.23% <0.00%> (-0.39%)`	⬇️
py/compile.c	`99.93% <0.00%> (-0.01%)`	⬇️
py/objtype.c	`100.00% <0.00%> (ø)`
py/emitnative.c	`99.26% <0.00%> (+<0.01%)`	⬆️
py/parse.c	`99.20% <0.00%> (+0.02%)`	⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 8c4ba57...c9bd52f. Read the comment docs.

jepler · 2021-08-27T14:55:16Z

Can someone let me know what's going on with this CI failure? I didn't immediately make sense of it.

The team named '~pybricks' has no PPA named 'ubuntu/ppa'

dlech · 2021-08-27T15:28:49Z

The team named '~pybricks' has no PPA named 'ubuntu/ppa'

This is a PPA to get a backported version of uncrustify for Ubuntu 20.04. It has been working for quite a while, so maybe there is an issue with the https://launchpad.net site where this is hosted?

dpgeorge

Is it at all possible to make a test for this?

py/gc.c

jepler · 2021-09-15T16:19:29Z

I've updated this PR:

the 1-byte gap is now named
it fixes a second bug that occurred when the gap byte was not initialized (Clear the buffer byte between ATB and FTB adafruit/circuitpython#5348)
it unifies two memset calls into one, slight flash savings
an assertion and a test were added. Before this PR, the test would hit the assertion error in the unix coverage build

jepler · 2021-09-15T16:20:07Z

CI is encountering difficulties:

Run source tools/ci.sh && ci_code_formatting_setup
'Error reading https://launchpad.net/api/devel/~pybricks/+archive/ubuntu/ppa?ws.op=getSigningKeyData (4 tries): Internal Server Error'
Error: Process completed with exit code 1.

This test must be separate from the "regular" coverage test, as it destroys the gc heap, and must exit via exit() rather than returning to python code. This test fails due to adafruit#7716. Signed-off-by: Jeff Epler <jepler@gmail.com>

jepler · 2021-09-15T16:35:15Z

.. and the Windows (appveyor) build is failing with almost every test saying "FATAL: uncaught NLR", which could be some new problem I've introduced even though no effect on nlr was intended.

Signed-off-by: Jeff Epler <jepler@gmail.com>

dpgeorge · 2022-11-25T04:28:25Z

I have rebased this here: https://github.com/dpgeorge/micropython/tree/py-gc-ensure-one-byte-after-atb

dpgeorge · 2022-11-25T04:28:58Z

py/gc.c

@@ -118,7 +128,8 @@ void gc_init(void *start, void *end) {
    // => T = A * (1 + BLOCKS_PER_ATB / BLOCKS_PER_FTB + BLOCKS_PER_ATB * BYTES_PER_BLOCK)
    size_t total_byte_len = (byte *)end - (byte *)start;
    #if MICROPY_ENABLE_FINALISER
-    MP_STATE_MEM(gc_alloc_table_byte_len) = total_byte_len * MP_BITS_PER_BYTE / (MP_BITS_PER_BYTE + MP_BITS_PER_BYTE * BLOCKS_PER_ATB / BLOCKS_PER_FTB + MP_BITS_PER_BYTE * BLOCKS_PER_ATB * BYTES_PER_BLOCK);
+    size_t gc_alloc_table_byte_len = (total_byte_len - ALLOC_TABLE_GAP_BYTE) * MP_BITS_PER_BYTE / (MP_BITS_PER_BYTE + MP_BITS_PER_BYTE * BLOCKS_PER_ATB / BLOCKS_PER_FTB + MP_BITS_PER_BYTE * BLOCKS_PER_ATB * BYTES_PER_BLOCK);
+    MP_STATE_MEM(gc_alloc_table_byte_len) = gc_alloc_table_byte_len;
    #else
    MP_STATE_MEM(gc_alloc_table_byte_len) = total_byte_len / (1 + MP_BITS_PER_BYTE / 2 * BYTES_PER_BLOCK);


I think this should be (total_byte_len - ALLOCA_TABLE_GAP_BYTE) ...

I fixed this during merge.

dpgeorge · 2022-11-25T04:29:29Z

ports/unix/coverage.c

+        heap_size++;
+    }
+    mp_printf(&mp_plat_print, "# end coverage.c gc test\n");
+    exit(0);


This is quite brutal, to exit so abruptly! I wonder if there's a better way to do it...

I don't think this is a good way to do it because it prevents coverage/testing of the tear down and exiting of the executable.

Two options I can think of:

save and then restore the mp_state_mem_t struct in mp_state_ctx_t at the start/end of this function, which should restore the original heap on exit of this function

run this code at the start or end of main() (probably at the start, just before gc_init()).

I did option (1) during merge.

dpgeorge · 2022-12-08T04:16:27Z

I rebased this, fixed up things as described above, and merged in 8407159 through d75ff42

dpgeorge · 2022-12-08T04:17:20Z

Thanks @jepler , this is an important fix.

jepler · 2023-01-01T15:54:18Z

Thanks! And thank you for fixing the test to be more appropriate

jepler added 3 commits August 27, 2021 09:23

gc.c: Ensure a gap of one byte before the finaliser table.

5a162eb

.. or, for !MICROPY_ENABLE_FINALISER, before the first block of the pool. Closes: adafruit#5021 Closes: micropython#7116 Signed-off-by: Jeff Epler <jepler@gmail.com>

Merge remote-tracking branch 'origin/master' into pool-fix-micropython

82eec03

jepler mentioned this pull request Aug 27, 2021

gc.c: Ensure a gap of one byte before the finaliser table. adafruit/circuitpython#5245

Merged

dpgeorge added the py-core Relates to py/ directory in source label Aug 28, 2021

dpgeorge reviewed Aug 28, 2021

View reviewed changes

py/gc.c Outdated Show resolved Hide resolved

py/gc.c Outdated Show resolved Hide resolved

tests: Add extra gc coverage test.

ddeb021

This test must be separate from the "regular" coverage test, as it destroys the gc heap, and must exit via exit() rather than returning to python code. This test fails due to adafruit#7716. Signed-off-by: Jeff Epler <jepler@gmail.com>

jepler force-pushed the pool-fix-micropython branch from 078f6fb to b776383 Compare September 15, 2021 16:21

jepler force-pushed the pool-fix-micropython branch 2 times, most recently from 794c4ed to 347046d Compare September 15, 2021 18:51

gc: Name the gap byte and ensure it is zeroed.

1ea027e

Signed-off-by: Jeff Epler <jepler@gmail.com>

jepler force-pushed the pool-fix-micropython branch from 347046d to 1ea027e Compare September 15, 2021 19:05

Merge branch 'test-for-7716' into pool-fix-micropython

c9bd52f

jepler requested a review from dpgeorge September 15, 2021 20:05

dpgeorge reviewed Nov 25, 2022

View reviewed changes

dpgeorge closed this Dec 8, 2022

projectgus mentioned this pull request Mar 27, 2024

gc: Support valgrind tracking of Python heap allocations #14196

Draft

2 tasks

Uh oh!

gc.c: Ensure a gap of one byte before the finaliser table. #7722

gc.c: Ensure a gap of one byte before the finaliser table. #7722

Uh oh!

Conversation

jepler commented Aug 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

codecov-commenter commented Aug 27, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

jepler commented Aug 27, 2021

Uh oh!

dlech commented Aug 27, 2021

Uh oh!

dpgeorge left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

jepler commented Sep 15, 2021

Uh oh!

jepler commented Sep 15, 2021

Uh oh!

jepler commented Sep 15, 2021

Uh oh!

dpgeorge commented Nov 25, 2022

Uh oh!

dpgeorge Nov 25, 2022

Choose a reason for hiding this comment

Uh oh!

dpgeorge Dec 8, 2022

Choose a reason for hiding this comment

Uh oh!

dpgeorge Nov 25, 2022

Choose a reason for hiding this comment

Uh oh!

dpgeorge Nov 25, 2022

Choose a reason for hiding this comment

Uh oh!

dpgeorge Dec 8, 2022

Choose a reason for hiding this comment

Uh oh!

dpgeorge commented Dec 8, 2022

Uh oh!

dpgeorge commented Dec 8, 2022

Uh oh!

jepler commented Jan 1, 2023

Uh oh!

Uh oh!

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

jepler commented Aug 27, 2021 •

edited

Loading

codecov-commenter commented Aug 27, 2021 •

edited

Loading