Skip to content

gh-132732: Automatically constant evaluate pure operations #132733

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 58 commits into from
Jun 27, 2025
Merged
Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
58 commits
Select commit Hold shift + click to select a range
1ffbb6b
Automatically constant evaluate pure operations
Fidget-Spinner Apr 19, 2025
691084d
📜🤖 Added by blurb_it.
blurb-it[bot] Apr 19, 2025
b89e4dc
Fix tests
Fidget-Spinner Apr 19, 2025
0959918
Merge branch 'pure' of github.com:Fidget-Spinner/cpython into pure
Fidget-Spinner Apr 19, 2025
2541683
Merge remote-tracking branch 'upstream/main' into pure
Fidget-Spinner Apr 24, 2025
d5b2208
Apply review suggestions
Fidget-Spinner Apr 25, 2025
71ced86
reduce diff
Fidget-Spinner Apr 25, 2025
a10d5a1
Merge remote-tracking branch 'upstream/main' into pure
Fidget-Spinner May 6, 2025
d22f165
Update pycore_opcode_metadata.h
Fidget-Spinner May 6, 2025
8ae38c7
Apply changes from code review
Fidget-Spinner May 10, 2025
712a810
Merge remote-tracking branch 'upstream/main' into pure
Fidget-Spinner May 10, 2025
dc2d922
Address review, add test
Fidget-Spinner May 10, 2025
f3f2a69
Add more tests
Fidget-Spinner May 12, 2025
53ce10f
Fix tests
Fidget-Spinner May 12, 2025
17634a8
Push fix noticed by Mark and Brandt
Fidget-Spinner May 19, 2025
4937c2f
Merge remote-tracking branch 'upstream/main' into pure
Fidget-Spinner May 19, 2025
ae08b79
Merge remote-tracking branch 'upstream/main' into pure
Fidget-Spinner May 21, 2025
c0c6600
remove pure from _POP_CALL_TWO_LOAD_CONST_INLINE_BORROW
Fidget-Spinner May 21, 2025
6bdd3f9
Merge remote-tracking branch 'upstream/main' into pure
Fidget-Spinner May 21, 2025
de8e170
use upstream changes for stackref
Fidget-Spinner May 21, 2025
c2f8e22
remove unused comment
Fidget-Spinner May 21, 2025
ac7e343
Address review
Fidget-Spinner May 22, 2025
05b822f
fix test
Fidget-Spinner May 22, 2025
b4c2e93
fix negative refcount
Fidget-Spinner May 23, 2025
d229f57
Merge remote-tracking branch 'upstream/main' into pure
Fidget-Spinner May 23, 2025
c4aae6c
Merge remote-tracking branch 'upstream/main' into pure
Fidget-Spinner May 27, 2025
8552182
Use `REPLACE_OPCODE_IF_EVALUTES_PURE`
Fidget-Spinner May 28, 2025
703dfc9
Fix test, move is_abstract to subclass attribute
Fidget-Spinner May 28, 2025
b278734
fix linter/mypy
Fidget-Spinner May 28, 2025
73a8b00
remove whitespace
Fidget-Spinner May 28, 2025
4116a31
Remove PyDict_Type
Fidget-Spinner May 28, 2025
548b67c
add bool type
Fidget-Spinner May 28, 2025
74a0208
reduce diff
Fidget-Spinner Jun 6, 2025
6a5dc12
Grab identifiers from REPLACE_OPCODE_IF_EVALUATES_PURE
Fidget-Spinner Jun 9, 2025
3896775
Merge remote-tracking branch 'upstream/main' into pure
Fidget-Spinner Jun 9, 2025
507c80a
Use replacer
Fidget-Spinner Jun 13, 2025
dc68b45
reduce diff
Fidget-Spinner Jun 13, 2025
41a271c
Merge remote-tracking branch 'upstream/main' into pure
Fidget-Spinner Jun 13, 2025
e88b71a
Update optimizer_generator.py
Fidget-Spinner Jun 13, 2025
866510f
Address review (add long functions to allowlist)
Fidget-Spinner Jun 13, 2025
01be0c6
fix mypy
Fidget-Spinner Jun 13, 2025
fce79a6
Merge remote-tracking branch 'upstream/main' into pure
Fidget-Spinner Jun 13, 2025
9bef4a4
revert changes for add,multiply,sub int
Fidget-Spinner Jun 13, 2025
1f76e2c
Fix tests
Fidget-Spinner Jun 13, 2025
efb9888
Merge remote-tracking branch 'upstream/main' into pure
Fidget-Spinner Jun 16, 2025
029c92b
Add deopt and error_if, and tests
Fidget-Spinner Jun 16, 2025
738d20e
remove is_abstract
Fidget-Spinner Jun 16, 2025
202af76
Merge remote-tracking branch 'upstream/main' into pure
Fidget-Spinner Jun 18, 2025
4c0c52c
fix tests
Fidget-Spinner Jun 18, 2025
e24e9a3
Fix test_opt
Fidget-Spinner Jun 18, 2025
f362866
Merge remote-tracking branch 'upstream/main' into pure
Fidget-Spinner Jun 20, 2025
d54a6f2
Update optimizer_bytecodes.c
Fidget-Spinner Jun 20, 2025
2644cb9
Address review
Fidget-Spinner Jun 20, 2025
634ed26
Add for _BINARY_OP
Fidget-Spinner Jun 20, 2025
0e9ce84
Address review
Fidget-Spinner Jun 27, 2025
f2fc3fc
reduce diff
Fidget-Spinner Jun 27, 2025
2408fb0
fix mypy
Fidget-Spinner Jun 27, 2025
e40cb06
fix wrong error message
Fidget-Spinner Jun 27, 2025
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Address review, add test
  • Loading branch information
Fidget-Spinner committed May 10, 2025
commit dc2d92254da8d3b26fddd4cbb8710cf5f2b96e25
35 changes: 35 additions & 0 deletions Lib/test/test_generated_cases.py
Original file line number Diff line number Diff line change
Expand Up @@ -2253,5 +2253,40 @@ def test_validate_uop_unused_size_mismatch(self):
"Inputs must have equal sizes"):
self.run_cases_test(input, input2, output)

def test_pure_uop_body_copied_in(self):
input = """
pure op(OP, (foo -- res)) {
res = body(foo);
}
"""
input2 = """
op(OP, (foo -- res)) {
res = sym_new_unknown(ctx);
}
"""
output = """
case OP: {
JitOptSymbol *res;
if (
sym_is_const(ctx, foo)
) {
JitOptSymbol *foo_sym = foo;
_PyStackRef foo = sym_get_const_as_stackref(ctx, foo_sym);
_PyStackRef res_stackref;
/* Start of pure uop copied from bytecodes for constant evaluation */
res_stackref = body(foo);
/* End of pure uop copied from bytecodes for constant evaluation */
res = sym_new_const_steal(ctx, PyStackRef_AsPyObjectBorrow(res_stackref));
stack_pointer[-1] = res;
}
else {
res = sym_new_unknown(ctx);
stack_pointer[-1] = res;
}
break;
}
"""
self.run_cases_test(input, input2, output)

if __name__ == "__main__":
unittest.main()
16 changes: 16 additions & 0 deletions Python/optimizer_cases.c.h

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

13 changes: 0 additions & 13 deletions Tools/cases_generator/generators_common.py
Original file line number Diff line number Diff line change
Expand Up @@ -75,19 +75,6 @@ def write_header(
"""
)

def skip_to(tkn_iter: TokenIterator, end: str) -> Token:
tkn = None
parens = 0
for tkn in tkn_iter:
if tkn.kind == end and parens == 0:
return tkn
if tkn.kind == "LPAREN":
parens += 1
if tkn.kind == "RPAREN":
parens -= 1
assert tkn is not None
return tkn

def emit_to(out: CWriter, tkn_iter: TokenIterator, end: str) -> Token:
parens = 0
for tkn in tkn_iter:
Expand Down
19 changes: 10 additions & 9 deletions Tools/cases_generator/optimizer_generator.py
Original file line number Diff line number Diff line change
Expand Up @@ -22,11 +22,9 @@
write_header,
Emitter,
TokenIterator,
emit_to,
skip_to,
)
from cwriter import CWriter
from typing import TextIO, Callable
from typing import TextIO
from lexer import Token
from stack import Local, Stack, StackError, Storage

Expand Down Expand Up @@ -87,7 +85,7 @@ def stackref_type_name(var: StackItem) -> str:
assert False, "Unsafe to convert a symbol to an array-like StackRef."
if var.type:
return var.type
return f"_PyStackRef "
return "_PyStackRef "

def declare_variables(uop: Uop, out: CWriter, skip_inputs: bool) -> None:
variables = {"unused"}
Expand Down Expand Up @@ -210,17 +208,19 @@ def write_uop_pure_evaluation_region_header(
# No reference management of outputs needed.
for var in storage.outputs:
var.in_local = True
emitter.emit_tokens(uop, storage, None, False, is_abstract=True)
emitter.emit("/* Start of pure uop copied from bytecodes for constant evaluation */\n")
emitter.emit_tokens(uop, storage, inst=None, emit_braces=False, is_abstract=True)
out.start_line()
emitter.emit("/* End of pure uop copied from bytecodes for constant evaluation */\n")
# Finally, assign back the output stackrefs to symbolics.
for outp in uop.stack.outputs:
# All new stackrefs are created from new references.
# That's how the stackref contract works.
if not outp.peek:
out.emit(f"{outp.name} = sym_new_const_steal(ctx, PyStackRef_AsPyObjectBorrow({outp.name}_stackref));\n")
emitter.emit(f"{outp.name} = sym_new_const_steal(ctx, PyStackRef_AsPyObjectBorrow({outp.name}_stackref));\n")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It may just be a week of conference sleep schedule, but my brain hurts trying to reason about the refcounting here. Why are we stealing a borrow? Shouldn't we be stealing a steal, or borrowing a borrow? Currently:

  • If the tag bit is unset on the stackref, stealing a borrow will leave the refcount on the object unchanged and the tag bit unset. When the symbol is cleared after optimizing, the refcount on the object will be one less, which is correct.
  • If the tag bit is set on the stackref, stealing a borrow will leave the refcount on the object itself unchanged and the tag bit still set. When the symbol is cleared after optimizing, the refcount on the object will be one less, which seems incorrect.

(I haven't looked at the peek code yet.)

Another option, that I might like better, is making all of our constant symbols use stackrefs under-the-hood. Then we could avoid refcounting entirely. But that's a bigger change that could happen later if needed.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The first thing is the tag bit is always unsets for new objects, at least for pure operations. If it's set, then the thing is likely immortal anyways, which we can ignore reference count semantics for. This ignores the FT build's deferred references. sym_new_const increfs. sym_new_const_steal doesn't incref. The key part is that all results of pure operations in bytecodes.c are always new references. So we want to essentially steal the reference on the symbolic side.

On the stackref side, I could just change it to use PyStackRef_AsPyObjectSteal, as this seems more correct.

else:
out.emit(f"{outp.name} = sym_new_const(ctx, PyStackRef_AsPyObjectBorrow({outp.name}_stackref));\n")
storage.flush(out)
emitter.emit(f"{outp.name} = sym_new_const(ctx, PyStackRef_AsPyObjectBorrow({outp.name}_stackref));\n")
storage.flush(out)
emitter.emit("}\n")
emitter.emit("else {\n")

Expand Down Expand Up @@ -266,7 +266,8 @@ def write_uop(
out.start_line()
if override:
emitter = OptimizerEmitter(out, {})
_, storage = emitter.emit_tokens(override, storage, None, False, is_abstract=True)
_, storage = emitter.emit_tokens(override, storage, inst=None,
emit_braces=False, is_abstract=True)
storage.flush(out)
else:
emit_default(out, uop, stack)
Expand Down
Loading
pFad - Phonifier reborn

Pfad - The Proxy pFad of © 2024 Garber Painting. All rights reserved.

Note: This service is not intended for secure transactions such as banking, social media, email, or purchasing. Use at your own risk. We assume no liability whatsoever for broken pages.


Alternative Proxies:

Alternative Proxy

pFad Proxy

pFad v3 Proxy

pFad v4 Proxy