gh-105858: Improve AST node constructors #105880
Fix pickling; fix a refleak
- Loading branch information
commit f941696deb85116c8333efcc9f2b5189fd78d812
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
|
|
@@ -1020,6 +1020,8 @@ def visitModule(self, mod): | |
| } | ||
| } | ||
| } | ||
| Py_DECREF(remaining_list); | ||
| Py_DECREF(field_types); | ||
| } | ||
| cleanup: | ||
| Py_XDECREF(fields); | ||
|
|
@@ -1041,14 +1043,75 @@ def visitModule(self, mod): | |
| return NULL; | ||
| } | ||
|
|
||
| PyObject *dict = NULL, *fields = NULL, *remaining_fields = NULL, | ||
| *positional_args = NULL; | ||
| if (PyObject_GetOptionalAttr(self, state->__dict__, &dict) < 0) { | ||
| return NULL; | ||
| } | ||
| PyObject *result = NULL; | ||
| if (dict) { | ||
| // Serialize the fields as positional args if possible, because if we | ||
| // serialize them as a dict, during unpickling they are set only *after* | ||
| // the object is constructed, which will now trigger a DeprecationWarning | ||
| // if the AST type has required fields. | ||
| PyObject *fields = NULL; | ||
| if (PyObject_GetOptionalAttr((PyObject*)Py_TYPE(self), state->_fields, &fields) < 0) { | ||
| goto cleanup; | ||
| } | ||
| if (fields) { | ||
| Py_ssize_t numfields = PySequence_Size(fields); | ||
| if (numfields == -1) { | ||
| Py_DECREF(dict); | ||
| goto cleanup; | ||
| } | ||
| PyObject *remaining_dict = PyDict_Copy(dict); | ||
| Py_DECREF(dict); | ||
| if (!remaining_dict) { | ||
| goto cleanup; | ||
| } | ||
| PyObject *positional_args = PyList_New(0); | ||
| if (!positional_args) { | ||
| goto cleanup; | ||
| } | ||
| for (Py_ssize_t i = 0; i < numfields; i++) { | ||
| PyObject *name = PySequence_GetItem(fields, i); | ||
| if (!name) { | ||
| goto cleanup; | ||
| } | ||
| PyObject *value = PyDict_GetItemWithError(remaining_dict, name); | ||
| if (!value) { | ||
| if (PyErr_Occurred()) { | ||
| goto cleanup; | ||
| } | ||
| break; | ||
| } | ||
| if (PyList_Append(positional_args, value) < 0) { | ||
| goto cleanup; | ||
| } | ||
| if (PyDict_DelItem(remaining_dict, name) < 0) { | ||
| goto cleanup; | ||
| } | ||
|
Comment on lines
+1081
to
+1092
There was a problem hiding this comment. There are leaks of |
||
| Py_DECREF(name); | ||
| } | ||
| PyObject *args_tuple = PyList_AsTuple(positional_args); | ||
| if (!args_tuple) { | ||
| goto cleanup; | ||
| } | ||
| result = Py_BuildValue("ONN", Py_TYPE(self), args_tuple, | ||
| remaining_dict); | ||
| } | ||
| else { | ||
| result = Py_BuildValue("O()N", Py_TYPE(self), dict); | ||
| } | ||
| } | ||
| else { | ||
| result = Py_BuildValue("O()", Py_TYPE(self)); | ||
| } | ||
| cleanup: | ||
| Py_XDECREF(fields); | ||
| Py_XDECREF(remaining_fields); | ||
| Py_XDECREF(positional_args); | ||
| return result; | ||
| } | ||
|
|
||
| static PyMemberDef ast_type_members[] = { | ||
|
|
||
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why do AST types even need their own custom reduce function? Shouldn't
object.__reduce__(which IIRC uses__new__to safely create an empty instance, then updates its__dict__) work fine for AST objects as well?There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The
__new__call will trigger DeprecationWarnings. If we don't have our own reducer, unpickling will essentially donode = ast.FunctionDef(); node.__dict__.update(...), and the first line raises warnings because we don't set the requirednamefield.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Why would the
__new__call raise deprecation warnings? Aren't the deprecation warnings in__init__, not__new__? That's whyobject.reduce()uses__new__, so that it can bypass whatever is happening in custom__init__methods (otherwise lots of custom classes would fail to unpickle by default). Soobject.reduce()is not equivalent tonode = ast.FunctionDef(); node.__dict__.update(...)-- it's more likenode = ast.FunctionDef.__new__(ast.FunctionDef); node.__dict__.update(...), which is a critical difference. So it seems to me like using the default reducer should work fine here, and not require all this extra work. But I could definitely be missing something; there's probably some reason these nodes had a custom reduce() in the first place.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I see what you mean. I tried commenting out the
__reduce__definition, and lots of tests fail with:I don't entirely understand what
_reduce_exis doing there, but it picksast.ASTas the base, which fails.There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Interesting. Well, I might have just nerd-sniped myself into digging into this more later to understand what's happening, but I don't think it should block the PR; your added code to use positional args works fine.