BUG: Fix reference leakage for output arrays in reduction functions #29358
Conversation
|
static int } If there's any chance obj isn't guaranteed to be a PyArrayObject, you might want to add: if (!PyArray_Check(obj)) { |
numpy/_core/src/umath/ufunc_object.c
Outdated
| @@ -3035,9 +3032,6 @@ PyUFunc_Reduceat(PyUFuncObject *ufunc, PyArrayObject *arr, PyArrayObject *ind, | |||
| return NULL; | |||
| } | |||
|
|
|||
| /* Take a reference to out for later returning */ | |||
| Py_XINCREF(out); | |||
There was a problem hiding this comment.
Thanks! I would prefer to keep the incref (somewhere in the function).
Removing the XINCREF here is correct and actually convenient! But it means the function takes one reference to out (if passed) and returns one new reference. Returning a new reference is obvious, but that means you have +1-1, i.e. the -1 is "stealing" the original reference.
But reference count stealing tends to be a bit confusing, largely on account it not very being common, so I would prefer to avoid introducing it (unless it is an **out API, where out is mutated/replaced in-place).
Also with alternative Python implementations and free-threading, reference stealing is a pattern that often needs to be avoided.
That is, re-add a decref in the calling code instead (I/we probably lost it when working on array_wrap improvements).
(Or never take a reference, since out= is currently always an ndarray already.)
We could add an explicit test as well via sys.getrefcount() (needs disabling on PyPy), almost surprised we don't have one.
There was a problem hiding this comment.
One way to do almost the same as here, is to use:
Py_XSETREF(out, function(..., out, ...));
at the call-site.
There was a problem hiding this comment.
Thanks for the pointers / context! That makes sense. I've reverted the reference stealing and instead readjusted some of the incref code to be a bit clearer (hopefully) and added a decref to PyUFunc_GenericReduction. Adding a test to check the reference counts!
There was a problem hiding this comment.
I've reverted to using XINCREF--we need the reference for the promotion! (and also seems clearer, actually). Also added a test.
…to `GenericReduction`
|
|
||
| out = np.empty(arr.shape, dtype=np.int32) | ||
| np.add.accumulate(arr, dtype=np.int32, out=out) | ||
| assert count == sys.getrefcount(arr) |
There was a problem hiding this comment.
You want to check the refcount on out (also)
| @@ -3727,6 +3727,8 @@ PyUFunc_GenericReduction(PyUFuncObject *ufunc, | |||
| if (ret == NULL) { | |||
| goto fail; | |||
| } | |||
|
|
|||
| Py_XDECREF(out); | |||
There was a problem hiding this comment.
This looks good, but also needed on error.
…umpy#29358) Add back missing DECREF for `out=` in ufunc.reduce (and etc.) when `out` is passed. * BUG: add Py_XDECREF to `out` for failure case in generic reduction * TST: add reference count checks for `out` in reduction leak tests
Fixes #29355.
We already increment a reference to
outin_set_out_array(inufunc_object.c):This PR removes additional increments and adds a missing decrement that caused the memory leakage.