gh-124397: Add threading.iter_locked #133908
Conversation
|
@serhiy-storchaka, what was the rationale behind putting this in the |
|
|
||
| The ``iter_locked`` makes non-atomic iterators atomic:: | ||
|
|
||
| class non_atomic_iterator: |
There was a problem hiding this comment.
I'm not sure non_atomic_iterator is a good example of something that exists in the wild, because it's not thread-safe on GIL-ful builds either. The thread can arbitrarily switch during execution of Python code, so this would need a lock anyway if concurrent iteration were a use case. For free-threading, __next__ itself should have its own locking to prevent races. How was this design reached?
There was a problem hiding this comment.
The iter_locked allows the non_atomic_iterator to be used in a threaded-setting, whether in the normal or FT build. Do you have suggestions for other tests?
|
|
|
|
||
| def __next__(self): | ||
| a = next(self.it) | ||
| b = next(self.it) |
There was a problem hiding this comment.
Add time.sleep(0) (or with small non-zero argument) or Event.wait() with a timeout to force or significantly increase the chance of concurrent access.
Modules/_threadmodule.c
Outdated
| iter_locked_object *lz = iter_locked_object_CAST(op); | ||
| PyObject *result = NULL; | ||
|
|
||
| Py_BEGIN_CRITICAL_SECTION(op); // lock on op or lz->it? |
There was a problem hiding this comment.
Isn't it a no-op in the GIL build?
There was a problem hiding this comment.
Good catch, it is indeed a no-op. The Py_BEGIN_CRITICAL_SECTION is defined using _PyCriticalSection_BeginMutex, but that is not available in the normal build. So we should either add it to the normal build, or use another lock implementation.
Modules/_threadmodule.c
Outdated
|
|
||
| Py_BEGIN_CRITICAL_SECTION(op); // lock on op or lz->it? | ||
| PyObject *it = lz->it; | ||
| result = PyIter_Next(it); |
There was a problem hiding this comment.
Let's use PyIter_NextItem, PyIter_Next has a weird return value.
Continuation of #133272
Name was discussed in https://discuss.python.org/t/name-for-new-itertools-object-to-make-iterators-thread-safe/90394
📚 Documentation preview 📚: https://cpython-previews--133908.org.readthedocs.build/