Is Py_ReprEnter/ReprLeave really meant to be used that way? it also allocates lists and dicts so I don't know whether it's better than using ints.

I think it is meant to be used for that and other objects are using it, you can search around.

/* Helpers for printing recursive container types */
PyAPI_FUNC(int) Py_ReprEnter(PyObject *);
PyAPI_FUNC(void) Py_ReprLeave(PyObject *);

The big difference is that for a dict (current markers) you have to create a hash-able key object (but we keep track of lists and dicts) and it leads to creation of new Long objects. The Py_Repr* functions allocate dictionary and a list only on the first call per thread and then does a linear scan which is CPU-cache friendly. It doesn't do any allocations per new objects being tracked.

I think it is meant to be used for that and other objects are using it, you can search around.

To me it doesn't because it uses a specific key namely Py_Repr in the thread dict. If the object being recursed into is already in the list, wouldn't we detect it as being circular? (I don't have a PoC so I'm wondering whether we won't have a conflicting check). Apart from that, I can only find them being used in __repr__ and never elsewhere. Here it's something that is not __repr__ so I really don't want to misuse the API despite the comment.

It doesn't do any allocations per new objects being tracked.

While we don't allocate a full new object, we're still growing the list, or am I wrong?

cc @serhiy-storchaka as the JSON expert

I think it is meant to be used for that and other objects are using it, you can search around.

To me it doesn't because it uses a specific key namely Py_Repr in the thread dict. If the object being recursed into is already in the list, wouldn't we detect it as being circular? (I don't have a PoC so I'm wondering whether we won't have a conflicting check). Apart from that, I can only find them being used in __repr__ and never elsewhere. Here it's something that is not __repr__ so I really don't want to misuse the API despite the comment.

Who is the best person to ask? In the worst case we can copy the function and call it JsonEnter/JsonLeave because it works better than the current dict approach. Another option would be to change the internal API and make markers a list not a dict.

It doesn't do any allocations per new objects being tracked.
While we don't allocate a full new object, we're still growing the list, or am I wrong?

It will increase the list size if there is insufficient space but it will do so few times per the lifetime of the thread.

Py_ReprEnter has a very specific purpose.

The first peculiarity is that it uses a thread local Py_Repr to track the already seen objects. This is not needed for JSON encoder, it would be inefficient and may conflict with repr(). We could use an encoder attribute instead.

Second difference is that it uses a list instead of a dict. This may be more efficient for shallow recursion but much more slow for deep recursion. What is the threshold? 10? 100? Who knows, it needs to be measured.

We could use a hybrid approach -- list for shallow recursion and dict for deep recursion. The threshold should be determined by the results of microbenchmarks. This is a complex solution, so it should only be used if the benefit is large enough. It will also slow down the code near the threshold, so we should ensure that the overhead is small.