Is there any meaningful difference between re.compile and re._compiler.compile? I must admit I don't know, and I don't know why the original developers chose sre_compile.

On python 3.11

import re
import sre_compile

sre_compile.compile == re._compiler.compile
Out[11]: True
sre_compile.compile == re.compile
Out[12]: False

The deprecation thread does not help me either https://bugs.python.org/issue47152

I don't think so, see https://stackoverflow.com/questions/32576273/why-do-both-the-sre-and-re-modules-exist-in-python

So re.compile() is here: https://github.com/python/cpython/blob/main/Lib/re/__init__.py#L280
It wraps re._compiler.compile() here: https://github.com/python/cpython/blob/main/Lib/re/__init__.py#L307
It uses caching for compiled expressions.

So... they are basically the same, except re.compile() has caching and is the official (future-safe) api.
But cpplint has it's own implementation of caching already (without eviction), so it's a little silly to use re.compile which has caching and then also do caching in cpplint.

I believe re caches 256 compiled expressions, and cpplint has < 100 so far, with little reason to believe it will be more.

So if we do the migration in this PR, we can also remove the caching from cpplint (maybe check caching in python3.7 re)

@jspricke : Is that also your understanding, do you have a strong preference?

That makes a lot of sense to me, I've added a commit to do that.

ahhh, sorry. Actually I worry a lot about this change. The number of used patterns is larger than the number of occurrences of Search that I counted yesterday. Some of those occurrences use variables when building the pattern, like Search(r'new\(\S+\)\s*' + matched_type, line, but then there is also Match(). Now even if we did exceed the re.compile cache sizes, it might not cause significant additional slowdown if the LRU cache hit rate is high. Bug I don't know if I want to risk this. My concern is that for very large codebases, running cpplint could become much slower with this change.

So either we find some loadtest with a representative very large codebase and check there is no slowdown, or we keep the duplicate caching logic as documented technical debt (by adding a small comment where cpplint does caching). Or we make very damn certain that we stay below a certain number of patterns.

Thoughts?

I don't really mind, could you simply cherry-pick from this what you see fit or do you prefer if I rework this again?

ok, don't worry I'll cherry-pick Sorry for being indecisive about this, only have limited time for cpplint.