Great write up @bradzacher!

I agree that we have matured to the point where this is the main area where we can add real remaining value, as long as a clear design goal is that users can still use the existing ESLint tooling as they always have done (probably initially through a lot of duplication in the implementation, and then probably some kind of gnarly compat logic) then I am definitely all for this!

Are there folks from the ESLint side who should weigh in here? I'm not 100% familiar with all the issues as described; although the contents of the RFC make sense as described it's a pretty big jump for users to make an entirely CLI for them to use.

An alternate approach would be to to work with the ESLint CLI maintainers and propose some kind of plugin system for the CLI.

@JoshuaKGoldberg that was the reason for my point about around not breaking any existing support. The CLI would be entirely opt in and be most relevant for folks having issues with the current generic approach. I would imagine most users would never even know/care that this additional tool would exist.

I don’t want to speak for @bradzacher but give his thumbs up on my comment I imagine his own thinking is not a million miles away from that

Another reason we would not want any breakage in support is the supplementary tooling such as IDE plugins that we would not look to replace, the highly optimised and specific CLI would be purely additive to the current ecosystem

Yeah definitely we would have no choice but to maintain compat with the existing ESLint ecosystem because the IDE ecosystem relies on the standard method of operation.

Essentially right now our advice for larger projects right now is to use lerna/nx/etc to run the ESLint process isolated on each of the projects when doing an entire repo lint.

This OFC does require extra effort to ensure the configs are fully separated so they don't influence one another.

With a custom CLI we could essentially automate it for people and give them a best-practice by default.

The IDE usecase shouldn't matter for most of the OOMs because you usually work on a subset of files and thus a subset of projects.

We can probably chat to the ESLint folks! This would definitely require a detailed RFC on that side before it could be worked into the CLI. We are like 60% of the ESLint userbase so we have some weight behind us.
The only issue is that an RFC would take a non-trivial amount of work to draft and collab on plus integration into the ESLint codebase - which would limit the impact to only codebases on ESLint 8+.

There are pros and cons to both approaches.

I imagine having a custom CLI would also provide a possible path to supporting .eslintrc.ts, which'd be cool to be able to explore further.

While it'd not be directly compatible with the ecosystem, I think most tools and IDEs should fine as they should primarily be caring about how to "talk" to eslint - aka the CLI.

Am excited to see where this goes 🚀

Initial proof of concept for the CLI: #4359

On our repo it looks to be consistently ~15-20s faster than a pure ESLint run.

Essentially right now our advice for larger projects right now is to use lerna/nx/etc to run the ESLint process isolated on each of the projects when doing an entire repo lint.

@bradzacher Is this documented anywhere? I want a reference to make sure my configs are right.

@cyrfer
https://typescript-eslint.io/docs/linting/typed-linting/monorepos

we don't explicitly talk about running things in isolated threads, but it's the logical next step if you're running into OOMs after separating your tsconfigs.

The root cause of OOMs is that we have to create TS data structures without cleaning them up.

• More files = more memory.
  • This should be obvious - each .ts file has a corresponding ts.SourceFile and TS generates a set of type objects for all the types in the file.
• More interdependencies between your packages = more memory.
  • This is less obvious. If package A depends on package B, then A needs the types from B. This means that TS will read in all of the .d.ts files for package B. If package C also needs package B, then that's another copy of these .d.ts files (because TS doesn't have mechanisms to share memory between programs).
  • This combines exponentially with "more files" because more files means more .d.ts files which are duplicated n times (once for each dependent package).
  • This could potentially be solved by project references but TS doesn't have good infra to do this for us.

So how can you solve the problem?
One way is to use one tsconfig to work around the interdependency problem. If you have one tsconfig which includes every single file in your workspace then there's no need to load .d.ts files or duplicate them! But this means your entire project

1. needs to be able to be represented in one config (which doesn't work if different packages have different libs for example)
2. needs to fit into memory (eg ts.SourceFiles + types all need to fit into <2GB of RAM)

Another way is to just ignore the problem entirely! By using a separate nodejs process per package, you no longer need to worry about any of the above problems because you get 2gb of ram per package to play with! Which is the same limitation placed on a tsc run (or your webpack run, or whatever).

Which is why I proposed this CLI - we can do this for you automatically and save everyone trying to figure out how to make it work.
It also gives us a nice entrypoint to do things like create a bug reporting tool, a templating tool, or a troubleshooting tool, etc.

also cc @JoshuaKGoldberg based on our discussion the other day.
This is my "disconnected" solution for the whole "eslint doesn't give us much information" problem. Instead of changing ESLint, we inject ourselves in front of ESLint so that we can get that information and act on it properly, solving many of our performance problems!