• I have tried restarting my IDE and the issue persists.
• I have updated to the latest version of the packages.
• I have read the FAQ and my problem is not listed.

Repro

At Chrome DevTools, we are currently using typescript-eslint to lint our TypeScript files: https://source.chromium.org/chromium/chromium/src/+/main:third_party/devtools-frontend/src/.eslintrc.js;l=127-128;drc=5d89e07921692d37ba1dc9b54e84c1533c0ac8cc Since we have a lot of TypeScript, we would like to turn on additional TypeScript ESLint rules that rely on type checking information for improved code correctness. Sadly, we have been running into the OOM on large projects issue (which is not unsurprising, as DevTools contains ~150k lines of TypeScript).

We have made several attempts, with @szuend attempting the latest. During our investigation, we discovered the following problem: our “extends” clause was mis-typed, which led to an interesting interaction between ESLint, typescript-eslint and TypeScript. Our source code is up at https://crrev.com/c/2095304 with most notably the following tsconfig.eslint.json:

{
  "extends": "tsconfig.json",
  "include": [
    "front_end/**/*.ts",
    "test/**/*.ts"
  ],
  "exclude": [
    "front_end/third_party",
    "node_modules",
    "out"
  ]
}

The tsconfig.eslint.json already includes the “minimal” (again, 150k lines of TypeScript…) amount of files and we ignore all other files. It references our “tsconfig.json”, that is also placed in the root directory of our repository: https://source.chromium.org/chromium/chromium/src/+/main:third_party/devtools-frontend/src/tsconfig.json;drc=1f7a8206536e544818f3dfbea589179dad1d53e8

We then try to run our ESLint normally via our integration script: https://source.chromium.org/chromium/chromium/src/+/main:third_party/devtools-frontend/src/scripts/test/run_lint_check_js.js;drc=0e8d7f4c64b83bef7a86c93af54704b4329b031e There is a lot going on here that is not really relevant to the problem at hand, but if you want to locally reproduce, you could do so with the following command:

node scripts/test/run_lint_check_js.js

Again, I don’t think that’s really necessary, as essentially the script runs eslint -c .eslintrc.js.

As linked in the above CL, we reference the tsconfig.eslint.json file in our parserOptions:

'parserOptions': {
    'project': './tsconfig.eslint.json',
  },

If you were to run the above script, you would run into the OOM issues as reported in #1192 @szuend started debugging both typescript-eslint as well as our configuration and discovered that typescript-eslint was silently swallowing errors:

DEBUG=* node scripts/test/run_lint_check_js.js

An example stack trace that we saw in our logs:

Error: File 'tsconfig.json' not found.
    at diagnosticReporter ($PATH/devtools-frontend/node_modules/@typescript-eslint/typescript-estree/dist/create-program/createWatchProgram.js:95:11)
    at Object.watchCompilerHost.afterProgramCreate ($PATH/devtools-frontend/node_modules/@typescript-eslint/typescript-estree/dist/create-program/createWatchProgram.js:234:13)
    at synchronizeProgram ($PATH/devtools-frontend/node_modules/typescript/lib/typescript.js:115853:22)
    at Object.createWatchProgram ($PATH/devtools-frontend/node_modules/typescript/lib/typescript.js:115777:9)
    at createWatchProgram ($PATH/devtools-frontend/node_modules/@typescript-eslint/typescript-estree/dist/create-program/createWatchProgram.js:287:22)
    at Object.getProgramsForProjects ($PATH/devtools-frontend/node_modules/@typescript-eslint/typescript-estree/dist/create-program/createWatchProgram.js:185:30)
    at Object.createProjectProgram ($PATH/devtools-frontend/node_modules/@typescript-eslint/typescript-estree/dist/create-program/createProjectProgram.js:22:74)
    at getProgramAndAST ($PATH/devtools-frontend/node_modules/@typescript-eslint/typescript-estree/dist/parser.js:87:36)
    at Object.parseAndGenerateServices ($PATH/devtools-frontend/node_modules/@typescript-eslint/typescript-estree/dist/parser.js:467:30)
    at Object.parseForESLint ($PATH/devtools-frontend/node_modules/@typescript-eslint/parser/dist/parser.js:97:51) +6s

As it turns out, our tsconfig.eslint.json was wrongly referring to the extending tsconfig.json. Instead of specifying "extends": "tsconfig.json", it should be "extends": "./tsconfig.json",. In other words, the “extends” clause must be a relative path.

Again, the stack trace was only visible if we turned on explicit debugging information. So the next question for us was: why is typescript-eslint not hard-failing on this error and we are running OOM?

After adding numerous debug statements and following the control flow in typescript-eslint, we believe that the following is happening:

1. createProjectProgram uses firstDefined to either get a previously used program or creates a new one:

   typescript-eslint/packages/typescript-estree/src/create-program/createProjectProgram.ts

   Lines 25 to 28 in 06c2d9b

   	const astAndProgram = firstDefined(
   	getProgramsForProjects(code, extra.filePath, extra),
   	currentProgram => getAstFromProgram(currentProgram, extra),
   	);

2. In getProgramsForProjects we create a fresh watch program:

   typescript-eslint/packages/typescript-estree/src/create-program/createWatchProgram.ts

   Line 231 in 06c2d9b

   const programWatch = createWatchProgram(tsconfigPath, extra);

3. In createWatchProgram, we are seeing errors being thrown that are processed in the afterProgramCreate:

   typescript-eslint/packages/typescript-estree/src/create-program/createWatchProgram.ts

   Lines 290 to 301 in 06c2d9b

   	watchCompilerHost.afterProgramCreate = (program): void => {
   	// report error if there are any errors in the config file
   	const configFileDiagnostics = program
   	.getConfigFileParsingDiagnostics()
   	.filter(
   	diag =>
   	diag.category === ts.DiagnosticCategory.Error && diag.code !== 18003,
   	);
   	if (configFileDiagnostics.length > 0) {
   	diagnosticReporter(configFileDiagnostics[0]);
   	}
   	};

4. The errors are immediately thrown:

   typescript-eslint/packages/typescript-estree/src/create-program/createWatchProgram.ts

   Lines 100 to 104 in 06c2d9b

   	function diagnosticReporter(diagnostic: ts.Diagnostic): void {
   	throw new Error(
   	ts.flattenDiagnosticMessageText(diagnostic.messageText, ts.sys.newLine),
   	);
   	}

So in this case, the error is thrown stating that the extended ts configuration can not be resolved. However, neither typescript-eslint nor ESLint halt program execution at this point. Instead, ESLint eagerly continues and goes to the next file. It then runs into the same exact error (since we don’t change the program configuration between file checks) and fails again. This process repeats over and over again.

Now that normally wouldn’t be a problem, but we are then observing that the watch program is keeping hold of memory. In other words:

1. typescript-eslint fails to create a dynamic program
2. ESLint catches the error
3. Even though the error is thrown, ESLint will continue with executing
4. typescript-eslint then is instructed to create a dynamic program again
5. typescript-eslint now holds memory both from the failed dynamic program in step 1 as well as step 4

This happens for every single file that we lint. Since we have many files (~1600), it means that typescript-eslint attempts to create dynamic programs for every single file, holding its memory after failure and eventually OOMs.

We believe that there are two issues at hand here:

1. typescript-eslint is not showing user configuration errors correctly, since they are getting swallowed by ESLint
2. When typescript-eslint fails to create a program, it does not free its corresponding memory

After @szuend observed the stack trace, we changed our configuration to correctly specify the base configuration with ./. We were then no longer observing the OOM issues. In other words: when we fixed our configuration issue, we were able to successfully complete 1 invocation of our ESLint script.

Therefore, we believe that (even though the OOM error is reported) fixing problem 1 will negate the need for addressing problem 2. If it turns out that solving problem 1 is not possible easily, then typescript-eslint would need to solve problem 2 and force freeing memory related to the program.

We believe that the errors not shown by ESLint as ESLint continues execution after errors and collects errors to show all errors at the end. Since typescript-eslint holds onto more memory than desired and OOMs, ESLint never had the possibility to report the error to the user.

Instead, we may have to request an update to ESLint to say “We are now reporting an error that we can’t recover from. Please halt execution of all rules immediately”. Then, ESLint can properly bubble the error from typescript-eslint up and show it to the user, prior to OOMing.

Versions