Cannot find type definition file for 'node'. Entry point of type library 'node' specified in compilerOptions (Azure Pipelines)

We have a node app that we are trying to deploy through Azure Pipelines. It works fine locally.
When the pipeline runs, it fails on the build step with this error immediately after the tsc command (build step) runs:

error TS2688: Cannot find type definition file for 'node'.
The file is in the program because:
Entry point of type library 'node' specified in compilerOptions

When built, the @types folder in node_modules on our build agent is missing everything except for validator and another random package.

I already did this command locally:
npm i --save-dev @types/node

The package-lock is updated and contains @types/node like so:

"@types/node": {
  "version": "14.18.47",
  "resolved": "https://registry.npmjs.org/@types/node/-/node-14.18.47.tgz",
  "integrity": "sha512-OuJi8bIng4wYHHA3YpKauL58dZrPxro3d0tabPHyiNF8rKfGKuVfr83oFlPLmKri1cX+Z3cJP39GXmnqkP11Gw==",
  "dev": true
},

This is the tsconfig.json:

{
  "compilerOptions": {
    "module": "commonjs",
    "declaration": true,
    "removeComments": true,
    "emitDecoratorMetadata": true,
    "experimentalDecorators": true,
    "allowSyntheticDefaultImports": true,
    "target": "es6",
    "sourceMap": true,
    "outDir": "./dist",
    "baseUrl": "./",
    "incremental": true,
    "esModuleInterop": true,
    "resolveJsonModule": true,
    "typeRoots": [ "./node_modules/@types" ], /* Specify multiple folders that act like './node_modules/@types'. */
    "types": [ "node"]
  }
}

devDependncies in package.json:

"devDependencies": {
    "@types/express": "^4.17.17",
    "@types/node": "^14.18.47",
    "@types/node-cron": "^3.0.7",
    "ts-loader": "^8.0.17",
    "ts-node": "^9.1.1",
    "tsconfig-paths": "^3.9.0",
    "typescript": "^4.1.5"
  }

The pipeline is just 2 Bash stages that run npm ci, followed by npm run build. Build agents are temporary and should not be caching or storing node modules anywhere.

I've tried installing @types/node multiple times in the pipeline, same error. Node modules just doesnt have the @types folder, or really much else.

This is everything inside of node_modules/@types

ls -lah node_modules/@types
total 16K
drwxr-xr-x   4 vsts docker 4.0K May 24 18:59 .
drwxr-xr-x 132 vsts docker 4.0K May 24 18:59 ..
drwxr-xr-x   2 vsts docker 4.0K May 24 18:59 triple-beam
drwxr-xr-x   4 vsts docker 4.0K May 24 18:59 validator

The weird part, which makes this impossible to debug, is that this pipeline ran green 3 days ago. My coworder did some seemingly irrelevant code changes that broke it (dotenv stuff) and caused this error. We reverted everything, wiped the agent, and fresh deployed and now 6 hours of debugging later we are still stuck on this issue. All of this matches the commit that was working before. Nothing else changed.

Every single thread online says to install types with save-dev, we have exhausted that solution and arrived nowhere.

We tried the save-dev command, which updates package json. We coupled this with updating the compilerOptions array in tsconfig.json.

Every single thread online is marked as complete with this solution, but the same error message persists for us.

Turns out, if NODE_ENV is set to production, it ignores devDependencies.

Nobody talks about this when they say to install the critical @types/node dependency to devDependencies, but I guess most people don't deploy their code to prod.

The solution for us is to set NODE_ENV to development in the pipeline, run the install, then set it back to production before the build.

Then prune with npm prune --production