Why do Mongoose v6 and v7 cause a significant increase in garbage collections and event loop latency compared to v5?

Background:
We have been trying to migrate from Mongoose v5 to v6 in our SaaS platform for the past year with no luck. The issues described here also occurs when running Mongoose 7.
Our software have a decent amount of web traffic(about 25 requests per second) and is using a single MongoDB Atlas cluster where each SaaS client is using its own database.
The software is hosted on Heroku.

The issue:
When we deploy a version of our software that is using Mongoose 6 or 7 it runs fine for up to an hour, then we start to see an increase in number of garbage collections, time spent in garbage collection, and event loop latency. This continues until the servers event loop is completely hogged and cannot handle any more incoming requests.
The memory usage is unaffected, so it doesn't seem to be a memory leak.

The code:

Upon start, we create a database connection with a pool of 300 connections:

global.databaseConnection = await mongoose.createConnection(global.config.dbConnectionString, {
	maxPoolSize: 300,
	minPoolSize: 300
});

When a request for a specific client is received, a connection to that specific clients database is drawn from the connection pool using the useDb-function:

clientDatabaseConnection = global.databaseConnection.useDb(`client-${client.databaseName}`, {
	noListener: true,
	useCache: true
});

The client database connection is then used to do various quesries required by the request, for example:

let clientUserCollection = connection.model<IClientUserModel>("ClientUser", ClientUserSchema, 'users');
let user = await clientUserCollection.findById(userId).exec();

This is basically the structure of the application. What is it that is different in v6 and v7 compared to v5 that could cause these performance implications?

I have tried to adjust the connection pool size, but this doesn't help.

We might have a similar issue. We are also using Heroku, MongoDB Atlas and Mongoose. We also have one DB per client.

We used to have one application per client. But one year ago, we switched to a multitenant architecture with several clients on a single heroku application, each with their own databases (we have several of those applications and around 20 clients on each). At the same time we implemented the architecture, we also moved from Mongoose 5 to Mongoose 6 (we are now on 7).

We are currently using createConnection and storing it ourselves in a map instead of useDb, but we also tried with useDb and had the same issue.

Today, we make it work by rebooting our dynos regularly. This is clearly a temporary solution.

In our case, it is pretty clear the crash is happening when many clients have connected to the same application. Memory seemed to increase, so we suspected issues due to how we managed mongoose connections and models storage, but we tried to move to dynos with more memory, and it crashes way before the memory limit. So, as in your case, it seems this is not a memory leak.

EDIT

1. Infinite loop in populate

So in our case, we found an infinite task (loop) in some cases generated by a nested populate. It would not break all the time, but rather when we connect to many databases at the same time. We don't know if it is linked to connections being dropped while the populate is on going or else.

The populate was like this :

ModelA.find(...)
  .populate([
    {
      path: 'propertytOfA',
      populate: {
        path: 'propertyB', // has discriminator
        populate: { path: 'propertyC' } // has discriminator
      }
    }
  ])

The important thing here could be that both of the last collections have discriminators. I wonder if this is what is called recursive embedded discriminators in the mongoose migration guide here.

2. maxPoolSize

You said you try to change this parameter, and it didn't help.

Well in our case, after solving the infinite task, we started seing other issues like for example Client network socket disconnected before secure TLS connection was established' so we decided to try tweaking the maxPoolSize.

We set it to 5 (default on Mongoose 5 that was increased to 100 in Mongoose 6) and it seemed to have solved our issues.