How to prevent memory leaks caused by threads in Tomcat9 and Netty when using Lettuce with Spring?

I am using io.lettuce.core.api.StatefulRedisConnection from GenericObjectPoolConfig to connect Paas Redis.

Configure:

    @Bean
public GenericObjectPoolConfig redisPoolConfig() {
GenericObjectPoolConfig poolConfig = new GenericObjectPoolConfig();
poolConfig.setMaxTotal(redisProperties.getPool().getMaxTotal());
poolConfig.setMaxIdle(redisProperties.getPool().getMaxIdle());
poolConfig.setMinIdle(redisProperties.getPool().getMinIdle());
poolConfig.setMaxWaitMillis(redisProperties.getPool().getMaxWaitMillis());
poolConfig.setMinEvictableIdleTimeMillis(redisProperties.getPool().getMinEvictableIdleTimeMillis());
return poolConfig;
}
@Bean(destroyMethod = "close")
public GenericObjectPool<StatefulRedisConnection<String, Object>> redisObjectPool(RedisClient redisClient, GenericObjectPoolConfig genericObjectPoolConfig) {
RedisCodecForObject redisCodecForObject = new RedisCodecForObject(Object.class, redisProperties.getKeyPrefix());
ObjectMapper om = new ObjectMapper();
om.setVisibility(PropertyAccessor.ALL, JsonAutoDetect.Visibility.ANY);
TypeResolverBuilder <?> typeResolver = new ObjectMapper.DefaultTypeResolverBuilder(ObjectMapper.DefaultTyping.NON_FINAL) {
{
init(JsonTypeInfo.Id.CLASS, null);
inclusion(JsonTypeInfo.As.WRAPPER_ARRAY);
}
};
om.setDefaultTyping(typeResolver);
redisCodecForObject.setObjectMapper(om);
return ConnectionPoolSupport.createGenericObjectPool(() -> redisClient.connect(redisCodecForObject), genericObjectPoolConfig);
}
@Bean(destroyMethod = "close")
public StatefulRedisConnection<String, Object> objectRedisConnection(GenericObjectPool<StatefulRedisConnection<String, Object>> redisObjectPool) throws Exception {
StatefulRedisConnection<String, Object> connection = redisObjectPool.borrowObject();
return connection;
}

> SEVERE [main] org.apache.catalina.core.StandardContext.listenerStart
> Exception sending context initialized event to listener instance of
> class [ch.qos.logback.ext.spring.web.LogbackConfigListener]
> java.lang.IllegalStateException: Web app root system property already set to different value:
> 'webapp.root' = [/app/tomcat/webapps/ROOT/] instead of [/app/tomcat/webapps/demo/] -
> Choose unique values for the 'webAppRootKey' context-param in your web.xml files!
>
>
> WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [my-app] appears to have started a thread named [lettuce-timer-3-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
> java.lang.Thread.sleep(Native Method)
> io.netty.util.HashedWheelTimer$Worker.waitForNextTick(HashedWheelTimer.java:569)
> io.netty.util.HashedWheelTimer$Worker.run(HashedWheelTimer.java:465)
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> java.lang.Thread.run(Thread.java:750)
> 02-Jun-2023 19:35:11.842 WARNING [main] org.apache.catalina.loader.WebappClassLoaderBase.clearReferencesThreads The web application [my-app] appears to have started a thread named [parallel-1] but has failed to stop it. This is very likely to create a memory leak. Stack trace of thread:
> sun.misc.Unsafe.park(Native Method)
> java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:2039)
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:1081)
> java.util.concurrent.ScheduledThreadPoolExecutor$DelayedWorkQueue.take(ScheduledThreadPoolExecutor.java:809)
> java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:1074)
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1134)
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> java.lang.Thread.run(Thread.java:750)
> 02-Jun-2023 19:35:11.842 SEVERE [main] org.apache.catalina.loader.WebappClassLoaderBase.checkThreadLocalMapForLeaks The web application [my-app] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@381d724e]) and a value of type [io.netty.util.internal.InternalThreadLocalMap] (value [io.netty.util.internal.InternalThreadLocalMap@4736bb9f]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.

My local development environment started Tomcat without the following warning. However, While start tomcat by jenkins, I am getting below warning logs(Currently, this warning doesn't affect system access and usage):

Here are some configuration in pom.xml :

    <dependency>
<groupId>org.springframework.data</groupId>
<artifactId>spring-data-redis</artifactId>
<version>1.5.2.RELEASE</version>
</dependency>
<dependency>
<groupId>org.apache.commons</groupId>
<artifactId>commons-pool2</artifactId>
<version>2.9.0</version>
</dependency>
<dependency>
<groupId>io.lettuce</groupId>
<artifactId>lettuce-core</artifactId>
<version>6.1.0.RELEASE</version>
</dependency>

How can i close these threads to prevent memory leak.

i tried some methods to prevent, such as io.netty exception when implementing Spring Session with Redis. but it does not work for me.

reference1 - logback

reference2 - lettuce client

cause I：Due to the failure of the container to load the LogbackConfigListener during Tomcat startup, it triggers the execution of methods related to shutting down the Redis component and connections, as well as other components within the Tomcat container (destroyMethod).

error info contains two parts：
【a thread named [lettuce-timer-3-1] but has failed to stop it】

【a thread named [parallel-1] but has failed to stop it】

cause II：AS-IS : The Demo Tomcat is configured to have the default access path ""
pointing to "demo-portal". The server.xml file has the following configuration added. By default, Logback sets the web application root path as the system property 'webapp.root', which is set to "/app/tomcat/webapps/ROOT/". This allows the application to reference it in the Logback configuration file. However, this causes an error when loading LogbackConfigListener.

> <Host name="localhost" appBase="webapps" unpackWARs="true"
> autoDeploy="true">
> <Context path="" docBase="demo-portal" reloadable="true"/> </Host>

/**
* Set a system property to the web application root directory.
* The key of the system property can be defined with the "webAppRootKey"
* context-param in {@code web.xml}. Default is "webapp.root".
* <p>Can be used for tools that support substition with {@code System.getProperty}
* values, like log4j's "${key}" syntax within log file locations.
* @param servletContext the servlet context of the web application
* @throws IllegalStateException if the system property is already set,
* or if the WAR file is not expanded
* @see #WEB_APP_ROOT_KEY_PARAM
* @see #DEFAULT_WEB_APP_ROOT_KEY
* @see WebAppRootListener
* @see Log4jWebConfigurer
*/
public static void setWebAppRootSystemProperty(ServletContext servletContext) throws IllegalStateException {
Assert.notNull(servletContext, "ServletContext must not be null");
String root = servletContext.getRealPath("/");
if (root == null) {
throw new IllegalStateException(
"Cannot set web app root system property when WAR file is not expanded");
}
String param = servletContext.getInitParameter(WEB_APP_ROOT_KEY_PARAM);
String key = (param != null ? param : DEFAULT_WEB_APP_ROOT_KEY);
String oldValue = System.getProperty(key);
if (oldValue != null && !StringUtils.pathEquals(oldValue, root)) {
throw new IllegalStateException(
"Web app root system property already set to different value: '" +
key + "' = [" + oldValue + "] instead of [" + root + "] - " +
"Choose unique values for the 'webAppRootKey' context-param in your web.xml files!");
}
System.setProperty(key, root);
servletContext.log("Set web app root system property: '" + key + "' = [" + root + "]");
}

soluctions:
step1: change redis configuration：add the destroyMethod method for the ClientResources and RedisClient bean when application closing.

 @Bean(destroyMethod = "shutdown")
public ClientResources clientResources() {
return DefaultClientResources.builder()
//reconnection interval：5s
.reconnectDelay(Delay.constant(Duration.ofSeconds(5)))
.build();
}
@Bean(destroyMethod = "shutdown")
public RedisClient redisClient(RedisURI redisURI, ClientResources clientResources) {
ClientOptions clientOptions = ClientOptions.builder()
//error auto retry 
.autoReconnect(true)
.pingBeforeActivateConnection(true)
.build();
RedisClient redisClient = RedisClient.create(clientResources, redisURI);
redisClient.setOptions(clientOptions);
return redisClient;
}

step2: Fix for loading LogbackConfigListener failure: Logback no longer sets the web application root path as a system property.

<context-param>
<param-name>logbackExposeWebAppRoot</param-name>
<param-value>false</param-value>
</context-param>