io.netty.handler.timeout.ReadTimeoutException: null error on AWS

We are experiencing the strangest problem. We are using WebFlux WebClient to make an API call. It was working fine, and then last week we started seeing the intermittent error:

io.netty.handler.timeout.ReadTimeoutException: null

To give some background. Our dev server is running on bare metal, and it is stable and has no issues at all. This is only happening in production on our AWS instance.

We see this happening on every second service call, which made me think load balancer, but my infrastructure team cannot see anything obvious.

This is the code that makes the service call.

Mono<String> responseMono = headersSpec.header("X-CORRELATION-ID", correlationId)
		.accept(MediaType.APPLICATION_JSON)
		.exchangeToMono(response -> {
			if (response.statusCode().equals(HttpStatus.OK)) {
				return response.bodyToMono(String.class);
			} else if (response.statusCode().equals(HttpStatus.BAD_REQUEST)) {
				return Mono.error(new DataViolationException("Data rejected","Data rejected",null,"ProductUse.useProduct"));
			} else {
				return Mono.error(new GeneralServiceException("Error status received: " + response.statusCode().toString() + " [" + response.bodyToMono(String.class) + "]",null));
			}
		});

I am wondering if it is an issue with our version of Spring? We are running the following:

Java Version: 18

Spring Version: 3.0.5

This application runs on Kubernetes.

Any suggestions would be much appreciated, at a bit of a loss here, have read other posts about this error but still unable to work out why it is happening to us, and why it only happens on AWS and not bare metal.

Let us break this down extensively...
The ReadTimeoutException can
occur when there is no response from the server within the specified
timeout period. But in your case, since the issue is happening
consistently on production and not on your development server, it is
possible that there could be a network issue between your AWS
instance and the external service.

Now, let us troubleshoot the problem together.

• Increase the timeout duration and see if it resolves this issue. You can use a debugger to step through the code and determine where
  the timeout is occurring. This can give you an idea of whether the
  issue is with the WebClient or somewhere else in your code.

• It is possible that there is an issue with the network connectivity between your AWS instance and the external service. You can try
  testing the network connectivity by using PING or TRACEROUTE commands
  from the AWS instance to the external service.

• You can as well upgrade to a newer version of Spring and Java. It appears that you are using older versions of Spring and Java, it is
  possible that there are compatibility issues that are causing the
  timeouts. You can try upgrading to newer versions and see if it
  resolves the issue.