High CPU load on Apache Karaf when using Kafka consumers in Camel route

We use Apache Camel 3.11.7 in apache-karaf-4.4.0 to read data from Apache-Kafka-2.13-3.3.1 using 3 nodes.
We have about 10000 Topics and one consumer for every topic.
The Apache-Karaf-VM has 2 CPUs and 32 GiB.

We use Apache Camel Route to consume the Messages from Topics using java, we integrate this code in Apache-Karaf as Bundles:

from ("kafka://crs.topic?brokers=PLAINTEXT://kafka1:9092,kafka2:9093,kafka3:9094&keyDeserializer=org.apache.kafka.common.serialization.StringDeserializer&valueDeserializer=com.rwe.remit.ejb.backend.kafkaclient.api.JacksonReadingSerializer&groupId=testing&heartbeatIntervalMs=120000&maxPollIntervalMs=86400000&sessionTimeoutMs=86400000&maxPollIntervalMs=86400000&deliveryTimeoutMs=86400000&requestTimeoutMs=86400000")

After starting all 900 Bundles in Apache Karaf, the consumers listen to the topics to get data from the topics.

Befor we beginn testing and sending data to Topics, we found on the Apache Karaf VM that the CPU has 100% load. This stays all the time.

We have this configuration on Kafka-Leader-Node:

process.roles=broker,controller
node.id=1
controller.quorum.voters=1@kafka1:19092,2@kafka2:19093,3@kafka3:19094
listeners=PLAINTEXT://kafka1:9092,CONTROLLER://kafka1:19092
inter.broker.listener.name=PLAINTEXT
advertised.listeners=PLAINTEXT://kafka1:9092                                                                                                                                                                                                 controller.listener.names=CONTROLLER
listener.security.protocol.map=CONTROLLER:PLAINTEXT,PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL
num.network.threads=3
num.io.threads=8
socket.send.buffer.bytes=102400
socket.receive.buffer.bytes=102400
socket.request.max.bytes=104857600
log.dirs=/data/apache-karaf/node1/log
num.partitions=3
num.recovery.threads.per.data.dir=1
offsets.topic.replication.factor=3
transaction.state.log.replication.factor=3
transaction.state.log.min.isr=3
log.retention.hours=168
log.segment.bytes=1073741824
log.retention.check.interval.ms=864000000
group.max.session.timeout.ms=86400000

We are migrating ActiveMQ to Kafka now.
We used before the Apache Karaf and Camel using Apache ActiveMQ instead of Kafka to consume the messages from the Queues. We did not have Performance Problem when using ActiveMQ.

Do you have any idea, why do we have much load on Linux-VM? Is there any way to show all internal processes and traffic in java or in Apache Karaf?

Below is the Thread Dump for two Kafka Consumers, the other 10000 Topics are the same.
The consumers are running all the time (100%). Why?
Is there a way to configure the consumers, that they run only when Topics have Data otherwise to remain in Standby-Mode?

"Camel (crs-rwe-tenants-test) thread #71 - KafkaConsumer[crs.topic1]" #394 daemon prio=5 os_prio=0 tid=0x00007f17b8374000 nid=0x1acf51 runnable [0x00007f1796253000]
   java.lang.Thread.State: RUNNABLE
	at java.lang.Integer.valueOf(Integer.java:832)
	at sun.nio.ch.EPollSelectorImpl.updateSelectedKeys(EPollSelectorImpl.java:120)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:98)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000006d72f07c8> (a sun.nio.ch.Util$3)
	- locked <0x00000006d72f07b8> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000006d72cea40> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at org.apache.kafka.common.network.Selector.select(Selector.java:873)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:465)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:280)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:251)
	at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1306)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1242)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1168)
	at org.apache.camel.component.kafka.KafkaConsumer$KafkaFetchRecords.doPollRun(KafkaConsumer.java:351)
	at org.apache.camel.component.kafka.KafkaConsumer$KafkaFetchRecords.doRun(KafkaConsumer.java:279)
	at org.apache.camel.component.kafka.KafkaConsumer$KafkaFetchRecords.run(KafkaConsumer.java:246)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

"Camel (crs-rwe-tenants-allgaeukraft) thread #79 - KafkaConsumer[crs.topic2]" #413 daemon prio=5 os_prio=0 tid=0x00007f17b895b000 nid=0x1acf64 runnable  [0x00007f1795142000]
java.lang.Thread.State: RUNNABLE
	at sun.nio.ch.EPollArrayWrapper.epollWait(Native Method)
	at sun.nio.ch.EPollArrayWrapper.poll(EPollArrayWrapper.java:269)
	at sun.nio.ch.EPollSelectorImpl.doSelect(EPollSelectorImpl.java:93)
	at sun.nio.ch.SelectorImpl.lockAndDoSelect(SelectorImpl.java:86)
	- locked <0x00000006d78c5610> (a sun.nio.ch.Util$3)
	- locked <0x00000006d78c5600> (a java.util.Collections$UnmodifiableSet)
	- locked <0x00000006d78a3710> (a sun.nio.ch.EPollSelectorImpl)
	at sun.nio.ch.SelectorImpl.select(SelectorImpl.java:97)
	at org.apache.kafka.common.network.Selector.select(Selector.java:873)
	at org.apache.kafka.common.network.Selector.poll(Selector.java:465)
	at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:560)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:280)
	at org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:251)
	at org.apache.kafka.clients.consumer.KafkaConsumer.pollForFetches(KafkaConsumer.java:1306)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1242)
	at org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:1168)
	at org.apache.camel.component.kafka.KafkaConsumer$KafkaFetchRecords.doPollRun(KafkaConsumer.java:351)
	at org.apache.camel.component.kafka.KafkaConsumer$KafkaFetchRecords.doRun(KafkaConsumer.java:279)
	at org.apache.camel.component.kafka.KafkaConsumer$KafkaFetchRecords.run(KafkaConsumer.java:246)
	at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
	at java.util.concurrent.FutureTask.run(FutureTask.java:266)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
	at java.lang.Thread.run(Thread.java:750)

Best regards, Amjad

You wrote:
"Befor we beginn testing and sending data to Topics, we found on the Apache Karaf VM that the CPU has 100% load. This stays all the time."

I suggest to take a series of thread dumps, e.g. 5-6 dumps every 5 seconds and analyse them carefully. It should show what threads are taking up all the CPU and what they are doing.

If you have access to Red Hat resources, an even better way would be to follow KCS https://access.redhat.com/solutions/46596.

In general, with 10000 topic subscribers connected to Kafka, I would expect some background noise in terms of idle CPU usage. Likely not using all of the available CPU.