为什么 Kafka 生产者在第一条消息上非常缓慢?

huangapple go评论85阅读模式
英文:

Why kafka producer is very slow on first message?

问题

我正在使用 Kafka 生产者将价格发送到主题。当我发送第一条消息时,它会打印生产者配置,然后再发送消息,因此发送第一条消息需要更多的时间。

在发送第一条消息后,发送一条消息只需要大约 1/2 毫秒的时间。

我的问题是,我们是否可以做一些操作,以便跳过配置部分,或者我们可以在发送第一条消息之前就开始配置?

我在项目中使用了 Spring Kafka。我也阅读了其他问题,但没有真正有帮助的内容。

应用配置 (Application.yml):

server:
  port: 8081
spring:
    kafka:
      bootstrap-servers: ***.***.*.***:9092
      producer:
          key-serializer: org.apache.kafka.common.serialization.StringSerializer
          value-serializer: org.apache.kafka.common.serialization.StringSerializer

生产者配置:

acks = 1
batch.size = 16384
bootstrap.servers = [192.168.1.190:9092]
buffer.memory = 33554432
client.dns.lookup = default
client.id = 
compression.type = none
connections.max.idle.ms = 540000
delivery.timeout.ms = 120000
enable.idempotence = false
interceptor.classes = []
...
(其他配置)
...
transactional.id = null
value.serializer = class org.apache.kafka.common.serialization.StringSerializer

我参考了以下问题,但没有帮助:

  1. https://stackoverflow.com/questions/56413670/why-is-camel-kafka-producer-very-slow
  2. https://stackoverflow.com/questions/53728258/kafka-producer-is-slow-on-first-message
英文:

I am using kafka producer to send prices to topic. When I send first message it prints producer config and then send message due to this it takes more time to send first message .

After first message it tooks hardly 1/2 milliseconds to send a message .

My question is can we do something so that configuration part will skip or we can start before to send first message ?

I am using spring kafka into my project. I read other question also but not really helpful .

Application.yml

server:
  port: 8081
spring:
    kafka:
      bootstrap-servers:   ***.***.*.***:9092
      producer:
          key-serializer: org.apache.kafka.common.serialization.StringSerializer
          value-serializer: org.apache.kafka.common.serialization.StringSerializer

Producer Values :

acks = 1
batch.size = 16384
bootstrap.servers = [192.168.1.190:9092]
buffer.memory = 33554432
client.dns.lookup = default
client.id = 
compression.type = none
connections.max.idle.ms = 540000
delivery.timeout.ms = 120000
enable.idempotence = false
interceptor.classes = []
key.serializer = class org.apache.kafka.common.serialization.StringSerializer
linger.ms = 0
max.block.ms = 60000
max.in.flight.requests.per.connection = 5
max.request.size = 1048576
metadata.max.age.ms = 300000
metric.reporters = []
metrics.num.samples = 2
metrics.recording.level = INFO
metrics.sample.window.ms = 30000
partitioner.class = class org.apache.kafka.clients.producer.internals.DefaultPartitioner
receive.buffer.bytes = 32768
reconnect.backoff.max.ms = 1000
reconnect.backoff.ms = 50
request.timeout.ms = 30000
retries = 2147483647
retry.backoff.ms = 100
sasl.client.callback.handler.class = null
sasl.jaas.config = null
sasl.kerberos.kinit.cmd = /usr/bin/kinit
sasl.kerberos.min.time.before.relogin = 60000
sasl.kerberos.service.name = null
sasl.kerberos.ticket.renew.jitter = 0.05
sasl.kerberos.ticket.renew.window.factor = 0.8
sasl.login.callback.handler.class = null
sasl.login.class = null
sasl.login.refresh.buffer.seconds = 300
sasl.login.refresh.min.period.seconds = 60
sasl.login.refresh.window.factor = 0.8
sasl.login.refresh.window.jitter = 0.05
sasl.mechanism = GSSAPI
security.protocol = PLAINTEXT
send.buffer.bytes = 131072
ssl.cipher.suites = null
ssl.enabled.protocols = [TLSv1.2, TLSv1.1, TLSv1]
ssl.endpoint.identification.algorithm = https
ssl.key.password = null
ssl.keymanager.algorithm = SunX509
ssl.keystore.location = null
ssl.keystore.password = null
ssl.keystore.type = JKS
ssl.protocol = TLS
ssl.provider = null
ssl.secure.random.implementation = null
ssl.trustmanager.algorithm = PKIX
ssl.truststore.location = null
ssl.truststore.password = null
ssl.truststore.type = JKS
transaction.timeout.ms = 60000
transactional.id = null
value.serializer = class org.apache.kafka.common.serialization.StringSerializer

I refer following questions but not helped.

  1. https://stackoverflow.com/questions/56413670/why-is-camel-kafka-producer-very-slow
  2. https://stackoverflow.com/questions/53728258/kafka-producer-is-slow-on-first-message

答案1

得分: 7

在首次调用KafkaProducer.send方法时,Kafka生产者会获取主题的分区元数据。获取元数据会阻止send方法立即返回。Kafka生产者会缓存这些元数据,因此后续的发送操作速度更快。Kafka生产者会缓存元数据,时间为metadata.max.age.ms(默认为5分钟),之后会再次获取元数据,以便主动发现任何新的代理或分区。

当您的应用程序启动时,可以调用KafkaProducer.partitionsFor方法来获取并缓存元数据,但在缓存在5分钟后过期时,下一次发送操作将变慢,因为它会再次获取元数据。如果您的Kafka环境是静态的,也就是说,在您的应用程序运行时不会创建新的代理或分区,那么考虑将metadata.max.age.ms配置为很长的持续时间,以便元数据在缓存中保留更长的时间。

英文:

During the first invocation of the KafkaProducer.send method, the Kafka producer fetches the partition metadata for the topic. Fetching the metadata blocks the send method from returning immediately. The Kafka producer caches the metadata, so subsequent sends are much faster. The Kafka producer caches the metadata for metadata.max.age.ms (default 5 minutes), after which it again fetches the metadata to proactively discover any new brokers or partitions.

When your application starts, you could invoke the KafkaProducer.partitionsFor method to fetch and cache the metadata, but when the cache expires after 5 minutes, the next send will be slow because it fetches the metadata again. If your Kafka environment is static, that is, new brokers and partitions are not created while your application is running, then consider configuring metadata.max.age.ms to a very long time duration, so the metadata is kept in the cache longer.

答案2

得分: -1

什么是“第一条消息速度慢”?Kafka发送是异步的,不应影响您自己的性能。它使用一个缓冲区来批量发送记录。

请阅读以下Java文档以获取更多信息:

https://kafka.apache.org/10/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html

英文:

what do mean by first message is slow? kafka send is asynchronous and should not hinder your own performance. it uses a buffer that it uses to send records in batches.

read below java docs for more info,

https://kafka.apache.org/10/javadoc/org/apache/kafka/clients/producer/KafkaProducer.html

huangapple
  • 本文由 发表于 2020年10月6日 23:08:15
  • 转载请务必保留本文链接:https://go.coder-hub.com/64228639.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定