How to limit kafka consumer poll result in bytes?

My setup has three Kafka brokers and a dozen of topics with 15 partitions each.
Java application starts 15 thread in which every consumer subscribes all topics. Let's say every consumer is assigned with three partitions of different topics. Topics has String records, each of them is less than 1Kb. Consumer config is:

  "key.deserializer" = "org.apache.kafka.common.serialization.StringDeserializer"
  "value.deserializer" = "org.apache.kafka.common.serialization.StringDeserializer"
  "bootstrap.servers" = "kafka-1:9092,kafka-2:9092,kafka-3:9092"
  "group.id" = "my-group"
  "enable.auto.commit" = "false"
  "fetch.max.bytes" = "60000"
  "max.poll.records" = "10000000"
  "auto.offset.reset" = "earliest"

I've set fetch.max.bytes to 60000 bytes to limit the whole consumer.poll result so as to send data on the next step to an API that has limited with 64Kb per call. But in practice consumer.poll returns much more bytes, for instance 150Kb. And API call fails.

What I've missed in the config and in the app design? How to strictly limit the result of consumer.poll in bytes?

The documentation says:

> Note that the consumer performs multiple fetches in parallel.

What does it mean? Should I limit a parallelism?

The problem is really connected with this note:

> Note that the consumer performs multiple fetches in parallel.

KafkaConsumer creates Fetcher and passes fetch.max.bytes to its constructor. Fetcher uses this value to send parallel requests to nodes.

Number of brokers thus should be taken into account to limit the poll result in bytes. In my cluster of three brokers fetch.max.bytes should be equal to 20000 so as to poll result will never exceed 60000 bytes.

One thing that you have to note when using fetch.max.bytes is that records are sent to the client in batches, and if the first record-batch that the broker has to send exceeds this size, the batch will be sent and the limit will be ignored. This guarantees that the consumer can continue making progress.

My suggestion is, since you have such tight limits of downstreams API, consider decreasing max.poll.records to around 50, so that at max you will poll 50 records and since your 1 record size is 1KB, the size of your poll request will be around 50KB. If there is a chance that some records can be 2KB or 5KB in size, consider decreasing size of your max.poll.records further (with an obvious downside that now your consumer will be doing more round-trips resulting in decrease in throughput).