Kafka message poll increase attempt

current setting of KAFKA Consumer:

 max_poll_records: 2000
 max_poll_interval_ms: 40000
 fetch_min_bytes: 104857600
 fetch_max_wait_ms: 10000
 fetch_max_bytes: 104857600
 request_timeout_ms: 300000
 max_partition_fetch_bytes: 104857600
 heartbeat_interval_ms: 60000
 session_timeout_ms: 180000

current setting at KAFKA Broker:

fetch.max.bytes: 104857600

with these settings I am able to poll 2000 messages at once of size 84MB approx (size of one poll).
but the second message i am receiving is lesser than before, while lag of sufficinet amount exists on that consumer.

I am pushing to only one partition so that poll also happens from one partition.

As seen in the screenshot of code, messages polled are in a pattern:
1st poll: 2000 messages
2nd poll: 484 messages
3rd poll: 2000 messages
4th poll: 484 messages
5th poll: 2000 messages
6th poll: 484 messages
7th poll: 548 messages
and so on

Can anyone help me figure this out, why is this happening?

I tried searching for the solution on google but didn't find much

Kafka consumer maintains an internal queue that on one hand is refilling with fetches from a broker and on the other hand is draining by your polls. After you poll 2000 records it becomes almost empty and Kafka consumer starts to fetch missing records. Since you set fetch_min_bytes to 100Mb it:

1. takes tangible time fetch request to complete. Probably more than it takes you to process polled messages.
2. consumer can either not call fetch since some records are still available in the internal queue, and a new fetch of such a high minimum size will lead to a queue overflow. So it takes another poll to drain remaining records in the internal queue before fetching messages from the broker.

So, try to dramatically (ten times, for example) decrease fetch_min_bytes or just remove that property since it's default is 1.

It is limited by max.partition.fetch.bytes,

Kafka Documentation
> MAX.PARTITION.FETCH.BYTES

This property controls the maximum number of bytes the server will return per partition. The default is 1 MB, which means that when KafkaConsumer.poll() returns ConsumerRecords, the record object will use at most max.partition.fetch.bytes per partition assigned to the consumer. So if a topic has 20 partitions, and you have 5 consumers, each consumer will need to have 4 MB of memory available for ConsumerRecords. In practice, you will want to allocate more memory as each consumer will need to handle more partitions if other consumers in the group fail. max. partition.fetch.bytes must be larger than the largest message a broker will accept (determined by the max.message.size property in the broker configuration), or the broker may have messages that the consumer will be unable to consume, in which case the consumer will hang trying to read them. Another important consideration when setting max.partition.fetch.bytes is the amount of time it takes the consumer to process data. As you recall, the consumer must call poll() frequently enough to avoid session timeout and subsequent rebalance. If the amount of data a single poll() returns is very large, it may take the consumer longer to process, which means it will not get to the next iteration of the poll loop in time to avoid a session timeout. If this occurs, the two options are either to lower max. partition.fetch.bytes or to increase the session timeout.