How to approach to design a near real-time data stream filtering solution to serve large data to a web client?

I have a problem of delivering a large CSV file which contains some user statistics events to a web client and by doing it I need to process and filter data to fulfill some metrics. This should be near realtime since data source will always have data.

I have investigated a bit, however I'm not sure how to approach this problem. I hear data streaming or push-based data streaming but I'm not sure how I can use for example Kafka to do stream processing or filtering in the broker before giving results to consumer.

When I approach this problem, as a first step I'm planning to split that file into chunks by utilizing a spliterator, then send this chunks into partitions, but this is the part that I'm confused: How and where filtering happens?

Let me explain the confusing part as best as I can do: Ok, I read file with chunks not to have out of memory exception, but to apply some filter, since the data is unsorted I think I need whole data which could result to memory exception again. So I'm not sure if I need to apply filter to each chucks in partitions and merge the results and in this case I think I need to apply same filter again to this merged results. Is this the idea in data stream processing when using Kafka in this case?

To make it more concrete, let's say this is the user activities data and I need to find average length of user sessions. In this case, I have user sessions scattered through chunks in several partitions. Should I need to find average in each chunk in each partition and and calculate again? Or If I need to filter followed users and in this case how can I accumulate results?

Filtering does not happen at the level of the broker. If you plan to use Kafka Streams, for example, you need to create a separate application, which performs the filtering and aggregation logic. You can read your file and send it to Kafka line by line. Your application will read the data from the topics and perform the filtering. If you need to calculate the average session per user, you should set the identifier of the user as a key, so that the users with the same id will go to the same partition. In this case you can have several instances of the application, each of them will read from their partitions and calculate the statistics.

The problem is that your task is about processing in batch and not streaming. So it will be hard to understand where is the end of the file and that you should stop processing. In streaming you typically calculate statistics using time windows.

Another possibility is to have you logic implemented in KSQL.

Hope this gives you an idea how to move on.