Efficient and Scalable Way to Handle Time Series Analysis with Large Datasets in Python

I'm working with a very large dataset (over 100 million rows) of time-series data in Python. Each row represents a separate event with a timestamp, and there are multiple events for each timestamp. I need to perform analysis on this data, such as aggregating events by time intervals (e.g., by minute, by hour), calculating moving averages, etc.

Here's a simplified example of what my data looks like:

import pandas as pd

# Example data
data = pd.DataFrame({
    'timestamp': pd.date_range(start='1/1/2020', periods=100000000, freq='S'),
    'event': ['event{}'.format(i % 100) for i in range(100000000)]
})

The standard pandas approach for time-series analysis (e.g., using resample or rolling) is too slow for this amount of data.

I'm aware of Dask as a potential solution for handling large datasets, but I'm not sure if it's the best tool for time-series data, and if so, how to use it effectively for this purpose.

So, my question is: What are some efficient and scalable ways to handle time-series analysis with large datasets in Python? Are there specific techniques, libraries, or paradigms that are well-suited to this kind of problem?

I'd appreciate any advice or examples that could point me in the right direction.

I attempted to use the built-in pandas function resample to aggregate my data by minute. Given the size of my dataset, I expected this operation to be time-consuming, but hoped it would complete in a reasonable timeframe.

However, the operation took far longer than expected, and I had to terminate it before it completed. As a result, I was unable to perform the required analysis on my data.

I also tried using the rolling function to calculate moving averages, but faced similar performance issues. The computation was not only slow, but it also consumed a large amount of memory, causing my system to slow down significantly.

Therefore, I'm looking for a more efficient method to handle time-series analysis on large datasets in Python. My objective is to find a solution that can perform these operations faster and with less memory usage.

For working with large datasets, I would recommend you working with PySpark. Apache Spark (whose python interface is PySpark) is a distributed data processing engine which optimises processing of large datasets both on clusters and single machines. Here is an interesting article which compares the performance of pandas and PySpark on a single machine and how PySpark performs better.

Also, if you ever want to move to a cluster for your computations (which might become necessary given the amount of data you have), writing your code in PySpark would allow you to make a seamless transition.

Edit: Primary Reason why I personally recommended pyspark and not DuckDB or Polars

Recommendations are always personal and depend on the preferences and criteria one considers most important - some people might recommend DuckDB or Polars over pyspark and, if they have valid reasons which make sense in their context, that seems perfectly fine to me. We know very little about the context of OP apart from the fact that OP uses a lot of data, which is why I won't elaborate on aspects such as "solution integration", "ability to leverage solution in the cloud", "ability to distribute work amongst team members", etc.

The primary reason for me to recommend PySpark is Community Size: When choosing between various approaches where all provide the functionality that is needed, I personally would always go for the one which has the biggest open-source community; generally the larger the community, the more features and the more support (also on StackOverflow) you'll get. In my opinion the picture below shows compellingly whose community is the largest.