is a simple python list faster than redis for pushing data in?

I have a time sensitive data stream coming through and the structure is basically one line of dict coming through at a time but there can be thousands or tens of thousands of lines each second.

So I have been told to use Redis for speed in order to get the data in. The processing will happen in a different thread that is not speed sensitive. However comparing the performance of redis for this simple task I'm at a loss why I should use redis instead of a simple python list of dicts.

chatgpt came up with this test of speed.

    import time
    import redis
    
    redis_pool = redis.ConnectionPool(host='localhost', port=6379, db=0)
    redis_conn = redis.Redis(connection_pool=redis_pool)
    
    # Pushing dictionaries into Redis
    start_time = time.time()
    for i in range(100000):
        data = {'id': i, 'name': 'user {}'.format(i), 'age': i % 100}
        redis_conn.hmset('user:{}'.format(i), data)
    end_time = time.time()
    print('Time taken to push data into Redis:', end_time - start_time)
    
    # Using Python list
    start_time = time.time()
    data_list = []
    for i in range(100000):
        data = {'id': i, 'name': 'user {}'.format(i), 'age': i % 100}
        data_list.append(data)
    end_time = time.time()
    print('Time taken to create list:', end_time - start_time)
    
    start_time = time.time()
    for data in data_list:
        # Process data here
        pass
    end_time = time.time()
    print('Time taken to process list:', end_time - start_time)

when i ran it redis came to 7.5 seconds while python lists only 0.09. Is this representative of likely actual performance? Am i missing something here?

Depends what you want here... Redis can persist the data, and also allow it to be shared easily with different Python or other language processes. With your in memory Python list, you've lost your data if the process crashes, machine reboots etc.

Redis is a network server so you're going to incur some round trip time to the Redis server, even if it is on your local machine. If you're running in the cloud, you should ideally run your code close (in network terms) to the server to minimize latency.

You could also modify your code to take advantage of Redis pipelining, which is a technique that allows multple commands (hmset in your case) to be sent to the Redis server in one network round trip.

Redis pipelining: https://redis.io/docs/manual/pipelining/

Implementation of pipelining in the redis-py client your code uses: https://github.com/redis/redis-py#pipelines