Python asyncio sleep is big memory usage

I have an asynchronous function, this function creates about 390 tasks and sleep a 1 second asyncio wait every 10 tasks, and I noticed that this python script is using a lot of ram. When I profiled with memory profiler, I realized that the most ram consuming thing as far as I could see was asyncio sleep. how can i solve this?

Profiller Output:

Line #    Mem usage    Increment  Occurrences   Line Contents
=============================================================
    91    667.5 MiB    667.5 MiB           1       @profile
    92                                             async def main(self):
    93    667.5 MiB      0.0 MiB           1           tasks = []
    94    667.5 MiB      0.0 MiB           1           start_time = time.time()
    95    667.5 MiB      0.0 MiB           1           following_listings_count = len(self.following_listings)
    96
    97   1018.8 MiB      0.0 MiB           3           async with self.api.session.create_async_oauth2_session() as session:
    98
    99    988.2 MiB      0.0 MiB         351               for offset in range(0, following_listings_count, 100):
   100    988.2 MiB      0.0 MiB         350                   paged_listings = self.following_listings[offset:offset + 100]
   101    988.2 MiB      0.0 MiB       36007                   listing_ids = [listing.listing_id for listing in paged_listings]
   102
   103    988.2 MiB      0.0 MiB         700                   tasks.append(
   104    988.2 MiB      0.0 MiB         350                       asyncio.create_task(self.get_listings(session, listing_ids, paged_listings))
   105                                                         )
   106    988.2 MiB      0.0 MiB         350                   if offset % 1000 == 0:
   107    988.2 MiB    320.6 MiB          70                      await asyncio.sleep(1.1)
   108    988.2 MiB      0.0 MiB         350                   print(offset)
   109   1018.8 MiB     30.6 MiB           2               data = await asyncio.gather(*tasks)
   110
   111
   112   1018.8 MiB      0.0 MiB           1               print(f"Data Scraping Finish Time: {time.time() - start_time}")
   113   1018.8 MiB      0.0 MiB           1           """start_time = time.time()
   114                                                 all_listings = []
   115                                                 for listings in data:
   116                                                     all_listings += listings
   117
   118                                                 self.create_reports(all_listings)
   119
   120                                                 print(f"Finished Time: {time.time() - start_time}")
   121                                                 """
   122   1018.8 MiB      0.0 MiB           1           return "Ok."

I used time sleep instead of asyncio sleep, but this time I got a limit warning because the api I used gave me a limit of 10 per second.

Edit: i using now thread pool executor but my problem is getting items from api. requests response.json() is Consumes 4-8 MB of memory all the time.

asyncio.sleep does not use that much memory - it is the profiler tool that is not asyncio aware, so one have to know how to interpret this

What it says is that during the steps where the line await asyncio.sleep were executed, memory usage increased by 300MB - but it is during those steps that other tasks are executed: the objects created by the other tasks are amounting to this much. In fact, in your code, it is just while awaiting asyncio.sleep and asyncio.gather that the tasks run at all (so replacing the asyncio.sleep by a synchronous sleep will do nothing but delay the start of execution of all tasks)

If you have to limit the total data rate or API usage, your use of asyncio.sleep in a single task would be incorrect anyway, as other tasks will proceed fetching data.

And, as you state, it looks like each API response is using a lot of RAM: the workaround them is to pick the relevant data for your app, and ensure the remaining of the response is discarded at each request.

Your listing is very hard to read, since you didn't have it separate from the post output. But the problem is obvious once one take the time to look at it: you create all your tasks in one batch and listen to all your tasks with a single asyncio.gather. The resources - including all local variables used in your tasks co-routines won't be freed until you collect the results, after all requests are made: tasks keep their state.

That kind of code is just good when you have memory and resources to spare to keep everything in memory. If memory is a concern, you have to change your code to create a limited number of tasks, and create more as those are completed and their results recorded. asyncio.wait can be a better option than .gather for this construct. Also, take a look at the use of asyncio Semaphores if limiting the number of concurrent requests is a concern.