Are my requests being processed asynchronously?

I have a scraping script which requests about 30 urls synchronously. It takes 95s to complete all requests.

I've rewritten the script using asynchronous libraries asyncio and aiohttp in order to improve speed.

Here are the performance statistics for 32 requests:

• synchronous requests total time = 95 seconds
• asychronous requests total time = 60 seconds
• single, manual hit on the url from the browser - about 1 second

I think speed improvement of 50% is very bad so I'm trying to suspect that my requests are not really firing asychronously (I'm new to asyncio).

In fact, since the single request takes only 1 second and I make only 32 requests I was expecting my total asychronous requests time to be less than 1.5 seconds. 32 requests is very few so I assume that they all would start almost at the same time, and so waiting for the last to complete shouldn't take more than 1.5 seconds.

I would appreciate any hints.

# Single asynchronous request
async def async_get_course(session, url_course):
    
    async with session.get(url_course) as res:
        response = await res.content.read() 
        return response

# Main coroutine
async def example(courses, root):

  starting_time = time.time()
    
  actions = []
  data = []
  data2 = []
  async with aiohttp.ClientSession() as session:
      for course in courses:
          url_course = f"{root}{course['course_link']}"
          data.append(url_course)
          actions.append(asyncio.ensure_future(async_get_course(session, url_course)))
      results = await asyncio.gather(*actions)
      
      for idx, res in enumerate(results):
          data2.append(get_info_from_course((courses[idx], data[idx], res)))

  total_time = time.time() - starting_time
  print('total_time', total_time)

  return data2

# Run the coroutine
courses_final = asyncio.run(example(courses, root))

Your actions will execute concurrently:

await asyncio.gather(*actions)

So you did everything correctly, you are running asynchronous and concurrent code. Your requests will not start at the same time. The event loop will process one more request when the other is blocked. This doesn't mean your strategy here is bad, in general, it's very helpful to use asyncio to process I/O heavy operations as you have. You can try to split your actions into several processes and still utilize asyncio.