How to optimize n async concurrent calls made in Kotlin

I have the following code in Kotlin and curious if there is a way to further optimize it compared to the Python implementation that I think is somewhat equivalent?

Kotlin version 1.8.20-release-327 (JRE 17.0.7+7

import java.net.http.HttpClient
import java.net.http.HttpRequest
import java.net.http.HttpResponse
import java.net.URI
import kotlin.system.measureTimeMillis

import kotlinx.coroutines.*

fun main() = runBlocking {
    val timeElapsed = measureTimeMillis {
        val deferreds: List<Deferred<String>> = (1..7).map {
            async {
                val client = HttpClient.newBuilder().build();
                val request = HttpRequest.newBuilder()
                    .uri(URI.create("URL_WITH_INCREASING_PATH_PARAMETER/$it"))
                    .build();

                val response = client.send(request, HttpResponse.BodyHandlers.ofString());
                response.body()
            }
        }

        val data = deferreds.awaitAll()
        println("$data")
    }

    println("time elapsed $timeElapsed")
}

// time elapsed 2126

Python version 3.11.1

import time

import aiohttp
import asyncio

start_time = time.time()

async def get_data(session, url):
    async with session.get(url) as response:
        data = await response.json()
        return data

async def main():
    async with aiohttp.ClientSession() as session:
        tasks = []
        for number in range(1, 7):
            url = f'URL_WITH_INCREASING_PATH_PARAMETER/{number}'
            tasks.append(asyncio.ensure_future(get_data(session, url)))
        data_list = await asyncio.gather(*tasks)
        print(data_list)

asyncio.run(main())
print('time elapsed %s' % (time.time() - start_time))

# time elapsed 0.7383

This is slow for multiple reasons:

1. You're using the JDK 11's HttpClient.send() method, which is blocking. Also you're using runBlocking without overriding its dispatcher, so you're getting a single-threaded event-loop as dispatcher. This means that nothing runs in parallel here, the requests will be run sequentially, blocking turn by turn the single thread you're using.

2. You're creating an entirely new HttpClient instance for every request. This is usually a bad idea, because these clients manage a bunch of resources (like thread pools), and it's better to reuse one rather than create many.

The first thing you can do is move the client creation out of the loop, so you reuse the same client.

Then you could use Dispatchers.IO to run these blocking IO calls like this:

fun main() = runBlocking(Dispatchers.IO) {
 ...
}

But that is using more threads than necessary, because most likely the HTTP client already has its own thread pool.

A better option would be to use the actual asynchronous variant of the send method: sendAsync. It returns a CompletableFuture that you can await in a suspending way by using CompletableFuture.await()*:

fun main() = runBlocking {
    val client = HttpClient.newHttpClient();
    val timeElapsed = measureTimeMillis {
        val deferreds = List(7) {
            async {
                val request = HttpRequest.newBuilder()
                    .uri(URI.create("URL_WITH_INCREASING_PATH_PARAMETER/$it"))
                    .build()

                val response = client.sendAsync(request, HttpResponse.BodyHandlers.ofString()).await()
                response.body()
            }
        }

        val data = deferreds.awaitAll()
        println("$data")
    }

    println("time elapsed $timeElapsed")
}

*Note: the CompletableFuture.await() extension function used to be in the kotlinx-coroutines-jdk8 module in versions < 1.7.0. Since 1.7.0, this extension is available in the core coroutines module.