Is there a way to multithread or batch REST API calls in Python?

I've got a very long list of keys, and I am calling a REST API with each key to GET some metadata about it.

The API can only accept one key at a time, but I wondered if there was a way I could batch or multi-thread the calls from my side?

Yes, there are ways to multithread or batch REST API calls in Python to improve the performance of your program. One way to do this is by using the concurrent.futures module which provides a high-level interface for asynchronously executing functions using threads or processes.

Here's an example code that shows how you can use concurrent.futures to perform multithreaded REST API calls in batches:

import requests
from concurrent.futures import ThreadPoolExecutor
from itertools import islice

API_ENDPOINT = 'https://api.example.com/metadata'

def get_metadata(keys):
    results = []
    with ThreadPoolExecutor(max_workers=5) as executor:
        for batch in iter(lambda: list(islice(keys, 5)), []):
            futures = [executor.submit(get_metadata_for_key, key) for key in batch]
            results += [future.result() for future in futures]
    return results
            
def get_metadata_for_key(key):
    url = f"{API_ENDPOINT}/{key}"
    response = requests.get(url)
    if response.status_code == 200:
        return response.json()
    else:
        return None

In this example, get_metadata function takes a list of keys and uses the ThreadPoolExecutor to execute get_metadata_for_key function for each key in batches of 5. The islice function is used to create an iterator that returns batches of 5 keys from the input list. The executor.submit function is used to submit a new task to the thread pool for each key in the batch, which returns a concurrent.futures.Future object. The future.result() function is used to retrieve the result of each task and append it to the results list.

You can modify the max_workers parameter to control the number of threads used for executing tasks. In this example, I'm using 5 threads.

The other reply to this looks like ChatGPT so it should be ignored.

I did, however, use its code as a base to write a function that does what I want.

import requests
from concurrent.futures import ThreadPoolExecutor

API_ENDPOINT = 'https://api.example.com/metadata'

def get_metadata_for_key(key):
    url = f"{API_ENDPOINT}/{key}"
    response = requests.get(url)
    if response.status_code == 200:
        return response.json()
    else:
        return None

def get_save_metadata(keys, workers):
    results = {}
    batches = [keys[i : i + workers] for i in range(0, len(keys), workers)]

    with ThreadPoolExecutor(max_workers=workers) as executor:
        for batch in tqdm(batches):     #tqdm shows a progress bar
            futures = {key: executor.submit(get_metadata_for_key, key) for key in batch}
            futures_clean = {k: v.result() for k, v in futures.items() if v is not None}
            results.update({k: xmltodict.parse(v) for k, v in futures_clean.items()})
    
    return results