how to perform multiprocessing inside Azure API run function

I am trying to reduce the execution time of an API call by using multiprocessing:

My original requirement is actually on generating and displaying ML explainability using LIME. For simplicity, let's assume I have below data:

import numpy as np
import pandas as pd

df = pd.DataFrame({'cust_id' : ['id1','id1','id2','id2','id2','id1','id2','id1','id1','id2'],
                      'prod_id' : ['p1','p2','p3','p1','p4','p5','p6','p6','p8','p9'],
                      'cost' : np.random.randint(100, 1000, 10)})

We have another dataframe:

 df1 = pd.DataFrame({'cust_id' : ['id1','id1','id1','id2','id2','id2'],
                       'prod_id' : ['p1','p8','p3','p8','p9','p7']})

My API function looks something like this:

import json

def main(data):
    input_data = json.loads(data)["data"]
    customer_id = input_data[0]
    print(customer_id)
    item_list = df1.loc[df1["cust_id"] == customer_id, "prod_id"].tolist()

    idx = df.loc[
        (df["cust_id"] == customer_id) & (df["prod_id"].isin(item_list))
    ].index.values.tolist()

    for i in idx:
        df.loc[i, "cost"] = df.loc[i, "cost"] * 2

    return df

The input is in json format:

data = '{"data":["id1"]}'

out = main(data)

My actual code consists of this inplace of the multiplication:

explainer.explain_instance(df.loc[idx], model.predict_proba)

In actual scenario, the for loop would run for 24 times, fetching the row and putting it in the explain_instance.

Could someone please let me know how to perform multiprocessing of the for loop such that the 24 iterations come down as much as possible. I have 12 CPU cores in my actual instance.

Here is one way to do it with ProcessPoolExecutor class and partial function from Python standard library's concurrent.futures and functools modules:

from concurrent.futures import ProcessPoolExecutor
from functools import partial


def helper(row, explainer=None, predict_fn=None):
    """Function that will be applied concurrently"""
    df_ = pd.DataFrame(explainer(row, predict_fn))
    # do things with df_
    # ...
    return df_


def main(data):
    # Same code as before your for-loop
    input_data = json.loads(data)["data"]
    customer_id = input_data[0]
    item_list = df1.loc[df1["cust_id"] == customer_id, "prod_id"].tolist()

    idx = df.loc[
        (df["cust_id"] == customer_id) & (df["prod_id"].isin(item_list))
    ].index.values.tolist()

    with ProcessPoolExecutor() as executor:
        # Here, you need partial to deal with additional arguments:
        # explainer, predict_fn
        future = executor.map(
            partial(
                helper,
                explainer=explainer.explain_instance,
                predict_fn=model.predict_proba,
            ),
            [df.loc[i] for i in idx],
        )

    # future is a generator
    return list(future)


if __name__ == "__main__":
    data = '{"data":["id1"]}'
    dfs = main(data)  # list of dataframes

You have to execute the module in the top-level code environment (using if __name__ == "__main__"), otherwise a BrokenProcessPool exception will be raised.