Multiprocessing won't run

I'm trying to run this multiprocessing pool and I can't figure out why it won't run. It just seems to be processing endlessly. I am confident the function I am calling works (I have tested without the pool) so the error seems to be here. Any thoughts? The code runs as far as the for loop based on what prints.

The function it is calling runs rasterstats.zonal_stats if that matters.

if __name__ == "__main__":
    #create and configure the process pool
    print('inside', flush=True)
    with multiprocessing.Pool(processes = multiprocessing.cpu_count()) as pool:
        print('inside2', flush=True)
        #prepare arguments for function as list of variables and bands
        items = [(var, band) for var in clim_rasts.keys() for band in bands]
        print('inside3', flush=True)
        #concat the results
        stime2 = time.time()
        for result in pool.starmap(main_climate_task, items):
            print('result', result, flush=True)
            climate_data = pd.concat([climate_data, result]) )
        etime2 = time.time()
        dur2 = etime2-stime2
        print(dur2, flush=True)

Have you considered imap?

You only need starmap if you have multiple arguments to the function. But you have only a single argument, that happens to be a tuple. So you can use the non-star methods.

Have you considered imap_unordered?

That allows each result to feed out immediately, rather than wait to come out in the correct order.

Try changing

    for result in pool.starmap(main_climate_task, items):
        print('result', result, flush=True)
        climate_data = pd.concat([climate_data, result]) )

To

    for result in pool.imap_unordered(main_climate_task, items):
        print('result', result, flush=True)
        climate_data = pd.concat([climate_data, result]) )

If this at least starts to output results, that is good news.

In general I try to use imap_unordered whenever possible, because it makes maximal use of all your cores. Any core that finishes its work is immediately given the next item to work on, because it is always possible to write the output, rather than a core having to wait for the completion of earlier-listed items by other cores.

If ordering of results matters

As pointed out by @9769953, you can use pool:

results = pool.map(main_climate_task, items) 
climate_data = pd.concat(results)

I have a bias to using imap_unordered, because I like seeing the results coming out "as early as possible". Moreover you are having a puzzle of receiving no results, i.e. you are not sure if (a) one of your items is not finishing, (b) some of your items are not finishing, (c) none of your items are not finishing. The imap_unordered makes it easy to distinguish these possibilities.

For my approach there is a little work to do if you need climate_data to be in the order of items. For example, into the items = [(var, band)..., you could insert an iterator:

items = [(var, band, i_var, i_band) for var,i_var in enumerate(clim_rasts.keys()) for band,i_band in enumerate(bands)]

Then arrange for main_climate_task to pass back the i_band and i_var into result, so you can re-order the result entries into the desired order.

All due thanks to @Eureka for pointing out the environment may be the issue. I found out here (https://stackoverflow.com/questions/48846085/python-multiprocessing-within-jupyter-notebook) that multiprocessing will not run in JupyterLab on Windows. I saved my code in a .py file and ran that and it works fine.

%%writefile multiprocess.py

(ALL THE CODE GOES HERE)

%run multiprocess.py