How to process elements of a list in python using multi threading without thread overlapping for each element?

I have a list of strings l = ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10']

I have a function to convert string to integer and return the number.

def to_int(s: str):
    print(int(s))
	return int(s)

I am beginner in multi threading with very minimal knowledge and to create a new thread in Python, I usually do this:

import time
from time import sleep
from threading import Thread

def to_int(s: str):
	print(int(s))
	return int(s)

# create a new thread
t1 = Thread(target=to_int, args=(l[0],))

# start the thread
t1.start()

# wait for the thread to complete
t1.join()

I can see the output of the function when the new thread starts. Now I have a huge list of size 50,000 elements and I'd like to process the entire list in smaller batches and I prepared batching as below:

batch_size = 2
for i in range(0, len(l), batch_size):
    sub_list = l[i:i+batch_size]

Is there anyway I can process each element of the sublist parallelly by calling the method to_int in parallell threads?
For ex: In the first iteration, I have the sublist with elements: ['1', '2', '3'] and I want to process all three elements parallelly and only run the next batch once current batch is completed.
Could anyone let me know is there a way I can achieve this functionality? Any help is really appreciated.

You could try to work with multiprocessing.pool.ThreadPool class like this:

from multiprocessing.pool import ThreadPool

# your task
def to_int(s: str): 
    return int(s)

# test data
l = ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10']

# create thread pool with 10 threads
pool = ThreadPool(processes=10)

# for each batch
batch_size = 2
for i in range(0, len(l), batch_size):
    sub_list = l[i:i+batch_size]
    # do the work
    print("batch", int(i/batch_size))
    for result in pool.map(to_int, sub_list):
        print(result)

Result:

batch 0
1
2
batch 1
3
4
batch 2
5
6
batch 3
7
8
batch 4
9
10

As already suggested I would not recommend to process every single element in a separate thread. It would be better if one thread processes an entire batch. But in the end it depends on your real task.

You can try this way for the manual method. The method that Marco provided is the ThreadPoolExecutor method. I tried to stick for the solution to be as similar to your question as possible.

import threading,time

# To create a list of string based integers to simulate the question.
lst = list(range(1,201))
lst1 = list(map(lambda x: str(x),lst))

# Function provided by the question
def to_int(s: str):
    print(int(s))
    return int(s)

batch_size = 4
# To see the sub-list groups
for i in range (0, len(lst1),batch_size):
    sub_list = lst1[i:i+batch_size]
    print(f"sub_list{i}:",end="")
    print(sub_list)
    

# Provided prints to easily visualize how the threads are processed.    
for i in range (0, len(lst1),batch_size):
    sub_list = lst1[i:i+batch_size]
    threads=[]
    for index in range(batch_size):
        print(f"Main : create and start thread {index}.")
        t = threading.Thread(target=to_int,name=str(index), args=(sub_list[index],))
        threads.append(t)
        t.start()
    for index, thread in enumerate(threads):
        print(f"Main : before thread {index}.")
        thread.join()
        print(f"Main : thread {index} done")
    print(f"All Threads completed for this sub_list{i}")

A part of the output will look as such:

All Threads completed for this sub_list188
Main : create and start thread 0.
Main : create and start thread 1.
Main : create and start thread 2.
Main : create and start thread 3.
Main : before thread 0.
Main : thread 0 done
Main : before thread 1.
Main : thread 1 done
Main : before thread 2.
Main : thread 2 done
Main : before thread 3.
Main : thread 3 done
All Threads completed for this sub_list192
Main : create and start thread 0.
Main : create and start thread 1.
Main : create and start thread 2.
Main : create and start thread 3.
Main : before thread 0.
Main : thread 0 done
Main : before thread 1.
Main : thread 1 done
Main : before thread 2.
Main : thread 2 done
Main : before thread 3.
Main : thread 3 done
All Threads completed for this sub_list196

You don't really need to do this in batches. Using map_async and a carefully calculated chunk size, you will get optimum performance.

Consider this:

from multiprocessing.pool import ThreadPool
from math import ceil
from time import perf_counter
from os import cpu_count

def to_int(v: str) -> int:
    try:
        return int(v)
    except ValueError:
        ...
    return -1

N = 50_000

# use half of available CPUs (minimum of 2)
if (NCPUS := cpu_count()):
    NCPUS = max(NCPUS//2, 2)
else:
    NCPUS = 2

# create N string representations of integer
my_list = [str(n) for n in range(N)]

with ThreadPool(NCPUS) as pool:
    chunksize = ceil(len(my_list) / NCPUS)
    start = perf_counter()
    results = list(pool.map_async(to_int, my_list, chunksize=chunksize).get())
    end = perf_counter()
    assert len(results) == N
    print(results[-20:])
    print(f'Duration={end-start:.4f}s')

Output:

[49980, 49981, 49982, 49983, 49984, 49985, 49986, 49987, 49988, 49989, 49990, 49991, 49992, 49993, 49994, 49995, 49996, 49997, 49998, 49999]
Duration=0.0094s

Create a global mutable object, in this case list result = list(). Then modify the to_int() function to work on array slices.

def to_int(arr_slice):
	for i in arr_slice:
		result.append(int(i))

Now you can start a thread for each slice. You can choose the size of the array slice in such a way that it optimizes your thread use.

Result should now contain all of the integers from your original list converted to int.

Applied to your example with 3 threads:

l = ['1', '2', '3', '4', '5', '6', '7', '8', '9', '10']
r = list()

t1 = Thread(target=to_int, args=(l[:3],))
t2 = Thread(target=to_int, args=(l[3:6],))
t3 = Thread(target=to_int, args=(l[6:],))

t1.start()
t2.start()
t3.start()

t1.join()
t2.join()
t3.join()