Python asyncio collect info from multiple tasks at timeout, then continue in parallel to main until all tasks finished

My use case:

1. I want to run N tasks in parallel
2. After a specific period (timeout) I need to know the results from the tasks already finished by then. This is my "main" thread. This is already working in the snippet below.
3. In a task parallel and independent to main, I want to continue waiting for the remaining tasks to finish. Onces all tasks finished successfully I need to do some calculations based on all N results, and store the computation in a persistent cache. "main" does not need to wait it.

Any hints on how to achieve no. 3?

        tasks = [
            asyncio.create_task(self.call_111(harmonized_address)),
            asyncio.create_task(self.call_222(harmonized_address)),
            asyncio.create_task(self.call_333(harmonized_address)),
            asyncio.create_task(self.call_444(harmonized_address))
        ]
        await asyncio.wait(tasks, timeout=timeout.total_seconds())

        res = []
        for t in tasks:
            try:
                r = t.result()
            except asyncio.InvalidStateError as e:
                res.append(None)
            except Exception as e:
                res.append(e)
            else:
                res.append(r)

Update: (3) probably needs a bit more context. We are using python to develop AWS lambda functions. Our lambda function must respond within a given timebox, and it is acceptable for it to be a partial one. But we also want to wait for the remaining tasks to finish to update a cache with the full response.

Python based AWS Lambdas are not async. Maybe I need to start a separate thread, and execute the loop and tasks in their (similar to how larks outlined it in his answer below). And combine it with what dmfigol has posted in a gist about "Python asyncio event loop in a separate thread "

Obviously the new thread will finish later then the "lambda". I hope that is supported by AWS.

I'm not 100% sure I understood your questions correctly, so please let me know if this isn't what you were looking for. The following code runs a series of concurrent tasks. It waits up to a specific timeout for some tasks to complete, at which time it returns any results that have been collected so far. At some point later, a second call is made that collects the results for any tasks that did not finish before the timeout.

We take advantage of the fact that asyncio.wait returns the tuple done,pending (where done is a list of tasks that completed before the timeout, and pending is the list of tasks that are still running).

import asyncio
import random


class AsyncExample:
    def __init__(self):
        self.tasks = []

    async def example_task(self, id: int):
        sleeptime = random.randint(1, 20)
        print(f"start task {id}, sleeptime={sleeptime}")
        await asyncio.sleep(random.randint(1, 20))
        print(f"end task {id}")
        return (id, sleeptime)

    async def start_tasks(self):
        tasks = []
        for x in range(10):
            tasks.append(asyncio.create_task(self.example_task(x)))

        done, self.tasks = await asyncio.wait(tasks, timeout=10)

        res = []
        for t in done:
            res.append(t.result())

        return res

    async def collect_tasks(self):
        res = []
        done, _ = await asyncio.wait(self.tasks)
        for t in done:
            res.append(t.result())

        return res


async def main():
    app = AsyncExample()
    res = await app.start_tasks()
    print("processing early results")
    for i in res:
        print("got early result:", i)

    # ... do something else...
    await asyncio.sleep(5)

    print("processing additional results")
    for i in await app.collect_tasks():
        print("got late result:", i)


asyncio.run(main())

Running this will produce output something like:

start task 0, sleeptime=3
start task 1, sleeptime=17
start task 2, sleeptime=10
start task 3, sleeptime=12
start task 4, sleeptime=20
start task 5, sleeptime=15
start task 6, sleeptime=19
start task 7, sleeptime=15
start task 8, sleeptime=4
start task 9, sleeptime=6
end task 5
end task 8
end task 6
processing early results
got early result: (8, 4)
got early result: (6, 19)
got early result: (5, 15)
end task 3
end task 1
end task 0
end task 2
end task 9
gathering additional results
end task 4
end task 7
got late result: (3, 12)
got late result: (9, 6)
got late result: (1, 17)
got late result: (0, 3)
got late result: (4, 20)
got late result: (7, 15)
got late result: (2, 10)