英文:
Using loop.run_in_executor to call sync functions from async ones
问题
我有3个函数:func_1
,func_2
和func_3
。我想要以异步方式运行它们,这样我就不必等待func_1
完成后再执行func_2
。
问题是,例如,func_1
的定义看起来像这样:
async def func_1(a, b):
x = some_sync_func(a)
y = some_other_sync_func(b)
z = another_sync_func(x, y)
return yet_another_sync_func(z)
我在func_1
中调用的函数都是同步函数,不可等待。因此,它们将阻塞func_2
和func_3
的执行。
我在这里阅读到可以使用loop.run_in_executor()
来从异步函数中调用同步函数而不阻塞执行。因此,我修改了func_1
的定义如下:
async def func_1(a, b):
loop = asyncio.get_event_loop()
x = await loop.run_in_executor(None, some_sync_func, a)
y = await loop.run_in_executor(None, some_other_sync_func, b)
z = await loop.run_in_executor(None, lambda: another_sync_func(a,b))
w = await loop.run_in_executor(None, yet_another_sync_func, z)
return w
这是处理这个问题的正确方法吗?我是否正确使用了loop.run_in_executor()
?这里的文档提供了一个示例,似乎支持这种方法。我不知道线程是什么,也不知道"进程池"是什么,而且一直没有能够很好地理解文档。
英文:
I have 3 functions: func_1
, func_2
, and func_3
. I would like to run these asynchronously, so that I do not have to wait for func_1
to finish before func_2
starts executing.
The problem is, that the definition of func_1
for example looks something like this:
async def func_1(a, b):
x = some_sync_func(a)
y = some_other_sync_func(b)
z = another_sync_func(x, y)
return yet_another_sync_func(z)
The functions that I am calling within func_1 are all synchronous functions which are non-awaitable. Thus, they will block the execution of func_2
and func_3
.
I read here that loop.run_in_executor()
can be used to call synchronous functions from asynchronous functions without blocking the execution.
Thus, I modified the definition of func_1
as follows:
async def func_1(a, b):
loop = asyncio.get_event_loop()
x = await loop.run_in_executor(None, some_sync_func, a)
y = await loop.run_in_executor(None, some_other_sync_func, b)
z = await loop.run_in_executor(None, lambda: another_sync_func(a,b))
w = await loop.run_in_executor(None, yet_another_sync_func, z)
return w
Is this the right way to deal with this problem? Am I using loop.run_in_executor()
correctly?
Here, the docs provide an example which seems to support this. I don't know what threads are, or what a "process pool" is, and haven't really been able to make much sense of the docs.
答案1
得分: 1
几乎正确,但由于您在每次函数调用时都急切地等待,因此在每种情况下(在等待之后)下一行代码只有在带有await
的行执行完成后才会被调用。
然而,如果您从其他地方并行调用func_1
,那么两个func_1
的实例将并行工作(我几乎确定这不是您想要的)。
因此,为了让这些其他函数实际并行运行(在其他线程中),您必须创建要运行每个函数的任务,但不立即等待它,而是收集所有要并行运行的任务,然后一次性等待它们(通常使用一个名为gather
的适当命名的函数):
...
async def func_1(a, b):
loop = asyncio.get_event_loop()
task_x = loop.run_in_executor(None, some_sync_func, a)
task_y = loop.run_in_executor(None, some_other_sync_func, b)
task_z = loop.run_in_executor(None, lambda: another_sync_func(a, b))
x, y, z = await asyncio.gather(task_x, task_y, task_z)
# 这取决于`z`,因此它不包括在gather中。
# 如果它的返回值不重要,您可以省略等待,返回任务,然后稍后某个时候等待它。
w = await loop.run_in_executor(None, yet_another_sync_func, z)
return w
...
英文:
Almost right, but since you are awaiting eagerly at each function call, the next line of code in each case (after the await) will only be called when the line with await
finishes execution.
However if you call func_1
in parallel from some other place, two instances of func_1
will work in parallel. (I am almost sure that is not what you want).
So, in order for these other functions to actually run in parallel (in other threads), you have to create the task to run each of them, but not await it immediately, instead, you gather all the tasks you want to run in parallel and await for them at once (usually with a function properly named gather
):
...
async def func_1(a, b):
loop = asyncio.get_event_loop()
task_x = loop.run_in_executor(None, some_sync_func, a)
task_y = loop.run_in_executor(None, some_other_sync_func, b)
task_z = loop.run_in_executor(None, lambda: another_sync_func(a,b))
x, y, z = await asyncio.gather(task_x, task_y, task_z)
# this depends on `z` so, it is not included in the gather.
# if its return value is not important, you can ommit the
# await, return the task, and await for it sometime later.
w = await loop.run_in_executor(None, yet_another_sync_func, z)
return w
...
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论