英文:
How the the computation code is transferreed to dask worker from the client?
问题
在上面的代码片段中,submit
如何将 xadd
、hello2
和 hello3
函数的代码传输到工作节点?
英文:
How does the client side source code get transferred to dask worker(s) ?
from dask.distributed import Client
def hello3(a, b):
print('I am hello 3')
return a + 5 + b * 3
def hello2(a, b):
print('I am hello2')
return hello3(a * 10, b + 5)
def xadd(a, b):
return hello2(a + 2, b * 5)
if __name__ == '__main__':
client = Client('127.0.0.1:8786')
x = client.submit(xadd, 1, 2)
print(x.result())
In the above code snippet, how will submit transfer code for xadd, hello2 and hello3 functions to the workers ?
答案1
得分: 1
Dask使用pickle
将大多数内容从客户端发送到调度程序,然后将内容发送到工作节点。对于在__main__
中定义的代码,您很可能会回退到使用cloudpickle
,这是一种类似的协议,可以处理更广泛的情况。
除其他事项外,cloudpickle还支持对lambda函数以及在
__main__
模块中以交互方式定义的函数和类进行pickle(例如在脚本、shell或Jupyter笔记本中)。
( https://github.com/cloudpipe/cloudpickle#readme )
英文:
Dask uses pickle
for sending most things from the client the scheduler, which then sends stuff on to workers. In the case of code defined in __main__
, you will (very likely) end up falling back to cloudpickle
, which is a similar protocol that's able to cope with a wider variety of situations.
> Among other things, cloudpickle supports pickling for lambda functions along with functions and classes defined interactively in the __main__
module (for instance in a script, a shell or a Jupyter notebook).
答案2
得分: 0
以下是已翻译好的部分:
"我不确定以下是否是正确的描述,但可能会提供一些有用的线索:
在通过client.submit
提交函数和参数后,dask
会对函数和参数(以及任何附加的kwargs)进行序列化。关于序列化的描述可以在文档中找到。
用于提取函数代码的具体函数是_get_computation_code
方法。"
英文:
I'm not certain that the following is a correct description, but might give some useful leads:
Upon submission of the function and arguments via client.submit
, dask
will serialize the function and arguments (as well as any additional kwargs). The description of the serialization is provided in the docs.
The specific function used to extract the function code is the _get_computation_code
method.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论