如何将计算代码从客户端传输到Dask工作节点?

huangapple go评论74阅读模式
英文:

How the the computation code is transferreed to dask worker from the client?

问题

在上面的代码片段中,submit 如何将 xaddhello2hello3 函数的代码传输到工作节点?

英文:

How does the client side source code get transferred to dask worker(s) ?

from dask.distributed  import Client


def hello3(a, b):
    print('I am hello 3')
    return a + 5 + b * 3


def hello2(a, b):
    print('I am hello2')
    return hello3(a * 10, b + 5)


def xadd(a, b):
    return hello2(a + 2, b * 5)

if __name__ == '__main__':
    client = Client('127.0.0.1:8786')
    x = client.submit(xadd, 1, 2)
    print(x.result())

In the above code snippet, how will submit transfer code for xadd, hello2 and hello3 functions to the workers ?

答案1

得分: 1

Dask使用pickle将大多数内容从客户端发送到调度程序,然后将内容发送到工作节点。对于在__main__中定义的代码,您很可能会回退到使用cloudpickle,这是一种类似的协议,可以处理更广泛的情况。

除其他事项外,cloudpickle还支持对lambda函数以及在__main__模块中以交互方式定义的函数和类进行pickle(例如在脚本、shell或Jupyter笔记本中)。

( https://github.com/cloudpipe/cloudpickle#readme )

英文:

Dask uses pickle for sending most things from the client the scheduler, which then sends stuff on to workers. In the case of code defined in __main__, you will (very likely) end up falling back to cloudpickle, which is a similar protocol that's able to cope with a wider variety of situations.

> Among other things, cloudpickle supports pickling for lambda functions along with functions and classes defined interactively in the __main__ module (for instance in a script, a shell or a Jupyter notebook).

( https://github.com/cloudpipe/cloudpickle#readme )

答案2

得分: 0

以下是已翻译好的部分:

"我不确定以下是否是正确的描述,但可能会提供一些有用的线索:

在通过client.submit提交函数和参数后,dask会对函数和参数(以及任何附加的kwargs)进行序列化。关于序列化的描述可以在文档中找到。

用于提取函数代码的具体函数是_get_computation_code方法。"

英文:

I'm not certain that the following is a correct description, but might give some useful leads:

Upon submission of the function and arguments via client.submit, dask will serialize the function and arguments (as well as any additional kwargs). The description of the serialization is provided in the docs.

The specific function used to extract the function code is the _get_computation_code method.

huangapple
  • 本文由 发表于 2023年7月3日 18:54:02
  • 转载请务必保留本文链接:https://go.coder-hub.com/76604061.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定