在Databricks作业集群上安装Maven包。

huangapple go评论73阅读模式
英文:

Install Maven Package on Databricks Job-Cluster

问题

I want to use a maven package in a Databricks Job, which shall run on a new automated Cluster. Regular interactive clusters have the option to install a maven package. This installation resolves all dependencies of this package. On automated cluster you only can assign downloaded jars to be installed on startup of the cluster.

My problem is, that the dependencies of this jar are missing. Of course I can download them and add them to the cluster, but the dependency-tree seems to be pretty large.
Can I just download a jar with all dependencies included (did not found one)? Or can I install my the package in another way?

The package I need is azure-eventhubs-spark.

英文:

I want to use a maven package in a Databricks Job, which shall run on a new automated Cluster. Regular interactive clusters have the option to install a maven package. This installation resolves all dependencies of this package. On automated cluster you only can assign downloaded jars to be installed on startup of the cluster.

My problem is, that the dependencies of this jar are missing. Of course I can download them and add them to the cluster, but the dependency-tree seems to be pretty large.
Can I just download a jar with all dependencies included (did not found one)? Or can I install my the package in another way?

The package I need is azure-eventhubs-spark.

答案1

得分: 4

最终找到了解决方案。

要将Maven包附加到作业(-cluster),您需要在您的工作区中创建该库。在Databricks-UI的起始页面上,选择'导入库',然后创建您想要的Maven包。然后可以将此包加载为作业设置中的依赖项。

这似乎是一个显而易见的解决方案,但我以前从未在Databricks中创建过库,因此不知道这个选项。

英文:

Finally found I solution.

To append a maven package to a job (-cluster) you have to create the library in your workspace. On the start page of the Databricks-UI choose 'Import Library', then create maven-package you'd like. This package can be loaded as dependency in the Job settings.

Was kind of a obvious solution, but I never created a lib in databricks and therefore wasn't aware of this option.

huangapple
  • 本文由 发表于 2020年8月11日 17:02:05
  • 转载请务必保留本文链接:https://go.coder-hub.com/63354948.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定