英文:
Install Maven Package on Databricks Job-Cluster
问题
I want to use a maven package in a Databricks Job, which shall run on a new automated Cluster. Regular interactive clusters have the option to install a maven package. This installation resolves all dependencies of this package. On automated cluster you only can assign downloaded jars to be installed on startup of the cluster.
My problem is, that the dependencies of this jar are missing. Of course I can download them and add them to the cluster, but the dependency-tree seems to be pretty large.
Can I just download a jar with all dependencies included (did not found one)? Or can I install my the package in another way?
The package I need is azure-eventhubs-spark.
英文:
I want to use a maven package in a Databricks Job, which shall run on a new automated Cluster. Regular interactive clusters have the option to install a maven package. This installation resolves all dependencies of this package. On automated cluster you only can assign downloaded jars to be installed on startup of the cluster.
My problem is, that the dependencies of this jar are missing. Of course I can download them and add them to the cluster, but the dependency-tree seems to be pretty large.
Can I just download a jar with all dependencies included (did not found one)? Or can I install my the package in another way?
The package I need is azure-eventhubs-spark.
答案1
得分: 4
最终找到了解决方案。
要将Maven包附加到作业（-cluster），您需要在您的工作区中创建该库。在Databricks-UI的起始页面上，选择'导入库'，然后创建您想要的Maven包。然后可以将此包加载为作业设置中的依赖项。
这似乎是一个显而易见的解决方案，但我以前从未在Databricks中创建过库，因此不知道这个选项。
英文:
Finally found I solution.
To append a maven package to a job (-cluster) you have to create the library in your workspace. On the start page of the Databricks-UI choose 'Import Library', then create maven-package you'd like. This package can be loaded as dependency in the Job settings.
Was kind of a obvious solution, but I never created a lib in databricks and therefore wasn't aware of this option.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论