英文:
How to install any Python package once and then reuse it in Databricks V2?
问题
In Databricks V1，在我的办公室里，我发现有一些Python包不是内置的，但被导入到笔记本中。然后我去了集群设置，但没有找到任何包 [我看到了集群设置，因为某个jar文件已经安装在那里供将来使用。所以我认为Python包可能以相同的方式处理。]。
但是，在Databricks V2中，在笔记本中，有一些安装命令'%pip install ...'。第一次运行笔记本后，我删除了这些安装命令，然后重新运行它。但它失败了，因为没有安装任何包。
我的问题是，那么当我在V2中运行'%pip install'时，安装的文件会去哪里？我可以创建一个全局共享的空间来存储安装文件，然后在V2中的所有笔记本中使用该存储吗？我不想一遍又一遍地使用安装命令。
英文:
In my office, in Databricks V1, I found that there are some Python packages which are not built-in, but is imported to the notebook. Then I went to the cluster settings, but didn't find any package there [I saw the cluster settings since some jar file was installed there for future use. So I thought that Python packages may be treated with same way.].
But, in Databricks V2, in the notebook, there are some installation commands '%pip install ...'. After running the notebook first time, I removed those installation commands, and reran it. But it failed as there is no packages installed.
My question is, then where does the installed files go when I run '%pip install' in V2? Can I create a global sharable space for the installation files, and then use that storage in all notebooks in V2? I don't want to use the installation commands again and again.
答案1
得分: 1
在Databricks中，每次启动集群时，它都是从头开始创建的，不会“记住”你在先前会话中安装的内容。
为了让这些库每次都可用，你可以将pip install命令放在你的代码开头，或者使用集群库来预先安装它们。具体指南请参考这里：https://docs.databricks.com/libraries/cluster-libraries.html
英文:
In Databricks, every time you start a cluster, it is created from scratch, and doesn't "remember" what you installed in the previouse sessions.
To have those libraries available each time, you can leave the pip install command in the beggining of your code, or use cluster libraries to pre-install them. Instruction here: https://docs.databricks.com/libraries/cluster-libraries.html
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论