英文:
Databricks code does not work anymore with 'directory not found' error
问题
这是来自四年前的一个Stack Overflow问题。它在Databricks Notebook中正常工作。
%python
import pandas as pd
from io import StringIO
data = """
CODE,L,PS
5d8A,N,P60490
5d8b,H,P80377
5d8C,O,P60491
"""
df = pd.read_csv(StringIO(data), sep=',')
#print(df)
df.to_csv('/dbfs/FileStore/NJ/file1.txt')
pandas_df = pd.read_csv("/dbfs/FileStore/NJ/file1.txt", header='infer')
print(pandas_df)
现在不再起作用。错误消息是:
FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/NJ/file1.txt'
想知道问题出在哪里。因为目录确实存在。
英文:
This is from an SO-question some 4 years ago. It worked in Databricks Notebook.
%python
import pandas as pd
from io import StringIO
data = """
CODE,L,PS
5d8A,N,P60490
5d8b,H,P80377
5d8C,O,P60491
"""
df = pd.read_csv(StringIO(data), sep=',')
#print(df)
df.to_csv('/dbfs/FileStore/NJ/file1.txt')
pandas_df = pd.read_csv("/dbfs/FileStore/NJ/file1.txt", header='infer')
print(pandas_df)
Now it does not. Error message is:
FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/NJ/file1.txt'
Curious what the issue is. As the directory is there.
答案1
得分: 0
从DBR 7.x开始,由于某些原因(可能是出于安全考虑,但我记不清了),Databricks社区版已禁用/dbfs/
(所谓的fuse mount)。
当前的解决方法是使用dbutils.fs.cp
(或dbutils.fs.mv
)在DBFS和本地文件系统之间复制文件,并在本地处理文件(如果未指定URI方案,则默认为DBFS)。
从DBFS复制到本地文件系统:
dbutils.fs.cp("/path-on-dbfs", "file:/local-path")
从本地文件系统复制到DBFS:
dbutils.fs.cp("file:/local-path", "/path-on-dbfs")
英文:
Starting from DBR 7.x, the /dbfs/
(so-called fuse mount) is disabled on the Databricks community edition due to some reasons (maybe for security, but I don't remember).
The current workaround is to use dbutils.fs.cp
(or dbutils.fs.mv
) to copy files between DBFS & local file system, and working with files locally. (if URI scheme isn't specified, then it's DBFS by default).
To copy from DBFS to local file system:
dbutils.fs.cp("/path-on-dbfs", "file:/local-path")
To copy from local file system to DBFS:
dbutils.fs.cp("file:/local-path", "/path-on-dbfs")
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论