Databricks代码不再工作,出现了“目录未找到”的错误。

huangapple go评论52阅读模式
英文:

Databricks code does not work anymore with 'directory not found' error

问题

这是来自四年前的一个Stack Overflow问题。它在Databricks Notebook中正常工作。

%python
import pandas as pd
from io import StringIO

data = """
CODE,L,PS
5d8A,N,P60490
5d8b,H,P80377
5d8C,O,P60491
"""

df = pd.read_csv(StringIO(data), sep=',')
#print(df)
df.to_csv('/dbfs/FileStore/NJ/file1.txt')

pandas_df = pd.read_csv("/dbfs/FileStore/NJ/file1.txt", header='infer') 
print(pandas_df)

现在不再起作用。错误消息是:

FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/NJ/file1.txt'

想知道问题出在哪里。因为目录确实存在。

英文:

This is from an SO-question some 4 years ago. It worked in Databricks Notebook.

%python
import pandas as pd
from io import StringIO

data = """
CODE,L,PS
5d8A,N,P60490
5d8b,H,P80377
5d8C,O,P60491
"""

df = pd.read_csv(StringIO(data), sep=',')
#print(df)
df.to_csv('/dbfs/FileStore/NJ/file1.txt')

pandas_df = pd.read_csv("/dbfs/FileStore/NJ/file1.txt", header='infer') 
print(pandas_df)

Now it does not. Error message is:

FileNotFoundError: [Errno 2] No such file or directory: '/dbfs/FileStore/NJ/file1.txt'

Curious what the issue is. As the directory is there.

答案1

得分: 0

从DBR 7.x开始,由于某些原因(可能是出于安全考虑,但我记不清了),Databricks社区版已禁用/dbfs/(所谓的fuse mount)。

当前的解决方法是使用dbutils.fs.cp(或dbutils.fs.mv)在DBFS和本地文件系统之间复制文件,并在本地处理文件(如果未指定URI方案,则默认为DBFS)。

从DBFS复制到本地文件系统:

dbutils.fs.cp("/path-on-dbfs", "file:/local-path")

从本地文件系统复制到DBFS:

dbutils.fs.cp("file:/local-path", "/path-on-dbfs")
英文:

Starting from DBR 7.x, the /dbfs/ (so-called fuse mount) is disabled on the Databricks community edition due to some reasons (maybe for security, but I don't remember).

The current workaround is to use dbutils.fs.cp (or dbutils.fs.mv) to copy files between DBFS & local file system, and working with files locally. (if URI scheme isn't specified, then it's DBFS by default).

To copy from DBFS to local file system:

dbutils.fs.cp("/path-on-dbfs", "file:/local-path")

To copy from local file system to DBFS:

dbutils.fs.cp("file:/local-path", "/path-on-dbfs")

huangapple
  • 本文由 发表于 2023年5月7日 19:17:29
  • 转载请务必保留本文链接:https://go.coder-hub.com/76193590.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定