英文:
Trying to Find Variable in Databricks Apache Table Fails- Why?
问题
org.apache.spark.SparkException: 无法获取默认数据库的表格
... 尝试运行以下代码时出现此错误:
databaseName = "database"
desiredColumn = "work_unit_code"
database = spark.sql(f"show tables in {databaseName} ").collect()
display(database)
tablenames = []
for row in database:
cols = spark.table(row.tableName).columns
**listColumns= spark.table(row.tableName).columns**
if desiredColumn in listColumns:
tablenames.append(row.tableName)
有人知道发生了什么吗？ 请提前感谢。
尝试在特定数据库表格中汇总总列，然后迭代列，看看是否存在特定变量。
英文:
I am running Apache Spark 3.3.0, Scala 2.12, and get this error:
org.apache.spark.SparkException: Unable to fetch tables of db default
... when trying to run this code:
databaseName = "database"
desiredColumn = "work_unit_code"
database = spark.sql(f"show tables in {databaseName} ").collect()
display(database)
tablenames = []
for row in database:
cols = spark.table(row.tableName).columns
**listColumns= spark.table(row.tableName).columns**
if desiredColumn in listColumns:
tablenames.append(row.tableName)
Does anyone know what's going on, please? Thanks in advance.
Trying to aggregate total columns within a specific database table, then iterate through columns to see if a specific variable exists there.
答案1
得分: 1
因为tableName列只包含表名，而不包括数据库名，所以会发生这种情况。
将spark.table(row.tableName)修改为spark.table(f"{databaseName}.{row.tableName}")。
英文:
This happens because the tableName column contains just the table name, but not the database name.
Change spark.table(row.tableName) to the spark.table(f"{databaseName}.{row.tableName})
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论