英文:
Unknown parameter in Targets DeltaTargets must be one of: S3Targets, JdbcTargets, MongoDBTargets, DynamoDBTargets, CatalogTargets
问题
以下是您提供的代码的翻译部分：
import boto3
bucket_name = "data-extract"
table_path = "BatchTable"
s3_target = f"s3://{bucket_name}/datalake/{table_path}"
glue_client = boto3.client('glue')
response = glue_client.create_crawler(
Name=f"{bucket_name}-{table_path}-crawler_test",
Role='AWSGlueServiceRoleDefault',
DatabaseName='crispy',
Targets={"DeltaTargets": [{"DeltaTables": "s3://data-extract/datalake/BatchTable/","CreateNativeDeltaTable": True}]}
)
请注意，我已将双引号 " 替换为正常的双引号 "，以使代码正常运行。
英文:
Ok so i am trying to create a glue crawler that will crawler over the delta path and create a delta table, in the official AWS SDK documentation the syntax is written and I'm following as it is
AWS SDK Documentation
Here is my code:
import boto3
bucket_name = "data-extract"
table_path = "BatchTable"
s3_target = f"s3://{bucket_name}/datalake/{table_path}"
glue_client = boto3.client('glue')
response = glue_client.create_crawler(
Name=f"{bucket_name}-{table_path}-crawler_test",
Role='AWSGlueServiceRoleDefault',
DatabaseName='crispy',
Targets={"DeltaTargets": [{'DeltaTables': "s3://data-extract/datalake/BatchTable/",'CreateNativeDeltaTable': True}]})
答案1
得分: 1
OK，经过长时间的搜索，我发现 boto3 版本是问题的根本原因，通常在 Glue 3.0 中，它使用的是 boto3 版本 1.18.10，而在这个版本中不支持 "DeltaTargets"，所以要使用它，我们需要至少使用版本大于 1.24.0 的 boto3。要完全使用所有的 Delta 功能，请使用 boto3 版本 1.28.4。将作业更改为 Glue 4.0 成功创建了 Delta 表的爬虫。
英文:
OK so after so much look around i found out the boto3 version was the culprit, normally in Glue 3.0 its using the boto3 version of 1.18.10 and in this version the "DeltaTargets" are not supported, so to use this we need at least a boto3 version of > 1.24.0. Also to completely use all the delta features use the boto3 version of 1.28.4. Changing the job to Glue 4.0 successfully created a crawler for delta table.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论