如何自动导入Azure Data Factory托管的Airflow中的新DAGs?

huangapple go评论60阅读模式
英文:

How to auto import new DAGs in Azure Data Factory Managed Airflow?

问题

我们从存储容器中导入文件后,如果有文件被修改、排除,或者容器中有新文件,自动导入是有道理的。

但目前这项任务需要手动操作,所以如果我在容器中修改了一个DAG,就必须一遍又一遍地转到Managed Airflow并执行“导入文件”。

是否有一种方法可以设置自动刷新同步或使用API构建等效的解决方案?

英文:

Once we imported files from a Container in Storage, it would make sense to auto import if any file is modified or excluded or if there is a new file in the Container.如何自动导入Azure Data Factory托管的Airflow中的新DAGs?

But currently this job is manually, so if I modify a DAG in the Container I have to go to Managed Airflow and do the "Import files" again and again.

如何自动导入Azure Data Factory托管的Airflow中的新DAGs?

Is there a way to set the auto refresh sync or an API to use to build an equivalent solution?

答案1

得分: 3

POST https://management.azure.com/subscriptions//resourcegroups//providers/Microsoft.DataFactory/factories//airflow/sync?api-version=2018-06-01

请求主体
{"IntegrationRuntimeName":"Airflow1","LinkedServiceName":"AzureBlobStorage1","StorageFolderPath":"airflow/","CopyFolderStructure":true,"Overwrite":true}

英文:

You can use the import DAG REST API for automation:

POST https://management.azure.com/subscriptions/<subscriptionid>/resourcegroups/<resourceGroupName>/providers/Microsoft.DataFactory/factories/<dataFactoryName>/airflow/sync?api-version=2018-06-01

Request body
{"IntegrationRuntimeName":"Airflow1","LinkedServiceName":"AzureBlobStorage1","StorageFolderPath":"airflow/","CopyFolderStructure":true,"Overwrite":true}

答案2

得分: 1

补充Abhishek Narain的正确答案,DAG REST API遵循标准的HTTP 202模式,因此响应的头部包含“Location” URI,可用于通过GET请求轮询导入的状态。

英文:

To add to Abhishek Narain's correct answer, the DAG REST API follows the standard HTTP 202 pattern, so the headers of the response contains the "Location" URI that can be polled with a GET request for the status of the import.

huangapple
  • 本文由 发表于 2023年3月21日 00:46:34
  • 转载请务必保留本文链接:https://go.coder-hub.com/75793062.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定