英文:
How to design ADF pipeline to split a container data into multiple containers of cosmos database
问题
我想创建一个ADF管道，将源容器（cosmos DB）的数据根据筛选条件移动到多个目标容器（cosmos DB），筛选条件如下所示。您能帮助我确定活动的顺序以及它们的表达式、变量或参数吗？非常感谢您的帮助，谢谢
| Source | target | Filter condition |
|---|---|---|
| Ingest | Sales | where c.source in ('salesOrderS4') |
| Ingest | Plan | where c.source in ('planning') |
| Ingest | Logic | where c.source in ('logisticsASN') |
示例源容器数据：
[
{
"id": "5961fc41-4508144870-90",
"source": "logisticsASN",
"blobTriggerStartTime": "2021-12-22T11:45:03.999Z",
"processEndTime": "2021-12-22T11:45:09.624Z"
},
{
"id": "1a0862e2-4301527947-20",
"source": "planning",
"blobTriggerStartTime": "2022-12-09T11:46:35.604Z",
"processEndTime": "2022-12-09T11:46:44.542Z"
},
{
"id": "bad34955-481d-60",
"source": "salesOrderS4",
"blobTriggerStartTime": "2022-06-01T00:23:23.090Z",
"processEndTime": "2022-06-01T00:23:27.965Z"
}
]
英文:
I want to create a ADF pipeline to move source container(cosmos DB) data into multiple target containers(cosmos DB) based on filter conditions (example below). can you please help me with the activities order and their expression, variables or parameters I need to use. your help is much appreciated, thanks
|Source | target | Filter condition |
|------ |-------- | ---------------- |
| Ingest | Sales | where c.source in ('salesOrderS4')|
| Ingest | Plan | where c.source in ('planning') |
| Ingest | Logic | where c.source in ('logisticsASN')|
Sample source container data :
[
{
"id": "5961fc41-4508144870-90",
"source": "logisticsASN",
"blobTriggerStartTime": "2021-12-22T11:45:03.999Z",
"processEndTime": "2021-12-22T11:45:09.624Z"
},
{
"id": "1a0862e2-4301527947-20",
"source": "planning",
"blobTriggerStartTime": "2022-12-09T11:46:35.604Z",
"processEndTime": "2022-12-09T11:46:44.542Z"
},
{
"id": "bad34955-481d-60",
"source": "salesOrderS4",
"blobTriggerStartTime": "2022-06-01T00:23:23.090Z",
"processEndTime": "2022-06-01T00:23:27.965Z"
}
]
答案1
得分: 1
将源容器数据拆分到目标容器上的筛选条件中，您应该拥有目标容器的列表以及要在条件中检查的字符串列表。
所以，我拿了一个示例的 Cosmos 源数据如下。这里我的 target 列与您的 source 列相同。
[
{
"id": "24",
"target": "销售",
},
{
"id": "26",
"target": "计划",
},
{
"id": "16",
"target": "逻辑",
}
]
我将容器列表和筛选字符串列表保存在一个 CSV 文件中，以便进行简单的比较。
在查找活动中使用这个，并将查找输出数组传递给 ForEach（检查顺序）。
在 ForEach 中，提供源 Cosmos DB 数据集，并在查询选项中使用以下查询：
select * from c where contains('@{item().filterstr}', c.target)
在接收端，对于 Cosmos DB 容器名称，请使用数据集参数，并像下面这样提供 @item().target。
执行这个操作，您可以看到数据复制到不同的容器中。
注意：复制活动不会自动创建目标容器。在管道运行之前，请确保已创建所需的容器。
英文:
To do split the source container data, into the target containers on a filter condition, you should have the list of target containers and strings that you want to check in condition.
So, I have taken a sample cosmos source data like below. Here my target column is same as your source column.
[
{
"id": "24",
"target": "Sales",
},
{
"id": "26",
"target": "Plan",
},
{
"id": "16",
"target": "logic",
}
]
I took the containers list and filter strings list in a csv file for the easy comparison.
Use this in a lookup activity and give the lookup output array to a Foreach(check the sequential).
Inside ForEach, give the source cosmos db dataset and use the below query in query option.
select * from c where contains('@{item().filterstr}',c.target)
In sink, use dataset parameter for the cosmos db container name and give the @item().target for it like below.
Execute this and you can see the data copied to different containers.
NOTE: Copy activity won't create the target containers automatically. Make sure the required containers created before the pipeline run.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论