英文:
Inserting value stored in a pipeline variable as a new row of sink dataset data
问题
在ADF中，我从各种活动中生成管道变量，例如管道运行时间、复制行数、复制状态等等。然后我使用“Set Variable”来创建一个包含上述数值的JSON字符串。现在我想将这个字符串存储到现有的CosmosDB中作为一个新记录。
我不能使用Azure Functions或Notebook作业来插入，因为我希望在ADF内部能够实现而不需要额外的依赖。
我已经尝试使用Data Flow，但它无法从管道变量中的值添加新记录。我不能使用单独的数据源来联合并添加到CosmosDB中，因为我想要插入的值是在运行时生成的。
英文:
In ADF, I generate pipeline variables such as the pipeline run time, rows copied count, copy status etc. from various activities. I then use Set Variable to create a JSON string consisting of the above values. Now I want to store this string into an existing CosmosDB as a new record.
I cannot use Azure Functions or a Notebook job to insert, as I want it to be possible within ADF without additional dependencies.
I have tried using Data Flow, but it is unable to add a new record from values in a pipeline variable. I cannot use a separate data source to union and add into the CosmosDB as the values I want inserted are generated during runtime.
答案1
得分: 0
你可以使用个别的管道变量，如pipeline run time，copied rows count，copy status，它们可以直接使用（无需创建包含所有值的JSON文档），然后在ADF中使用复制活动将这些值插入Cosmos DB容器。以下是具体步骤。
- 三个管道变量
pipeline run time，copied rows count，copy status被获取并在此处使用“设置变量活动”来设置它们的值。
- 然后，使用一个带有虚拟源数据集的复制活动。源数据集只有一行。在复制活动的源选项卡中，通过点击“+新建”来添加新列。所有三个列都使用“设置变量活动”定义的值进行设置。
- 接下来，为Cosmos DB选择了一个Sink数据集，写操作选择
insert。
- 然后，在映射部分，通过为管道变量提供示例值导入模式。然后将源中创建的所有附加列映射到目标列。
- 当管道运行时，数据将被插入到Cosmos DB中。
英文:
You can use the individual pipeline variables like pipeline run time , copied rows count, copy status as they are (without creating a json document consisting of all values) and use the copy activity in adf to insert the values in cosmos db container. Below is the approach.
- Three pipeline variables
pipeline run time,copied rows count,copy statusare taken and their values are set using set variable activity here.
- Then a copy activity is taken with a dummy source dataset. Source dataset has one row. In source tab of copy activity, New columns are added by clicking
+Newin additional columns. All three columns are set with the value that is defined using set variable activity.
- Then sink dataset is taken for cosmos dB
withinsertas write behavior.
- Then in Mapping section, Schema is imported by giving sample values for the pipeline variables. Then all additional columns that is created in source are mapped to sink columns.
- When pipeline is run, data is inserted to cosmos dB.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论