如何修改 Azure 数据工厂的复制活动中的源列

huangapple go评论73阅读模式
英文:

How to modify source column in Copy Activity of Azure Data Factory

问题

我正在尝试将CSV文件中的数据复制到数据库表中。我在Azure数据工厂中创建了一个管道,并添加了“复制活动”。我设置了源数据集和目标数据集。

在复制操作期间,我希望在源CSV文件中的列Col1的值前缀一个固定字符串。在UI的“映射”部分底部,有一个名为“添加动态内容”的选项。我猜我可以在这里添加一个表达式来进行这种映射,但我找不到如何操作的任何信息。

完成这个任务的正确方法是什么?

英文:

I am trying to copy data from a csv file into a database table. I created a pipeline in Azure Data Factory and added a Copy Activity. I set the source and the destination datasets.

During the copy operation, I want to prefix the values of the column Col1 in the source csv file with a fixed string. At the bottom of the Mapping section in the UI, there is an option such as "Add dynamic content". I guess I would be able to add an expression here for such a mapping but I could not find any information on how to do that.

What is the correct way of accomplishing this?

答案1

得分: 2

"Adding a dynamic column value in copy activity supports only the ADF parameter /function values which would be common for all rows.

As of now there is no direct support to modify a column at row by row level in copy activity.

You can do it via 2 ways:

  1. use copy activity to copy into staging table and then use an SP activity to modify from staging table to final tables
  2. use dataflow derived column transformation

https://learn.microsoft.com/en-us/azure/data-factory/data-flow-derived-column"

英文:

Adding a dynamic column value in copy activity supports only the ADF parameter /function values which would be common for all rows.

As of now there is no direct support to modify a column at row by row level in copy activity.

如何修改 Azure 数据工厂的复制活动中的源列

You can do it via 2 ways:

  1. use copy activity to copy into staging table and then use an SP activity to modify from staging table to final tables
  2. use dataflow derived column transformation

https://learn.microsoft.com/en-us/azure/data-factory/data-flow-derived-column

答案2

得分: 1

  • @Zorkolot@Nandan 指示,一般情况下,数据复制活动是用于将文件/文件夹从不同来源移动到不同目的地的处理。
  • 由于要在现有列中添加固定字符串前缀的要求,您需要使用数据流。数据流可以帮助您按需操作/转换数据。
  • 您可以使用如下所示的派生列转换,将 pre 字符串添加为 id 列的前缀。

如何修改 Azure 数据工厂的复制活动中的源列

  • 另一种方法是使用一个数据复制活动和一个脚本活动将数据复制到数据库,并编写一个带有查询前缀的 concat 函数的更新查询,类似以下方式:
update t1 set <your_col>=concat('pre',<your_col>)

如何修改 Azure 数据工厂的复制活动中的源列

  • 另一种方法是使用Python笔记本将前缀添加到所需列,然后将其移动到数据库。
英文:
  • As directed by @Zorkolot and @Nandan, copy data activity in general is to deal with moving files/folders from different sources to different sinks.
  • Since the requirement is to add a fixed string prefix to an existing column, you need to use dataflows. Dataflows can help with manipulating/converting data as required.
  • You can use derived column transformation as shown in the below. I have added pre string as a prefix to id column.

如何修改 Azure 数据工厂的复制活动中的源列

  • Another way is to use one copy data activity and a script activity to copy to the database and write an update query with concat function on the required column with prefix with a query like this:
update t1 set <your_col>=concat('pre',<your_col>)

如何修改 Azure 数据工厂的复制活动中的源列

  • Another way would be to use Python notebook to add the prefix to required column and then move it to database.

huangapple
  • 本文由 发表于 2023年4月11日 02:05:03
  • 转载请务必保留本文链接:https://go.coder-hub.com/75979543.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定