ADF:在ForEach循环内部的复制活动是否多次访问源数据库?

huangapple go评论63阅读模式
英文:

ADF: Does a Copy activity within a ForEach loop hit the source DB multiple times?

问题

I need to upsert an entire table from a source DB to multiple destination DBs (all Azure SQL). I have a Lookup activity that pulls the list of server/DB names and passes it to a ForEach loop, inside which I've placed a Copy Data activity with a static dataset as the Source and a dynamic Destination.

My question is this: Will this arrangement hit the source DB every time the loop runs? I really only need to pull the data once since the exact same data needs to be upserted to all the destination DBs. Is there a way to pull the data one time and dump it to memory somehow instead of going back to the source DB every time the loop runs? Or by definition, does my use of a static dataset as the Source mean that the data is only pulled once per pipeline run?

Does any of this make any sense? ADF:在ForEach循环内部的复制活动是否多次访问源数据库?

英文:

I need to upsert an entire table from a source DB to multiple destination DBs (all Azure SQL). I have a Lookup activity that pulls the list of server/DB names and passes it to a ForEach loop, inside which I've placed a Copy Data activity with a static dataset as the Source and a dynamic Destination.

My question is this: Will this arrangement hit the source DB every time the loop runs? I really only need to pull the data once since the exact same data needs to be upserted to all the destination DBs. Is there a way to pull the data one time and dump it to memory somehow instead of going back to the source DB every time the loop runs? Or by definition, does my use of a static dataset as the Source mean that the data is only pulled once per pipeline run?

Does any of this make any sense? ADF:在ForEach循环内部的复制活动是否多次访问源数据库?

答案1

得分: 1

在Azure数据工厂中,使用静态数据集作为“复制数据活动”中的“源”意味着数据将仅在管道运行时拉取一次,而不是每次循环运行时。这种行为是有意设计的,它确保数据仅从源读取一次,然后在“ForEach”循环的每个迭代中进行处理,而不会多次访问源数据库。

因此,您可以放心,数据将仅在管道运行开始时从源数据库中提取一次,并存储在内存中供所有ForEach循环的迭代使用,其中它将被更新到多个目标数据库中。

英文:

In Azure Data Factory, using a static dataset as the Source in the Copy Data activity means that the data will be pulled only once per pipeline run, not every time the loop runs. This behavior is by design, and it ensures that the data is read from the source once and then processed for each iteration of the ForEach loop without hitting the source DB multiple times.

So, you can rest assured that the data will be fetched from the source database only once at the beginning of the pipeline run and will be stored in memory for use in all the iterations of the ForEach loop, where it will be upserted to the multiple destination databases.

huangapple
  • 本文由 发表于 2023年8月5日 06:26:18
  • 转载请务必保留本文链接:https://go.coder-hub.com/76839398.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定