英文:
Does ADF dataflow reads the source multiple times?
问题
如果我从数据流源（比如cosmosDB）读取数据，并根据筛选条件将源数据分成两个数据集流，数据流是否会两次读取源数据以应用两个筛选条件？由于数据流使用Spark，我认为除非您将源数据保存在Spark集群中，否则必须两次读取。ADF数据流是否也会保存中间转换结果？
英文:
If I read from dataflow source, say cosmosDB, and branch the source data into two streams of dataset based on filter conditions, does dataflow reads the source twice to apply two filter conditions ? As dataflow uses spark, I believe unless you save the source data in spark cluster, the read has to happen twice. Does ADF dataflow saves intermediate transformations as well ?
答案1
得分: 1
如果您使用两次源转换添加相同的源数据，那么它将会像下面这样读取源数据两次：
如果您使用条件拆分或筛选转换来拆分数据，它不会读取两次。它将视为单一源数据。
英文:
If you Add the same source twice using two source transformation , then only it will read the source data twice like below:
If you use conditional split, or filter transformation, and split the data, it wont read it twice. It will consider it as single source
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论