为什么我的Azure数据工厂数据流失败,出现’store is not defined’错误?

huangapple go评论66阅读模式
英文:

Why is my Azure Data Factory data flow failing with 'store is not defined' error?

问题

在执行数据流时出现了此错误。我已正确提供了所有数据流参数,如PipelineStartTime、PipelineEndTime等。

作业失败原因:com.microsoft.dataflow.Issues:DF-ARG-007 - 未定义存储 - EXE-0001,无法将Dataflow分析为图形,[564 737],
source(
) -> dummy
DF-EXPR-012 - 缺少FileName的参数值 - [418 464 564 737],com.microsoft.dataflow.Parameter@597aa1ed,EXE-0001,无法将Dataflow分析为图形,
dummy derive(
) -> derivedColumn1
DF-EXPR-012 - 缺少PipelineStartTime的参数值 - [418 473 564 737],EXE-0001,无法将Dataflow分析为图形,
dummy derive(
) -> derivedColumn1,com.microsoft.dataflow.Parameter@5954ed58
DF-EXPR-012 - 缺少PipelineEndTime的参数值 - EXE-0001,无法将Dataflow分析为图形,com.microsoft.dataflow.Parameter@1819981a,
dummy derive(
) -> derivedColumn1,[418 473 564 737]
DF-EXPR-012 - 缺少NoRowsRead的参数值 - [418 473 564 737],EXE-0001,无法将Dataflow分析为图形,com.microsoft.dataflow.Parameter@47c44f0c,
dummy derive(
) -> derivedColumn1
DF-EXPR-012 - 缺少Status的参数值

我做了什么?

我创建的数据流以复制活动的值作为参数,并且我正确提供了所有参数值,如PipelineStartTime、PipelineEndTime、Status、Errors等。但是数据流失败并显示此错误。
在将错误变量作为参数之前,它运行得很正常,但在那之后开始出现此类错误。

对于错误变量,我提供了以下值:

@concat('"',replace(substring(activity('Copy data1').output.errors[0].Message, 0,100 ),'\r\n|\r|\n',''),'"')

如何解决这个问题?

英文:

Getting this error while executing the data flow. I have supplied all the data flow parameters correctly such as PipelineStartTime, PipelineEndTime.

Job failed due to reason: com.microsoft.dataflow.Issues: DF-ARG-007 - store is not defined - EXE-0001,Dataflow cannot be analyzed as a graph,[564 737],
source(
) ~> dummy
DF-EXPR-012 - Parameter value for FileName missing - [418 464 564 737],com.microsoft.dataflow.Parameter@597aa1ed,EXE-0001,Dataflow cannot be analyzed as a graph,
dummy derive(
) ~> derivedColumn1
DF-EXPR-012 - Parameter value for PipelineStartTime missing - [418 473 564 737],EXE-0001,Dataflow cannot be analyzed as a graph,
dummy derive(
) ~> derivedColumn1,com.microsoft.dataflow.Parameter@5954ed58
DF-EXPR-012 - Parameter value for PipelineEndTime missing - EXE-0001,Dataflow cannot be analyzed as a graph,com.microsoft.dataflow.Parameter@1819981a,
dummy derive(
) ~> derivedColumn1,[418 473 564 737]
DF-EXPR-012 - Parameter value for NoRowsRead missing - [418 473 564 737],EXE-0001,Dataflow cannot be analyzed as a graph,com.microsoft.dataflow.Parameter@47c44f0c,
dummy derive(
) ~> derivedColumn1
DF-EXPR-012 - Parameter value for Status missing

What have I done

The data flow that I created takes the values from the copy activity as parameters and I supplied all the parameter values correctly such as PipelineStartTime, PipelineEndTime,Status, Errors etc. But the data flow is failing by giving this error.
Before giving the errors variable as parameter it was working fine but after that it started to give such errors.

For the errors variable I supplied this value:

@concat('"',replace(substring(activity('Copy data1').output.errors[0].Message, 0,100 ),'\r\n|\r|\n' ,'' ),'"')

How to resolve this?

答案1

得分: 0

我遇到了相同的错误(DF-ARG-007 - 未定义的存储)。在我的情况下,我想将包含单引号的名称传递给管道 For-Each 循环中的数据流(字符串参数)。经过几个小时的检查和测试,我现在认为 Data Factory 可能会搞乱它的内部 JSON 编码。

摘要
我需要处理的名称字段包含一个引号,例如 "Miles O'Brian":

症状
在监视器中显示如下(请注意名称的双引号):

{
"dataflow": {
    "referenceName": "PostProcessContact",
    "type": "DataFlowReference",
    "parameters": {
        "accountid": "'12345678-abcd-1234-abcd-123456789abc'",
        "name": "'Miles O''Brian'",
        ...
    },
    ...
}

错误

由于原因失败的作业:com.microsoft.dataflow.Issues: DF-ARG-007 - 未定义的存储 -
source(
) ~> dummysource,EXE-0001,无法将 Dataflow 分析为图形,[564 737]
DF-ARG-007 - 未定义的存储 - EXE-0001,无法将 Dataflow 分析为图形,
操作 sink(
) ~> parentaccountid,[564 737]

潜在的相关 DF-activity
在所调用的流程中,我使用了一个没有记录的虚拟源,因为我只想根据给定的参数构建一条记录。

"source(useSchema: false,",
"     allowSchemaDrift: true,",
"     validateSchema: false,",
"     inferDriftedColumnTypes: true,",
"     ignoreNoFilesFound: true,",
"     format: 'json',",
"     fileSystem: 'power-platform-dataflows',",
"     fileName: 'nuxnildummy.json',",
"     documentForm: 'singleDocument') ~> dummysource",

解决方法
在我的情况下,我通过将输入字符串中的单引号替换为 '(字符145)来解决了这个问题。这不是一个很好的解决方案,但它有效,并且我只是为了参考而传递了名称。

// 之前的赋值:
"'@item()['name']'",

// 修正后的赋值:
"'@{if(equals(item()['name'], null), null, replace(item()['name'], '''', '‘'))}'",

结论
虽然在我的情况下解决方法还可以,但这仍然是一个糟糕的实现,因此 @Microsoft:

  • 如果在将来的更新中解决此问题,将会很棒。
  • 如果添加一个内存中的 JSON 源将会很棒,它不需要您在 Flowlets 中创建虚拟源(例如将一系列记录 [{}, {}, ...] 转换为可用的表)。
英文:

I encountered the same error (DF-ARG-007 - store is not defined). In my case, I wanted to hand over a name containing a single quote to a data flow from a pipeline For-Each loop (string parameter). After hours of checking and testing I'd now blame Data Factory to potentially screw up it's internal JSON encoding.

Summary
The name field I needed to process contained a quote, e.g. "Miles O'Brian":

Symptom
Inside monitor something like below is displayed (note the double-quotation of name):

{
"dataflow": {
    "referenceName": "PostProcessContact",
    "type": "DataFlowReference",
    "parameters": {
        "accountid": "'12345678-abcd-1234-abcd-123456789abc'",
        "name": "'Miles O''Brian'",
        ...
    },
    ...
}

Error

> Job failed due to reason: com.microsoft.dataflow.Issues: DF-ARG-007 - store is not defined -
source(
) ~> dummysource,EXE-0001,Dataflow cannot be analyzed as a graph,[564 737]
DF-ARG-007 - store is not defined - EXE-0001,Dataflow cannot be analyzed as a graph,
operation sink(
) ~> parentaccountid,[564 737]

Potential, related DF-activity
In the called flow I am using a dummy source without records as I want to build a record solely on the parameters given.

"source(useSchema: false,",
"     allowSchemaDrift: true,",
"     validateSchema: false,",
"     inferDriftedColumnTypes: true,",
"     ignoreNoFilesFound: true,",
"     format: 'json',",
"     fileSystem: 'power-platform-dataflows',",
"     fileName: 'nuxnildummy.json',",
"     documentForm: 'singleDocument') ~> dummysource",

Workaround
In my case, I was able to solve the problem by replacing the single quote in my input string with ‘ which is char(145). Not a nice solution but it works and I carry over the name just for reference.

// Previous Assignment: 
"'@item()['name']'",

// Fixed Assignment:
"'@{if(equals(item()['name'], null), null, replace(item()['name'], '''', '‘'))}'",

Conclusion
Although the workaround is OK in my situation, it is still a bad implementation, therefore @Microsoft:

  • Would be great, if this issue will be addressed in a future update.
  • Would be great, if an in-memory JSON source would be added which doesn't require one to create dummy-sources in Flowlets (e.g. casting an array of records like [{}, {}, ...] into a useable table).

huangapple
  • 本文由 发表于 2023年6月1日 21:16:14
  • 转载请务必保留本文链接:https://go.coder-hub.com/76382306.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定