需要在Azure Data Factory中生成具有动态键的JSON文件。

huangapple go评论59阅读模式
英文:

Need to generate JSON file in Azure Data Factory with dynamic Key

问题

以下是翻译好的内容:

我在CSV文件中有以下数据。

ID userId Name
719A070E-4874-E811-9CCE-02152146006A 123 Joe
5d7d0a74-f1b9-483a-8e9f-c7f45ccfeda0 456 Mike

现在,我需要使用Azure数据工厂生成以下JSON,

[
    {
        "719A070E-4874-E811-9CCE-02152146006A":{
                 "userId":"123",
                  "Name":"Joe"
             }    
    },
    {
        "5d7d0a74-f1b9-483a-8e9f-c7f45ccfeda0":{
                 "userId":"456",
                  "Name":"Mike"
             }    
    }
 ]

我已经尝试了所有可能的方法,我能够创建内部JSON,使用用户ID和姓名。但我无法弄清楚如何设置JSON中的动态键,该键必须从CSV中的ID列派生。

英文:

I have a below data in csv file.

ID userId Name
719A070E-4874-E811-9CCE-02152146006A 123 Joe
5d7d0a74-f1b9-483a-8e9f-c7f45ccfeda0 456 Mike

Now, I need to generate below json using Azure Data Factory,

[
    {
        "719A070E-4874-E811-9CCE-02152146006A":{
                 "userId":"123",
                  "Name":"Joe"
             }    
    },
    {
        "5d7d0a74-f1b9-483a-8e9f-c7f45ccfeda0":{
                 "userId":"456",
                  "Name":"Mike"
             }    
    }
 ]

I have tried all the possible ways, I can able to form the inner json using userid and name. But I cannot able to figure out how to set the dynamic KEY in JSON which has to be derived from ID column in csv.

答案1

得分: 1

我可以使用ForEach和附加变量活动来实现您的要求。

我采用了与您的CSV相同的数据并将其提供给了一个查找活动(在查找活动中取消选中“第一行”复选框)。在CSV数据集中,请取消选中“第一行作为标题”的复选框。

这将是查找输出数组:

需要在Azure Data Factory中生成具有动态键的JSON文件。

然后,在变量中创建一个数组变量。创建一个ForEach活动,并为其提供以下表达式(在这里选中“Sequential”复选框):

@skip(activity('Lookup1').output.value, 1)

在ForEach内部,创建一个附加变量活动,其中包含以下表达式:

@json(concat('{"', item().Prop_0, '":{"', activity('Lookup1').output.value[0].Prop_1, '":"', item().Prop_1, '","', activity('Lookup1').output.value[0].Prop_2, '":"', item().Prop_2, '"}}'))

在这里,上述表达式在每次迭代中生成对象并附加到数组变量中(这是结果数组)。

为了显示输出,我使用了另一个数组变量,并在ForEach之后将结果数组的值分配给它。

需要在Azure Data Factory中生成具有动态键的JSON文件。

您可以参考以下是我的管道JSON:

{
    "name": "pipeline1",
    "properties": {
        "activities": [
            {
                "name": "Lookup1",
                "type": "Lookup",
                "dependsOn": [],
                "policy": {
                    "timeout": "0.12:00:00",
                    "retry": 0,
                    "retryIntervalInSeconds": 30,
                    "secureOutput": false,
                    "secureInput": false
                },
                "userProperties": [],
                "typeProperties": {
                    "source": {
                        "type": "DelimitedTextSource",
                        "storeSettings": {
                            "type": "AzureBlobFSReadSettings",
                            "recursive": true,
                            "enablePartitionDiscovery": false
                        },
                        "formatSettings": {
                            "type": "DelimitedTextReadSettings"
                        }
                    },
                    "dataset": {
                        "referenceName": "sourcecsv",
                        "type": "DatasetReference"
                    },
                    "firstRowOnly": false
                }
            },
            {
                "name": "ForEach1",
                "type": "ForEach",
                "dependsOn": [
                    {
                        "activity": "Lookup1",
                        "dependencyConditions": [
                            "Succeeded"
                        ]
                    }
                ],
                "userProperties": [],
                "typeProperties": {
                    "items": {
                        "value": "@skip(activity('Lookup1').output.value, 1)",
                        "type": "Expression"
                    },
                    "isSequential": true,
                    "activities": [
                        {
                            "name": "Append variable1",
                            "type": "AppendVariable",
                            "dependsOn": [],
                            "userProperties": [],
                            "typeProperties": {
                                "variableName": "arr",
                                "value": {
                                    "value": "@json(concat('{"', item().Prop_0, '":{"', activity('Lookup1').output.value[0].Prop_1, '":"', item().Prop_1, '","', activity('Lookup1').output.value[0].Prop_2, '":"', item().Prop_2, '"}}'))",
                                    "type": "Expression"
                                }
                            }
                        }
                    ]
                }
            },
            {
                "name": "Set variable1",
                "type": "SetVariable",
                "dependsOn": [
                    {
                        "activity": "ForEach1",
                        "dependencyConditions": [
                            "Succeeded"
                        ]
                    }
                ],
                "userProperties": [],
                "typeProperties": {
                    "variableName": "newarr",
                    "value": {
                        "value": "@variables('arr')",
                        "type": "Expression"
                    }
                }
            }
        ],
        "variables": {
            "arr": {
                "type": "Array"
            },
            "newarr": {
                "type": "Array"
            }
        },
        "annotations": []
    }
}
英文:

I am able to achieve your requirement using ForEach and append variable activities.

I took csv same data like yours and given it to a lookup activity(uncheck the first row in lookup activity). Here in the csv dataset, uncheck the first row as header checkbox.

This will be the lookup output array:

需要在Azure Data Factory中生成具有动态键的JSON文件。

Then, in the variables create an array variable. Create a ForEach activity and give the following expression for it(check the Sequential checkbox here).

@skip(activity('Lookup1').output.value, 1)

Inside ForEach, create an append variable activity with following expression in it.

@json(concat('{"',item().Prop_0,'":{"',activity('Lookup1').output.value[0].Prop_1,'":"',item().Prop_1,'","',activity('Lookup1').output.value[0].Prop_2,'":"',item().Prop_2,'"}}'))

需要在Azure Data Factory中生成具有动态键的JSON文件。

Here, the above expression generates the object in each iteration and appends to the array variable(This is result array).

For showing the output, I have used another array variable and assigned the result array value to it after Foreach.

需要在Azure Data Factory中生成具有动态键的JSON文件。

My Pipeline JSON for your reference:

{
"name": "pipeline1",
"properties": {
"activities": [
{
"name": "Lookup1",
"type": "Lookup",
"dependsOn": [],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"source": {
"type": "DelimitedTextSource",
"storeSettings": {
"type": "AzureBlobFSReadSettings",
"recursive": true,
"enablePartitionDiscovery": false
},
"formatSettings": {
"type": "DelimitedTextReadSettings"
}
},
"dataset": {
"referenceName": "sourcecsv",
"type": "DatasetReference"
},
"firstRowOnly": false
}
},
{
"name": "ForEach1",
"type": "ForEach",
"dependsOn": [
{
"activity": "Lookup1",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"items": {
"value": "@skip(activity('Lookup1').output.value, 1)",
"type": "Expression"
},
"isSequential": true,
"activities": [
{
"name": "Append variable1",
"type": "AppendVariable",
"dependsOn": [],
"userProperties": [],
"typeProperties": {
"variableName": "arr",
"value": {
"value": "@json(concat('{\"',item().Prop_0,'\":{\"',activity('Lookup1').output.value[0].Prop_1,'\":\"',item().Prop_1,'\",\"',activity('Lookup1').output.value[0].Prop_2,'\":\"',item().Prop_2,'\"}}'))",
"type": "Expression"
}
}
}
]
}
},
{
"name": "Set variable1",
"type": "SetVariable",
"dependsOn": [
{
"activity": "ForEach1",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"variableName": "newarr",
"value": {
"value": "@variables('arr')",
"type": "Expression"
}
}
}
],
"variables": {
"arr": {
"type": "Array"
},
"newarr": {
"type": "Array"
}
},
"annotations": []
}
}

huangapple
  • 本文由 发表于 2023年5月14日 01:37:14
  • 转载请务必保留本文链接:https://go.coder-hub.com/76244108.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定