Adf copy activity

I have 3 files at destination a.txt, b.txt and c.txt. however a.txt and c.txt is already present in my destination (adls) and my copy activity should only copy the remaining b.txt file. How can i achieve this using adf pipeline activities.

I have tried using get metadata for both source and destination if files exist at destination or not. But while copying inside if activity it copies all the files. Not the ones which are not present.

1. use 2 get meta data activities in parallel to get the child items list of the source and sink
2. post success of both get meta data activity, use filter activity with
   items as source child items array
   and condition as
   @not(contains(child items of sink array,item()))
3. then leverage For each activity with filter activity output as input for iteration and copy the missing files.

I agree with what @Nandan has suggested. The following is the demonstration of the same approach. The following is the output of the get metadata activity on source.

• And my sink has files as shown in the image below:

• Using filter, get the filenames that are not present in your sink and then use for each to copy these filtered files. The following dynamic content can be used as filter condition:

items: @activity('source file list').output.childItems
condition: @not(contains(activity('sink file list').output.childItems,item()))

• Now, you can iterate through this filtered list using @activity('Filter1').output.Value as items for your for each activity. The following is the entire pipeline JSON for the above implementation:

{
"name": "pipeline2",
"properties": {
"activities": [
{
"name": "source file list",
"type": "GetMetadata",
"dependsOn": [],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"dataset": {
"referenceName": "DelimitedText1",
"type": "DatasetReference"
},
"fieldList": [
"childItems"
],
"storeSettings": {
"type": "AzureBlobFSReadSettings",
"enablePartitionDiscovery": false
},
"formatSettings": {
"type": "DelimitedTextReadSettings"
}
}
},
{
"name": "sink file list",
"type": "GetMetadata",
"dependsOn": [],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"dataset": {
"referenceName": "DelimitedText2",
"type": "DatasetReference"
},
"fieldList": [
"childItems"
],
"storeSettings": {
"type": "AzureBlobFSReadSettings",
"enablePartitionDiscovery": false
},
"formatSettings": {
"type": "DelimitedTextReadSettings"
}
}
},
{
"name": "Filter1",
"type": "Filter",
"dependsOn": [
{
"activity": "source file list",
"dependencyConditions": [
"Succeeded"
]
},
{
"activity": "sink file list",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"items": {
"value": "@activity('source file list').output.childItems",
"type": "Expression"
},
"condition": {
"value": "@not(contains(activity('sink file list').output.childItems,item()))",
"type": "Expression"
}
}
},
{
"name": "ForEach1",
"type": "ForEach",
"dependsOn": [
{
"activity": "Filter1",
"dependencyConditions": [
"Succeeded"
]
}
],
"userProperties": [],
"typeProperties": {
"items": {
"value": "@activity('Filter1').output.Value",
"type": "Expression"
},
"isSequential": true,
"activities": [
{
"name": "Copy data1",
"type": "Copy",
"dependsOn": [],
"policy": {
"timeout": "0.12:00:00",
"retry": 0,
"retryIntervalInSeconds": 30,
"secureOutput": false,
"secureInput": false
},
"userProperties": [],
"typeProperties": {
"source": {
"type": "DelimitedTextSource",
"storeSettings": {
"type": "AzureBlobFSReadSettings",
"recursive": true,
"enablePartitionDiscovery": false
},
"formatSettings": {
"type": "DelimitedTextReadSettings"
}
},
"sink": {
"type": "DelimitedTextSink",
"storeSettings": {
"type": "AzureBlobFSWriteSettings"
},
"formatSettings": {
"type": "DelimitedTextWriteSettings",
"quoteAllText": true,
"fileExtension": ".txt"
}
},
"enableStaging": false,
"translator": {
"type": "TabularTranslator",
"typeConversion": true,
"typeConversionSettings": {
"allowDataTruncation": true,
"treatBooleanAsNumber": false
}
}
},
"inputs": [
{
"referenceName": "DelimitedText3",
"type": "DatasetReference"
}
],
"outputs": [
{
"referenceName": "DelimitedText4",
"type": "DatasetReference"
}
]
}
]
}
}
],
"annotations": []
}
}

• Since the end goal here would result in sink having all the files that source has, another way that you can consider is to use a delete activity inside your for each to delete the file in sink first and then copy it.

• First get the list of files present in source using get metadata activity. Iterate through this list, first apply delete operation on the file and then copy.

NOTE: If the content of the file that already exist in both source and sink is different, go with the approach demonstrated above (as suggest by @Nandan).