How to replicate contents of a specific folder of a S3 bucket to another specific folder of a different S3 bucket

huangapple go评论57阅读模式
英文:

How to replicate contents of a specific folder of a S3 bucket to another specific folder of a different S3 bucket

问题

I have a requirement to replicate the contents of a specific folder of an S3 bucket to a different folder of another S3 bucket. Both S3 buckets reside in the same region and are owned by the same account.

Eg: Say I have an S3 bucket named "ABC-Inbound" and a folder inside called "from_A".
Also, I have another S3 bucket in the same region named workloads and a folder inside called "A_data".

Is it possible to write a Cloudformation replication rule to replicate the contents of the from_A folder to the A_data folder? I don't want to copy the entire folder, just the contents only.

Tried this by adding a Filter to replication configuration, but that didn't work as it copied the entire folder to the other bucket.

{
  "Role": "IAM-role-ARN",
  "Rules": [
    {
      "Status": "Enabled",
      "Priority": 1,
      "DeleteMarkerReplication": { "Status": "Disabled" },
      "Filter" : { "Prefix": "from_A/"},
      "Destination": {
        "Bucket": "arn:aws:s3:::workloads"
      }
    }
  ]
}
英文:

I have a requirement to replicate the contents of a specific folder of an S3 bucket to a different folder of another S3 bucket. Both S3 buckets reside in the same region and are owned by the same account.

Eg: Say I have an S3 bucket named "ABC-Inbound" and a folder inside called "from_A".
Also, I have another S3 bucket in the same region names workloads and a folder inside called "A_data"

Is it possible to write a Cloudformation replication rule to replicate the contents of the from_A folder to the A_data folder? I don't want to copy the entire folder just the contents only.

Tried this by adding a Filter to replication configuration, but that didn't work as it copied entire folder to other bucket.

    {
  "Role": "IAM-role-ARN",
  "Rules": [
    {
      "Status": "Enabled",
      "Priority": 1,
      "DeleteMarkerReplication": { "Status": "Disabled" },
      "Filter" : { "Prefix": "from_A/"},
      "Destination": {
        "Bucket": "arn:aws:s3:::workloads"
      }
    }
  ]
}

答案1

得分: 3

Amazon S3复制将在存储桶之间复制对象,并且您可以指定前缀以仅复制存储桶的一部分。

但是,在复制对象时,S3复制不会更改对象的“Key”(文件名)。键包括对象的完整路径。这意味着源和目标文件夹将相同。

如果您必须复制到目标存储桶中的不同文件夹/路径,那么您需要自行管理复制,例如:

  • 定期运行“aws s3 sync”命令以在存储桶之间复制文件,或
  • 创建一个AWS Lambda函数,该函数在源存储桶中创建对象时触发。编写Lambda函数以将对象复制到目标存储桶和路径。

以下是这样一个Lambda函数的示例:在S3存储桶之间复制文件

英文:

Amazon S3 Replication will replicate objects between buckets and you can specify a Prefix to only replicate part of a bucket.

However, when replicating objects, S3 Replication will not change the Key (filename) of an object. The Key includes the full path of the object. This means that the source and destination folders will be the same.

If you must replicate to a different folder/path in the destination bucket, then you would need to manage the replication yourself, such as:

  • Periodically run the aws s3 sync command to copy files between the buckets, or
  • Create an AWS Lambda function that is triggered upon creation of an object in the source bucket. Code the Lambda function to copy the object to the destination bucket and path.

Here is an example of such a Lambda function: Copying files between S3 buckets

huangapple
  • 本文由 发表于 2023年7月6日 16:49:40
  • 转载请务必保留本文链接:https://go.coder-hub.com/76627069.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定