将Deep Glacier S3数据从一个AWS帐户复制到另一个帐户。

huangapple go评论60阅读模式
英文:

Copy Deep Glacier S3 Data From One AWS account to another account

问题

我有4个S3存储桶,其中包含近40TB的数据,其中90%是深度归档S3层级。

我需要将这些数据迁移到另一个AWS账户,因为我计划关闭我的账户。

> 注意:两个S3存储桶都位于ap-south-1区域。

建议一种简便、经济、快速的方法来完成这个任务,因为当我尝试使用AWS S3 CLI命令时,处理这么大量的数据需要好几天。

英文:

I have 4 S3 Buckets, in which I have almost 40Tbs of data which includes 90% of Deep archive S3 Tiers.

I need to move this data to another AWS Account since am planning to close my account.

> NOTE: Both s3 is in the region ap-south-1

Suggest a way to do this easily and cost-friendly and fast, because when I tried with AWS s3 CLI commands, it takes days to complete this much amount of data.

答案1

得分: 1

由于您想要复制现有的S3对象,您可以执行Amazon S3对象上的大规模批处理操作 - Amazon Simple Storage Service

您的步骤将是:

  • 使用Amazon S3库存生成列出所有现有对象的清单文件(如果只想复制对象的子集,则可以编辑它)
  • 使用S3批处理操作暂时还原Glacier对象(会产生费用),提供列出要还原的对象的清单文件
  • 使用S3批处理操作将对象复制到另一个帐户中的S3存储桶

目标存储桶将需要一个允许S3批处理操作写入该存储桶的存储桶策略

参见:使用Amazon S3批处理操作进行跨帐户大规模文件传输 | AWS存储博客

由于存储桶位于同一区域,不会产生数据传输费用。然而,将会产生还原存储在Glacier存储类别中的对象的费用。还会针对读取源对象和写入目标对象产生费用。这些费用将基于API请求的数量计算,如果您有大量小对象,则费用会更高。

英文:

Since you are wanting to copy existing S3 objects, you can Perform large-scale batch operations on Amazon S3 objects - Amazon Simple Storage Service.

Your steps would be:

  • Use Amazon S3 Inventory to produce a manifest file listing all existing objects (and edit it if you only wish to copy a subset of objects)
  • Use S3 Batch Operations to temporarily restore the Glacier objects (charges apply) by supplying the manifest file that lists the objects to restore
  • Use S3 Batch Operations to copy the objects to the S3 bucket in the other account

The destination bucket will require a Bucket Policy that permits S3 Batch Operations to write to that bucket.

See: Cross-account bulk transfer of files using Amazon S3 Batch Operations | AWS Storage Blog

Data Transfer charges will not apply since the buckets are in the same Region. However, costs will be incurred for restoring the objects stored in the Glacier Storage Class. Charges will also apply for reading the source objects and writing the destination objects. These charges will be based on the number of API requests, so it will be more expensive if you have lots of small objects.

huangapple
  • 本文由 发表于 2023年6月19日 19:50:50
  • 转载请务必保留本文链接:https://go.coder-hub.com/76506339.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定