将所有文件从S3存储桶移动到冷归档(Glacier Deep Archive),不包括一列文件。

huangapple go评论66阅读模式
英文:

Move all files from S3 bucket to Glacier Deep Archive excluding a list of files

问题

我有一个包含许多文件的存储桶,我需要将所有这些文件移动到 Glacier Deep Archive,但要排除一个文件列表,我需要将这些文件保留在它们当前的存储类别中。

最好的做法是什么?

我尝试了以下方法:

aws s3 cp s3://Bucket_Name/*.mp4 s3://Bucket_Name/*.mp4 --recursive --storage-class DEEP_ARCHIVE

我需要添加的是,如果文件存在于 files.txt(文件列表)中,那么将其保留在当前的存储类别中。

英文:

I have a bucket with a lot of files and I need to move all of them to Glacier Deep Archive
excluding a list of files that I need keep in their current storage class.

What is the best way to do it?

I used the following:

aws s3 cp s3://Bucket_Name/*.mp4 s3://Bucket_Name/*.mp4 --recursive --storage-class DEEP_ARCHIVE

I need to add if the file exist in files.txt (list of files) then keep it in current storage class.

答案1

得分: 1

Your current method 'copies' the object to itself while changing the Storage Class. Please note that objects under 128KB is the minimum size for objects in the Glacier Deep Archive storage class, so small objects might not change class.

使用AWS CLI很方便,但难以'exclude'选定的文件。与其使用AWS CLI,您可能想要使用您选择的编程语言中的AWS SDK(例如Python中的boto3)来进行Amazon S3的API调用。然后,您可以在脚本中编写逻辑以排除您的文件列表。

另一种方法是使用Amazon S3批处理操作来复制对象并同时更改存储类别。批处理操作以清单文件作为输入,其中列出了要'copy'的所有文件。清单文件可以通过使用Amazon S3 Inventory来创建,它可以提供每日或每周列出所有对象的CSV文件。

如果您正在使用批处理操作来更改存储类别,您可以编辑S3 Inventory创建的清单文件,并删除您不希望归档的文件

英文:

Your current method 'copies' the object to itself while changing the Storage Class. Please note that objects under 128KB is the minimum size for objects in the Glacier Deep Archive storage class, so small objects might not change class.

Using the AWS CLI is convenient, but can be difficult to 'exclude' selected files. Instead of using the AWS CLI, you might want to use an AWS SDK in a language of your choice (eg boto3 for Python) to make the API calls to Amazon S3. You could then code logic into your script to exclude your list of files.

An alternate method is to use Amazon S3 Batch Operations to copy objects while changing storage class. Batch Operations takes a manifest file as input that lists all files to 'copy'. The manifest file can be created by using Amazon S3 Inventory, which can provide a daily or weekly CSV file listing all objects.

If you are using Batch Operations to change storage class, you could edit the manifest file created by S3 Inventory and remove the files you do not wish to archive.

huangapple
  • 本文由 发表于 2023年5月11日 16:31:12
  • 转载请务必保留本文链接:https://go.coder-hub.com/76225618.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定