英文:
Using TransferManager for uploading a large number of small sized objects
问题
我有大量的文件（约100万个），每个文件大小约为2KB。我目前正在使用Java中的s3Client的PutObject方法将每个文件上传到S3存储桶。但是所有文件的上传时间非常长（约4-5小时）。我已经了解了TransferManager，知道它可以用于使用multipart上传大文件。但这适用于至少5MB大小的文件，不符合我的用例。是否有办法可以使用TransferManager来提高上传速度？如果不能，我还能做些什么？
英文:
I have a large number of files (~ 1 million) and each of them is around 2KB in size. I am currently using s3Client's PutObject method in Java. to upload each of the file to an s3 bucket. But the upload for all the files is taking very long (~ 4-5 hours). I have read about TransferManager and know that it can be used to upload large sized objects using multipart. But this is for files that are atleast 5MB in size which does not satisfy my use case. Is there any way I can use TransferManager to increase the upload speed? If not, what else can I do?
答案1
得分: 1
如果您正在使用HTTPS请求上传文件，可以尝试使用多个前缀，还可以拥有多个应用程序的副本，以便分担负载并减少时间。另一种解决方案是，如果您未来不打算读取单个文件，可以将数据分成多个部分并进行上传。您也可以参考AWS文档。
AWS S3性能优化
英文:
If you are using https requests to upload your file, you can try using multiple prefixes, also you can have multiple replicas of the application so that load can be divided and time will decrease. Other solution is if you are not intending to read individual file in future, you can zip the data in multiple part and upload. You can refer aws docs also.
aws s3 performance optimization
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论