英文:
S3 Cost-effectively upload and download many small objects
问题
我在S3中存储了许多（>10M）小对象。每个对象都相当小（〜5KB）。
尽管存储成本可接受，但我意识到请求成本变得非常高。
上传这10M个对象的成本为50美元，有点昂贵。
有没有一种更具成本效益的方式来上传新数据？我也愿意使用其他服务。使用情况大致如下：每年上传新版本，可能每月下载一次。
我在研究时发现了这个问题，但它是10年前的，所以我想知道是否有什么变化。
英文:
I store many (>10M) small objects in S3. Each object is rather small (~5KB).
While the storage costs are acceptable, I realised that request costs are becoming very high.
Uploading the 10M objects costs 50$, which is a bit expensive.
Is there a way to upload new data more cost-effectively? I'm open to using other services as well. The usage is roughly: upload new versions once a year, download potentially once a month.
I found this question when researching, but it is from 10 years ago, so I was wondering if something changed.
答案1
得分: 1
平均大小为5KB对于S3来说相对较小。您将面临延迟、性能和成本方面的问题。考虑以下两种解决方案之一：
- 将数据合并成更大的块（S3在数百MB范围内最优）。您可以继续发布小数据包，但定期将它们合并成稍后使用的较大对象。
- 使用更适合您需求的不同存储引擎。考虑DynamoDB、Cassandra、Couchbase，甚至是PostgreSQL、MySQL，根据您的要求而定。10M*5KB=50GB，这很小，如果不需要持久性，甚至可以存储在Redis中。
英文:
The average size 5kb is rather small for S3. You'll face latency, performance, and cost issues down the road. Consider either of 2 solutions:
- Consolidate data into bigger chunks (S3 is optimal at hundreds of megabytes range). You can keep publishing with small packets of data, but regularly aggregate them into bigger objects used later.
- Use a different storage engine which suits your needs better. Consider DynamoDB, Cassandra, Couchbase, or even PostgreSQL, MySQL, depends on your requirements. 10M*5kb=50GB which is tiny, you can store it even in Redis if you don't need persistence.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论