英文:
Optimal method for storing memes on Google Cloud in terms of costs, speed, and scalability
问题
我正在构建一个类似Reddit的应用程序,您可以在其中分享表情包(暂时不支持视频)。我在网上阅读了有关存储它们的最佳方法,发现选择主要取决于是否使用文件系统。另一个选择是像我目前所做的那样简单地将它们存储在数据库中。由于表情包大多不依赖于质量,我想知道是否值得重新设计系统以使用文件系统并在数据库中存储路径。
目前我是通过在MongoDB中以base64编码的方式存储它们的表示来处理的。我还发现这并不是最佳选择,因为base64编码相对于原始二进制数据会消耗大约33%更多的存储空间。因此,我考虑的第一个优化是存储BLOBs。
当前的计算大致如下 - 约300个表情包约30MB。
因此,每个表情包大约占用100KB,获取时间相当不错。所以我在考虑是否应该实现类似Google Buckets的东西?
英文:
I'm building a reddit-like app on which you can share memes (no videos for now). I read online on the best way to store them, and I found out that the choice comes down on whether you use file system or not. The other option is to simply store them in a database like I currently do. Since memes mostly aren't quality dependent images, I was wondering if it's worth to redesign the system to use file systems and paths stored in db.
The way I currently do it is by storing theirs representations base64-encoded in MongoDB. I also found out that this isn't optimal since base64 encoding consumes about 33% more storage space in comparison to raw binary. So the first optimization I'm thinking of is storing BLOBs
Current calculation is roughly as following - about 300 memes ~ 30mb
So we get about 100KB per meme, and fetching times are pretty good. So I'm wondering if I should implement something like google buckets?
答案1
得分: 0
以下是翻译好的部分:
"存储图像的最先进解决方案是使用Google存储(或AWS S3或类似服务)。它非常便宜且可扩展。
此外,您还可以在存储桶前添加Google Cloud CDN(或AWS CloudFront),进一步提高性能。
通过这种方式,您可以:
- 解除应用程序服务器的负担,不需要每次用户查看表情图时都查询数据库,这是一个很大的优势。
- 使用预签名URL上传图像的Google存储API。
但您还需要存储表情图的元数据(存储桶名称、存储桶中图像的路径、上传者、上传时间、与谁共享以及获得多少点赞等等)。
而这些数据可以存储在MongoDB中。"
英文:
The state of the art solution to store images is Google storage (or AWS S3 or similar). It's very cheap and scalable.
Furthermore you can add Goocle Cloud CDN (or AWS CloudFront) in front of the storage bucket and increase performance even further.
Doing this way you:
- offload your application servers from the need to query db every time user is viewing meme and this is a big gain.
- get out of the box Google Storage api to upload image with pre signed url
But you also need to store memes metadata (bucket name, the path to the image in bucket, who uploaded, when, with whom it's shared, how many likes it has, etc.).
And this data can be kept in MongoDB.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论