我应该使用redis来存储大量二进制文件吗? [英] Should I use redis to store a large number of binary files?

问题描述

我需要存储大量二进制文件(10 - 20 TB，每个文件大小从 512 kb 到 100 MB).

I need to store huge amount of binary files (10 - 20 TB, each file ranging from 512 kb to 100 MB).

我需要知道 Redis 对我的系统是否有效.我的系统需要以下属性:

I need to know if Redis will be efficient for my system. I need following properties in my system:

• 高可用性
• 故障转移
• 分片

我打算使用一组商品硬件来尽可能地降低成本.请建议使用 Redis 构建这样一个系统的利弊.我也很关心Redis对内存的高要求.

I intend to use a cluster of commodity hardware to reduce costing as much as possible. Please suggest pros and cons of building such a system using Redis. I am also concerned about high ram requirements of Redis.

推荐答案

我不会将 Redis 用于这样的任务.其他产品将更适合 IMO.

I would not use Redis for such a task. Other products will be a better fit IMO.

Redis 是一种内存数据存储.如果要存储 10-20 TB 的数据，则需要 10-20 TB 的 RAM，这很昂贵.此外，内存分配器针对小对象而不是大对象进行了优化.您可能不得不将文件切成各种小块，这会很不方便.

Redis is an in-memory data store. If you want to store 10-20 TB of data, you will need 10-20 TB of RAM, which is expensive. Furthermore, the memory allocator is optimized for small objects, not big ones. You would probably have to cut your files in various small pieces, it would not be really convenient.

Redis 不为 HA 和故障转移提供临时解决方案.提供了主/从复制(并且工作得很好)，但不支持此故障转移的自动化.客户端必须足够聪明才能切换到正确的服务器.服务器端的某些内容(但未指定)必须以可靠的方式在主节点和从节点之间切换角色.换句话说，Redis 只提供了一个 DIY 的 HA/failover 解决方案.

Redis does not provide an ad-hoc solution for HA and failover. A master/slave replication is provided (and works quite well), but with no support for the automation of this failover. Clients have to be smart enough to switch to the correct server. Something on server-side (but this is unspecified) has to switch the roles between master and slaves nodes in a reliable way. In other words, Redis only provides a do-it-yourself HA/failover solution.

分片必须在客户端实现(如使用 memcached).一些客户支持它，但不是全部.最快的客户端 (hiredis) 没有.无论如何，重新平衡之类的事情必须在 Redis 之上实现.应该支持这种分片能力的Redis Cluster还没有准备好.

Sharding has to be implemented on client-side (like with memcached). Some clients have support for it, but not all of them. The fastest client (hiredis) does not. Anyway, things like rebalancing has to be implemented on top of Redis. Redis Cluster which is supposed to support such sharding capabilities is not ready yet.

我建议使用其他一些解决方案.带有 GridFS 的 MongoDB 是一种可能.带有 HDFS 的 Hadoop 是另一个.如果您喜欢前沿项目，您可能想尝试一下 Elliptics Network.

I would suggest to use some other solutions. MongoDB with GridFS can be a possibility. Hadoop with HDFS is another one. If you like cutting edge projects, you may want to give the Elliptics Network a try.

这篇关于我应该使用redis来存储大量二进制文件吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！