选择分布式共享内存解决方案 [英] Choosing a distributed shared memory solution

问题描述

我的任务是为可大规模扩展的分布式共享内存 (DSM) 应用程序构建原型.原型仅用作概念验证，但我想通过选择稍后将在实际解决方案中使用的组件来最有效地利用我的时间.

I have a task to build a prototype for a massively scalable distributed shared memory (DSM) app. The prototype would only serve as a proof-of-concept, but I want to spend my time most effectively by picking the components which would be used in the real solution later on.

此解决方案的目的是从外部源获取数据输入，对其进行搅拌并使结果可用于多个前端.这些前端"只会从缓存中获取数据并提供它而无需额外处理.前端对这些数据的点击量可以达到每秒数百万次.

The aim of this solution is to take data input from an external source, churn it and make the result available for a number of frontends. Those "frontends" would just take the data from the cache and serve it without extra processing. The amount of frontend hits on this data can literally be millions per second.

数据本身非常不稳定；它可以(并且确实)迅速改变.然而，前端应该看到旧"数据，直到最新的数据被处理和缓存.处理和写入由单个(冗余)节点完成，而其他节点仅读取数据.换句话说:没有通读行为.

The data itself is very volatile; it can (and does) change quite rapidly. However the frontends should see "old" data until the newest has been processed and cached. The processing and writing is done by a single (redundant) node while other nodes only read the data. In other words: no read-through behaviour.

我正在研究诸如 memcached 之类的解决方案，但是这个特殊的解决方案并不能满足所有我们的要求要求如下:

I was looking into solutions like memcached however this particular one doesn't fulfil all our requirements which are listed below:

1. 该解决方案必须至少具有 Java 客户端 API，由于应用程序的其余部分是用 Java 编写的，而且我们是经验丰富的 Java 开发人员，因此维护得相当好；
2. 解决方案必须完全弹性:应该可以在不重启集群中的其他节点的情况下添加新节点；
3. 解决方案必须能够处理故障转移.是的，我意识到这意味着一些开销，但整体服务的数据大小并不大(最大 1G)所以这应该不是问题.故障转移"是指无缝执行，无需像在 memcached 客户端中那样在节点出现故障时硬编码/更改服务器 IP 地址；
4. 理想情况下，应该可以指定数据重叠的程度(例如，应该在 DSM 集群中存储多少相同数据的副本)；
5. 不需要永久存储所有数据，但可能需要对某些数据进行后处理(例如序列化到数据库).
6. 价格.显然，我们更喜欢免费/开源，但如果解决方案值得，我们很乐意支付合理的费用.无论如何，必须签订 24 小时/天的付费支持合同.
7. 整个过程必须托管在我们的数据中心，因此 Amazon SimpleDB 等 SaaS 产品超出了范围.只有在没有其他选项可用的情况下，我们才会考虑这一点.
8. 理想情况下，解决方案应严格一致(如在 CAP 中)；但是，最终一致性可以被视为一种选择.

1. The solution must at least have Java client API which is reasonably well maintained as the rest of app is written in Java and we are seasoned Java developers;
2. The solution must be totally elastic: it should be possible to add new nodes without restarting other nodes in the cluster;
3. The solution must be able to handle failover. Yes, I realize this means some overhead, but the overall served data size isn't big (1G max) so this shouldn't be a problem. By "failover" I mean seamless execution without hardcoding/changing server IP address(es) like in memcached clients when a node goes down;
4. Ideally it should be possible to specify the degree of data overlapping (e.g. how many copies of the same data should be stored in the DSM cluster);
5. There is no need to permanently store all the data but there might be a need of post-processing of some of the data (e.g. serialization to the DB).
6. Price. Obviously we prefer free/open source but we're happy to pay a reasonable amount if a solution is worth it. In any way, paid 24hr/day support contract is a must.
7. The whole thing has to be hosted in our data centers so SaaS offerings like Amazon SimpleDB are out of scope. We would only consider this if no other options would be available.
8. Ideally the solution would be strictly consistent (as in CAP); however, eventual consistence can be considered as an option.

提前感谢您的任何想法.

Thanks in advance for any ideas.

推荐答案

看看 Hazelcast.它是纯 Java、开源(Apache 许可)高度可扩展的内存数据网格产品.它确实提供 7X24 支持.它确实解决了我在下面试图解释的所有问题:

Have a look at Hazelcast. It is pure Java, open source (Apache license) highly scalable in-memory data grid product. It does offer 7X24 support. And it does solve all of your problems I tried to explain each of them below:

1. 它有一个本机 Java 客户端.
2. 它是 100% 动态的.动态添加和删除节点.不需要改变任何东西.
3. 同样，一切都是动态的.
4. 您可以配置备份节点的数量.
5. Hazelcast 支持持久性.
6. Hazelcast 提供的一切都是免费的(开源)，并且确实提供企业级支持.
7. Hazelcast 是单个 jar 文件.超级容易使用.只需将 jar 添加到您的类路径中.查看主页中的屏幕投射.
8. Hazelcast 是严格一致的.您永远无法读取过时的数据.

这篇关于选择分布式共享内存解决方案的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！