是否有任何利用NoSQL系统的多核? [英] Is there any multicore exploiting NoSQL system?

问题描述

从昨天开始，我就在MongoDB上玩游戏，绝对喜欢它.我正在尝试导入大量数据(20亿行)并对其进行索引，但它似乎并未使用我的系统已导入的8个内核，并且导入速度正常(60000条记录/秒).我只能想象索引此集合中的两列可能需要多长时间.是否有任何利用CPU的多核性质的MondoDB类型数据库?

I am playing with MongoDB since yesterday and absolutely love it. I am trying to import lots of data (2 billion rows) and index it but it doesn't seem to be using the 8 cores that my system has adn the import is going at normal rates (60000 records/sec). I can only imagine how long it might take to index two columns in this collection. Are there any MondoDB type databases that exploit multicore nature of CPUs?

推荐答案

如果MongoDB有一个致命弱点，那就是它仅支持单线程写入和单线程映射减少.

If MongoDB has an achilles heel it's the fact that it only supports single-threaded writes and single-threaded map-reduces.

一如既往，这里需要权衡.单线程写是避免锁定问题并最小化开销的最简单方法.同样，多线程映射减少是锁定数据的好方法.因此，在生产系统上使用单线程map-reduce可能更容易，更安全.

As always, there are trade-offs here. Single-threaded writes are the simplest way to avoid lock problems and minimize overhead. In the same fashion multi-threaded map-reduces are a great way to lock your data. So single-threaded map-reduces on a production system are probably easier and safer.

但是，这里并非没有工具. MongoDB将为每个实例提供一个写线程.因此，如果对MongoDB进行分片，则每个分片将获得一个写线程.

However, you're not without tools here. MongoDB will provide one write thread to each instance. So if you shard MongoDB, then you'll get one write thread for each shard.

如果要在20亿行上使用多个索引，则无论如何都要考虑分片.这里有一些快速的数学运算:MongoID是12个字节. MongoID的索引将为2B * 12字节= 22GB +.如果您现在想再添加两个索引(甚至只是两个4个字节的整数)，那么我们将为每个索引分配7.5GB.

If you want multiple indexes on 2 billion rows, you'll want to look at sharding anyways. Some quick math here: MongoID is 12 bytes. Index on MongoID will be 2B * 12 bytes = 22GB+. If you now want to add two more indexes (even just two 4-byte integers) we're talking about 7.5GB for each.

因此，在2B行中，您所谈论的是索引中的空间超过37GB(最小).在大多数8核服务器上，这意味着您甚至无法将索引保留在内存中，更不用说任何数据了.

So at 2B rows, you're talking about having over 37GBs in indexes (minimum). On most 8-core servers, that means that you won't even be able to keep you indexes in memory, let alone any of the data.

因此，如果您想在这里获得出色的性能，则需要开始研究分片.仅基于一般数字. FWIW，MySQL将不再擅长处理2B文档.有了这么多的数据，您真的会希望多台服务器来承受负载.

So if you want serious performance here, you'll want to start looking at sharding. Just based on the general numbers. FWIW, MySQL would be no more adept at handling 2B documents. With that much data, you're really going to want multiple servers to keep up with the load.

这篇关于是否有任何利用NoSQL系统的多核?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！