在多核服务器上的mongodb map减少 [英] mongodb map reduce on multicore server

问题描述

我有一个mongodb，其中有成千上万条包含非常长的向量的记录. 我正在使用某种算法寻找输入向量与MDB数据集之间的相关性.

I have a mongodb with thousands of records holding very long vectors. I am looking for correlations between an input vector with my MDB data set using a certain algorithm.

伪代码:

function find_best_correlation(input_vector)
    max_correlation = 0
    return_vector = []
    foreach reference_vector in dataset:
        if calculateCorrelation(input_vector,reference_vector) > max_correlation then:
            return_vector = reference_vector
    return return_vector

这是地图缩减模式的很好候选者，因为我不在乎计算的运行顺序.

This is a very good candidate for map-reduce pattern as I don't care for the order the calculations are run in.

问题是我的数据库位于一个节点上. 我想同时运行许多映射(我有一台8核计算机)

The issue is that my database is on one node. I would like to run many mappings simultaneously (I have an 8 core machine)

据我了解，MongoDb每个节点仅使用一个执行线程-实际上，我是在串行运行数据集. 这是正确的吗?

From what I understand, MongoDb only uses one thread of execution per node - in practice I am running my data set serially. Is this correct?

如果可以，我是否可以配置每个map-reduce运行的进程/线程数? 如果我管理并行运行map-reduce的多个线程，然后汇总结果，我的性能会大大提高(有人尝试过)吗? 如果不是，我可以在同一节点上对数据库进行多个复制，并在两个复制上运行技巧" mongoDb吗?

If so can I configure the number of processes/threads per map-reduce run? If I manage multiple threads running map-reduce in parallel and then aggregate the results will I have substantial performance increase (Has anybody tried)? If not - can i have multiple replications of my DB on the same node and "trick" mongoDb to run on 2 replications?

谢谢！

推荐答案

MongoDB中的Map reduce使用Spidermonkey(单线程Javascript引擎)，因此无法配置多个进程(并且没有技巧").有一个使用多线程JS引擎的JIRA票证，您可以在此处遵循: https://jira.mongodb.org/browse/SERVER-2407

Map reduce in MongoDB uses Spidermonkey, a single-threaded Javascript engine, so it is not possible to configure multiple processes (and there are no "tricks"). There is a JIRA ticket to use a multi-threaded JS engine, which you can follow here: https://jira.mongodb.org/browse/SERVER-2407

如果可能，我会考虑研究新的聚合框架(在MongoDB 2.2版中可用)，该框架是用C ++而不是Javascript编写的，并且可能会提高性能: http://docs.mongodb.org/manual/applications/aggregation/

If possible, I would consider looking into the new aggregation framework (available in MongoDB version 2.2), which is written in C++ instead of Javascript and may offer performance improvements: http://docs.mongodb.org/manual/applications/aggregation/

这篇关于在多核服务器上的mongodb map减少的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！