在 mongodb 中的索引列上查找重复项的快速方法 [英] Fast way to find duplicates on indexed column in mongodb

问题描述

我在 mongodb 中有一个 md5 集合.我想找到所有重复项.md5 列已编入索引.您知道使用 map reduce 的任何快速方法吗?还是应该只遍历所有记录并手动检查重复项?

I have a collection of md5 in mongodb. I'd like to find all duplicates. The md5 column is indexed. Do you know any fast way to do that using map reduce. Or should I just iterate over all records and check for duplicates manually?

我目前使用 map reduce 的方法几乎两次迭代集合(假设有非常少量的重复项):

My current approach using map reduce iterates over the collection almost twice (assuming that there is very small amount of duplicates):

res = db.files.mapReduce(
    function () {
        emit(this.md5, 1);
    }, 
    function (key, vals) {
        return Array.sum(vals);
    }
)

db[res.result].find({value: {$gte:1}}).forEach(
function (obj) {
    out.duplicates.insert(obj)
});

推荐答案

最简单的方法是一次性按md5排序，然后适当处理.

The easiest way to do it in one pass is to sort by md5 and then process appropriately.

类似:

var previous_md5;
db.files.find( {"md5" : {$exists:true} }, {"md5" : 1} ).sort( { "md5" : 1} ).forEach( function(current) {

  if(current.md5 == previous_md5){
    db.duplicates.update( {"_id" : current.md5}, { "$inc" : {count:1} }, true);
  }

  previous_md5 = current.md5;

});

那个小脚本对 md5 条目进行排序并按顺序循环它们.如果一个 md5 被重复，那么它们将在排序后背靠背".所以我们只保留一个指向 previous_md5 的指针并比较它 current.md5.如果我们找到重复项，我会将其放入 duplicates 集合中(并使用 $inc 来计算重复项的数量).

That little script sorts the md5 entries and loops through them in order. If an md5 is repeated, then they will be "back-to-back" after sorting. So we just keep a pointer to previous_md5 and compare it current.md5. If we find a duplicate, I'm dropping it into the duplicates collection (and using $inc to count the number of duplicates).

此脚本意味着您只需遍历一次主数据集.然后你可以遍历 duplicates 集合并执行清理.

This script means that you only have to loop through the primary data set once. Then you can loop through the duplicates collection and perform clean-up.

这篇关于在 mongodb 中的索引列上查找重复项的快速方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！