如何根据 Mongodb 中的键删除重复项? [英] How to remove duplicates based on a key in Mongodb?

问题描述

我在 MongoDB 中有一个集合，其中大约有(约 300 万条记录).我的示例记录看起来像，

I have a collection in MongoDB where there are around (~3 million records). My sample record would look like,

 { "_id" = ObjectId("50731xxxxxxxxxxxxxxxxxxxx"),
   "source_references" : [
                           "_id" : ObjectId("5045xxxxxxxxxxxxxx"),
                           "name" : "xxx",
                           "key" : 123
                          ]
 }

我在集合中有很多具有相同 source_references.key 的重复记录.(我的意思是重复，source_references.key 不是 _id).

I am having a lot of duplicate records in the collection having same source_references.key. (By Duplicate I mean, source_references.key not the _id).

我想根据source_references.key删除重复记录，我正在考虑编写一些PHP代码来遍历每条记录，如果存在则删除记录.

I want to remove duplicate records based on source_references.key, I'm thinking of writing some PHP code to traverse each record and remove the record if exists.

有没有办法删除 Mongo 内部命令行中的重复项?

Is there a way to remove the duplicates in Mongo Internal command line?

推荐答案

这个答案已经过时了: dropDups 选项是 在 MongoDB 3.0 中删除，所以在大多数情况下需要不同的方法.例如，您可以按照以下建议使用聚合:MongoDB 重复文档即使添加了唯一键.

This answer is obsolete : the dropDups option was removed in MongoDB 3.0, so a different approach will be required in most cases. For example, you could use aggregation as suggested on: MongoDB duplicate documents even after adding unique key.

如果您确定 source_references.key 标识重复记录，则可以使用 dropDups:true MongoDB 2.6 或更早版本中的索引创建选项:

If you are certain that the source_references.key identifies duplicate records, you can ensure a unique index with the dropDups:true index creation option in MongoDB 2.6 or older:

db.things.ensureIndex({'source_references.key' : 1}, {unique : true, dropDups : true})

这将为每个 source_references.key 值保留第一个唯一文档，并删除任何可能导致重复键违规的后续文档.

This will keep the first unique document for each source_references.key value, and drop any subsequent documents that would otherwise cause a duplicate key violation.

重要提示:任何缺少 source_references.key 字段的文档都将被视为具有 null 值，因此后续文档缺少该键字段将被删除.您可以添加 sparse:true 索引创建选项，因此索引仅适用于具有 source_references.key 字段的文档.

Important Note: Any documents missing the source_references.key field will be considered as having a null value, so subsequent documents missing the key field will be deleted. You can add the sparse:true index creation option so the index only applies to documents with a source_references.key field.

明显的警告:备份您的数据库，如果您担心意外的数据丢失，请先在暂存环境中尝试此操作.

Obvious caution: Take a backup of your database, and try this in a staging environment first if you are concerned about unintended data loss.

这篇关于如何根据 Mongodb 中的键删除重复项?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！