MongoDB批量插入忽略重复 [英] MongoDB Bulk Insert Ignore Duplicate

问题描述

我在Google周围搜索，找不到有关使用批量插入时如何忽略重复错误的可靠信息.

I've Googled around and can't find any solid information on how to ignore duplicate errors when using bulk insert.

这是我当前正在使用的代码:

Here's the code I'm currently using:

MongoClient.connect(mongoURL, function(err, db) {
      if(err) console.err(err)
      let col = db.collection('user_ids')
      let batch = col.initializeUnorderedBulkOp()

      ids.forEach(function(id) {
        batch.insert({ userid: id, used: false, group: argv.groupID })
      })

      batch.execute(function(err, result) {
        if(err) {
          console.error(new Error(err))
          db.close()
        }

        // Do some work

        db.close()
      })
    })

有可能吗?我尝试将{continueOnError: true, safe: true}添加到bulk.insert(...)，但这没用.

Is it possible? I've tried adding {continueOnError: true, safe: true} to bulk.insert(...) but that didn't work.

有什么想法吗?

推荐答案

一种替代方法是使用

An alternative is to use bulk.find().upsert().replaceOne() instead:

MongoClient.connect(mongoURL, function(err, db) {
    if(err) console.err(err)
    let col = db.collection('user_ids')
    let batch = col.initializeUnorderedBulkOp()

    ids.forEach(function(id) {        
        batch.find({ userid: id }).upsert().replaceOne({ 
            userid: id, 
            used: false,  
            group: argv.groupID 
        });
    });

    batch.execute(function(err, result) {
        if(err) {
            console.error(new Error(err))
            db.close()
        }

        // Do some work

        db.close()
    });
});

使用上述方法，如果文档与查询{ userid: id }相匹配，它将被新文档替换，否则将被创建，因此不会引发重复的键错误.

With the above, if a document matches the query { userid: id } it will be replaced with the new document, otherwise it will be created hence there are No duplicate key errors thrown.

对于3.2或更高版本的MongoDB服务器，请使用 bulkWrite 为:

For MongoDB server versions 3.2+, use bulkWrite as:

MongoClient.connect(mongoURL, function(err, db) {

    if(err) console.err(err)

    let col = db.collection('user_ids')
    let ops = []
    let counter = 0

    ids.forEach(function(id) {
        ops.push({
            "replaceOne": {
                "filter": { "userid": id },
                "replacement": { 
                    userid: id, 
                    used: false,  
                    group: argv.groupID 
                },
                "upsert": true
            }
        })

        counter++

        if (counter % 500 === 0) {
            col.bulkWrite(ops, function(err, r) {
                // do something with result
                db.close()
            })
            ops = []
        }
    })

    if (counter % 500 !== 0) {
        col.bulkWrite(ops, function(err, r) {
            // do something with result
            db.close()
        }
    } 
})

这篇关于MongoDB批量插入忽略重复的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！