使用pymongo读取和更新mongodb文档的最佳方法 [英] Best way to read and update mongodb documents using pymongo

问题描述

iam试图逐个文档读取mongodb集合文档，以获取每个记录，对记录中的某些字段进行加密，然后将其放回数据库中.

iam trying to read a mongodb collection document by document in order to fetch every record encrypt some of fields in the record and put it back to database.

for record in coll.find():
    #modifying record here
    coll.update(record)

这将导致严重的问题，即已被游标再次读取已更新的文档，并且循环再次处理同一文档(同一文档正在尝试再次更新)

this is causing a serious problem i.e already updated documents are read again by cursor and same document is processed again in loop (same document is trying to update again)

希望这可能是解决问题的方法之一.

hope this may be one of the solution to the problem.

list_coll = [record for record in coll.find()]
for rec in list_coll:
   #modifying record
   coll.update(rec)

但这是最好的方法吗?即，如果集合很大，会发生什么?大list_coll会导致ram溢出吗? 请给我一个最好的方法.

but is this the best way of doing? i.e what happens if the collection is large ? can large list_coll causes ram overflow? kindly suggest me a best way of doing it.

谢谢

推荐答案

您需要.通常是在MongoDB 2.6中引入的，因此，如果您目前还没有升级的话，那么就有必要进行升级.

You want the "Bulk Operations API" from MongoDB. Mostly introduced with MongoDB 2.6, so a compelling reason to be upgrading if you currently have not.

bulk = db.coll.initialize_ordered_bulk_op()
counter = 0

for record in coll.find(snapshot=True):
    # now process in bulk
    # calc value first
    bulk.find({ '_id': record['_id'] }).update({ '$set': { 'field': newValue } })
    counter += 1

    if counter % 1000 == 0:
        bulk.execute()
        bulk = db.coll.initialize_ordered_bulk_op()

if counter % 1000 != 0:
    bulk.execute()

更好的是，您不会将每个"请求发送到服务器，而每1000个请求中只有一次. 批量API"实际上可以为您解决一些问题，但实际上您想更好地管理"此问题，而又不会在应用程序中占用过多内存.

Much better as you are not sending "every" request to the server, just once in every 1000 requests. The "Bulk API" actually sorts this out for you somewhat, but really you want to "manage" this a little better and not consume too much memory in your app.

未来之路.使用它.

这篇关于使用pymongo读取和更新mongodb文档的最佳方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！