如何快速获取所有文档MongoDB pymongo [英] How to quickly fetch all documents MongoDB pymongo

问题描述

当前，我通过在pymongo中通过游标进行迭代来获取文档，例如:

Currently I fetch documents by iterating through cursor in pymongo, for example:

for d in db.docs.find():
    mylist.append(d)

作为参考，对同一组数据(7m条记录)执行fetchall大约需要20秒，而上述方法则需要几分钟.

For reference, performing a fetchall on the same set of data (7m records) takes around 20 seconds while the method above takes a few minutes.

有没有更快的方法来读取mongo中的批量数据?抱歉，我是mongo的新手，如果需要更多信息，请告诉我.

Is there a faster way read bulk data in mongo? Sorry I'm new to mongo, please let me know if more information is needed.

推荐答案

使用$ natural排序将绕过索引并按文档在磁盘上存储的顺序返回文档，这意味着mongo不必进行重击在磁盘上随机读取.

using the $natural sort will bypass the index and return the documents in the order in which they are stored on disk, meaning that mongo doesn't have to thrash around with random reads on your disk.

https://docs.mongodb.com /manual/reference/method/cursor.sort/#return-natural-order

如果要使用查询，性能会严重下降.您永远不应依赖FIFO排序. Mongo允许自己在其存储层内移动文档.如果您不关心订单，那就去吧.

The performance becomes severely degraded if you want to use a query. You should never rely on FIFO ordering. Mongo allows itself to move documents around within it's storage layer. If you don't care about the order, so be it.

此排序是内部实施功能，您应该 不依赖于i内的任何特定结构

This ordering is an internal implementation feature, and you should not rely on any particular structure within i

for d in db.docs.find().sort( { $natural: 1 } ):
    mylist.append(d)

在python中，您还想使用 EXHAUST 游标类型，该游标类型告诉mongo服务器无需等待pymongo驱动程序确认每个批次即可流回结果

in python, you also want to use an EXHAUST cursor type that tells the mongo server to stream back the results without waiting for the pymongo driver to acknowledge each batch

https://api. mongodb.com/python/current/api/pymongo/cursor.html#pymongo.cursor.CursorType.EXHAUST

请介意，它永远不会像shell那样快.在mongo/bson-> pymongo-> you之间移动数据的最慢方面是python中的UTF8字符串解码.

Mind you, it'll never be as fast as the shell. The slowest aspect of moving data between mongo/bson->pymongo->you is UTF8 string decoding within python.

这篇关于如何快速获取所有文档MongoDB pymongo的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！