如何提高MongoDB插入性能 [英] How to improve MongoDB insert performance
问题描述
结果:
如果您正在容错的数据集上进行操作或可以执行一次处理，则可以将WriteAcknowledge更改为Unacknowledged会有所帮助.
此外，默认情况下，批量操作默认为IsOrdered，我不清楚.将此设置为False实际上会使操作批量执行，否则它将作为一个更新线程进行操作.
MongoDB 3.0/WiredTiger/C#驱动程序
MongoDB 3.0 / WiredTiger / C# Driver
我有一个包含147,000,000个文档的集合，我正在(希望)每秒对其进行更新. 3000个文档.
I have a collection with 147,000,000 documents, of which I am performing updates each second (hopefully) of approx. 3000 documents.
以下是示例更新:
"query" : {
"_id" : BinData(0,"UKnZwG54kOpT4q9CVWbf4zvdU223lrE5w/uIzXZcObQiAAAA")
},
"updateobj" : {
"$set" : {
"b" : BinData(0,"D8u1Sk/fDES4IkipZzme7j2qJ4oWjlT3hvLiAilcIhU="),
"s" : true
}
}
这是一个典型的更新，我将以每秒3000的速度插入它.
This is a typical update of which I my requirements are to be inserted at a rate of 3000 per second.
不幸的是，这些更新花费了两倍的时间，例如，上一次更新是对1723个文档进行的，花费了1061毫秒.
Unfortunately these are taking twice as long, for instance the last update was for 1723 documents, and took 1061ms.
该集合仅在_id上有一个索引，没有其他索引，并且该集合的平均文档大小为244个字节，无上限.
The collection only has an index on the _id, no other indexes, and the average document size for the collection is 244 bytes, uncapped.
该服务器具有64GB内存，12个线程.插入数据的性能出色，收集量较小，例如大约为5000万，但实际上大约有8000万开始下降.
The server has 64GB of memory, 12 threads. Insert performance is excellent with lower collection sizes, say around 50 million, but after about 80 million really starts to drop off.
可能是因为整个集合都不位于内存中吗?数据库由RAID0 SSD支持，因此IO性能不应成为瓶颈，如果是的话，它应该在一开始就显示出来吗?
Could it be because the entire set does not sit in memory? Database is backed by RAID0 SSDs so IO performance should not become a bottleneck and if it was it should have shown this at the beginning?
希望能得到一些指导，因为我相信MongoDB与使用的某些应用程序相比可以满足我相当微薄的要求.数据库上的读取率不高，因此分片无法改善问题，尽管也许我错了
Would appreciate some guidance as I'm confident MongoDB can fulfill my rather meager requirements compared to some applications it is used in. There is not a substantial read rate on the database so Sharding would not improve matters, although perhaps I am wrong.
无论哪种方式，当前的插入率都不够好.
Either way, the current insert rate is not good enough.
更新:这是查询的explain()...
Update: Here is the explain() of just the query...
"queryPlanner" : {
"plannerVersion" : 1,
"namespace" : "Collection",
"indexFilterSet" : false,
"parsedQuery" : {
"_id" : {
"$eq" : { "$binary" : "SxHHwTMEaOmSc9dD4ng/7ILty0Zu0qX38V81osVqWkAAAAAA", "$type" : "00" }
}
},
"winningPlan" : {
"stage" : "IDHACK"
},
"rejectedPlans" : []
},
"executionStats" : {
"executionSuccess" : true,
"nReturned" : 1,
"executionTimeMillis" : 1,
"totalKeysExamined" : 1,
"totalDocsExamined" : 1,
"executionStages" : {
"stage" : "IDHACK",
"nReturned" : 1,
"executionTimeMillisEstimate" : 0,
"works" : 2,
"advanced" : 1,
"needTime" : 0,
"needFetch" : 0,
"saveState" : 0,
"restoreState" : 0,
"isEOF" : 1,
"invalidates" : 0,
"keysExamined" : 1,
"docsExamined" : 1
},
"allPlansExecution" : []
},
它本身的查询非常快，更新操作大约需要25毫秒，而使用BulkWriter将它们推入Mongo:await m_Collection.BulkWriteAsync(updates);
The query it self is very fast, and the update operation takes about 25ish milliseconds, they are being pushed to Mongo by use of the BulkWriter: await m_Collection.BulkWriteAsync(updates);
推荐答案
您可以尝试修改 1000个组时，此应该加快这一过程.
You can try to modify the Write concern levels. Obviously there is a risk on this, as you wouldn't be able to catch any writing error, but at least you should still be able to capture network errors. As MongoDB groups the bulk insert operations in groups of 1000, this should speed up the process.
W 默认为1:
当您将其更改为0时:
如果您不关心元素的顺序，则可以提高调用无序批量操作的速度
If you are not concern about the order of elements, you can gain some speed calling the unordered bulk operation
await m_Collection.BulkWriteAsync(updates, new BulkWriteOptions() { IsOrdered = false });
使用无序操作列表，MongoDB可以并行执行 以任何顺序在列表中写入操作. 链接
With an unordered operations list, MongoDB can execute in parallel the write operations in the list and in any order. Link
这篇关于如何提高MongoDB插入性能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
