WiredTiger和就地更新 [英] WiredTiger and in-place updates
问题描述
我有一组用户.每个用户都有一个地理位置"字段，该字段经常更新(每次用户进行大量移动时).由于更新时我希望在文档级别而不是集合级别上具有并发性，因此我使用了WiredTiger存储引擎.
I have a collection of users. Each user has a field "geoposition" that is updated quite often (every time the user moves significantly). As I want concurrency on the document level instead of the collection level when updating, I am using the WiredTiger storage engine.
我了解到，通过WiredTiger，文档中的每次更新都会导致创建新文档:
I learned that with WiredTiger every update in the document results in the creation of a new document:
WiredTiger不支持就地更新
WiredTiger does not support in place updates
但是，本文还说:即使[WiredTiger]不允许就地更新，在许多工作负载下，它的性能仍比MMAP更好".这是什么意思?使用WiredTiger时，我必须清楚确切的含义是什么?例如，如果没有就地更新，数据库大小会迅速增加吗?还有其他需要注意的事情吗?
However, this article also says that "Even though [WiredTiger] does not allow for in-place updates, it could still perform better than MMAP for many workloads". What does it mean? What are the exact implications that I must be aware of when I use WiredTiger? For example, without in-place updates will the database size grow quickly? Are there other things to be aware of?
我还了解到，MongoDB 3.6中的WiredTiger增加了存储增量的功能，而不是重写整个文档(
I also learned that WiredTiger in MongoDB 3.6 added the capability to store deltas rather than re-writing the entire document (https://jira.mongodb.org/browse/DOCS-11416). What does this mean, exactly?
注意:另外，我不明白的是，当今大多数(如果不是全部)硬盘驱动器的扇区大小为4096字节，因此您不能仅向硬盘驱动器写入4字节(例如)，但必须写入4096字节的完整块(因此，请先读取它，更新其中的4个字节，然后再写入).由于大多数文档通常< 4096字节确实意味着在任何情况下都必须重写整个文档(即使使用MMAP).我想念什么?
NOTE: Also what I don't understand is that nowadays most (if not all) hard drives have a sector size of 4096 bytes, so you cannot write to the hard drive only 4 bytes (for example) but instead must write the full block of 4096 bytes (so read it first, update the 4 bytes in it and then write it). As most document are often < 4096 bytes does this mean that re-writing the whole document is necessary in any case (even with MMAP). What did I miss?
推荐答案
对于MMAPv1存储引擎，就地更新经常被作为一种优化策略来突出显示，因为文档的索引直接指向文件的位置和偏移量.与仅更新已更改字段的就地更新相比，将文档移动到新的存储位置(尤其是如果有许多索引条目要更新)会给MMAPv1带来更多开销.请参阅: MMAPv1中的记录存储特征.
With the MMAPv1 storage engine, in-place updates are frequently highlighted as an optimization strategy because indexes for a document point directly to file locations and offsets. Moving a document to a new storage location (particularly if there are many index entries to update) has more overhead for MMAPv1 than an in-place update which only has to update the changed fields. See: Record Storage Characteristics in MMAPv1.
WiredTiger不支持就地更新，因为它在内部使用 MVCC(多版本并发控制) ，数据库管理系统通常使用的.这是对MMAP中简单视图的一项重大技术改进，它允许构建更高级的功能，例如隔离级别和交易. WiredTiger的索引具有间接级别(引用内部RecordID而不是文件位置和偏移)，因此在存储级别移动文档并不是一个重大难题.
WiredTiger does not support in-place updates because internally it uses MVCC (Multiversion concurrency control), which is commonly used by database management systems. This is a significant technical improvement over the simplistic view in MMAP, and allows for building more advanced features like isolation levels and transactions. WiredTiger's indexes have a level of indirection (referencing an internal RecordID instead of the file location & offset), so document moves at the storage level are not a significant pain point.
但是，本文还说:即使[WiredTiger]不允许就地更新，它在许多工作负载下的性能仍比MMAP更好."
However, this article also says that "Even though [WiredTiger] does not allow for in-place updates, it could still perform better than MMAP for many workloads".
这意味着，尽管MMAPv1可能具有更有效的就地更新路径，但WiredTiger还具有其他优势，例如压缩和改进的并发控制.您也许可以构建一个工作负载，该工作负载仅包含对一些文档的就地更新，这些更新在MMAPv1中可能会更好地表现出来，但是实际的工作负载通常变化更大.确认对给定工作负载的影响的唯一方法是在具有代表性的环境中进行测试.
It means that although MMAPv1 may have a more efficient path for in-place updates, WiredTiger has other advantages such as compression and improved concurrency control. You could perhaps construct a workload consisting only of in-place updates to a few documents which might perform better in MMAPv1, but actual workloads are typically more varied. The only way to confirm the impact for a given workload would be to test in a representative environment.
但是，如果您想为将来做计划，那么MMAPv1与WiredTiger的一般选择就没有意义了:由于MongoDB 3.2和MMAPv1不支持某些较新的产品功能，因此WiredTiger一直是默认的存储引擎.例如，MMAPv1不支持多数阅读关注，转意味着它不能用于副本设置配置服务器(在MongoDB 3.4+中为分片所必需)或更改流(MongoDB 3.6+). MMAPv1在MongoDB的下一个主要版本(4.0)中将被弃用，目前计划在MongoDB 4.2中被删除.
However, the general choice of MMAPv1 vs WiredTiger is moot if you want to plan for the future: WiredTiger has been the default storage engine since MongoDB 3.2 and some newer product features are not supported by MMAPv1. For example, MMAPv1 does not support Majority Read Concern which in turn means it cannot be used for Replica Set Config Servers (required for sharding in MongoDB 3.4+) or Change Streams (MongoDB 3.6+). MMAPv1 will be deprecated in the next major release of MongoDB (4.0) and is currently scheduled to be removed in MongoDB 4.2.
使用WiredTiger时，我必须确切了解哪些含义?例如，如果没有就地更新，数据库大小会迅速增长吗?
What are the exact implications that I must be aware of when I use WiredTiger? For example, without in-place updates will the database size grow quickly?
存储结果取决于几个因素，包括架构设计，工作量，配置和MongoDB服务器的版本. MMAPv1和WiredTiger使用不同的记录分配策略，但是两者都将尝试使用标记为可用/可重用的预分配空间.通常，WiredTiger在使用存储空间时效率更高，并且还具有压缩数据和索引的优势. MMAPv1 分配额外的存储空间以尝试针对可以进行更新并避免文档移动，但是您可以为工作量不会随时间改变文档大小的集合选择无填充"策略.
Storage outcomes depend on several factors including your schema design, workload, configuration, and version of MongoDB server. MMAPv1 and WiredTiger use different record allocation strategies, but both will try to use preallocated space that is marked as free/reusable. In general WiredTiger is more efficient with use of storage space, and it also has the advantage of compression for data and indexes. MMAPv1 allocates additional storage space to try to optimize for in-place updates and avoid document moves, although you can choose a "no padding" strategy for collections where the workload does not change the document size over time.
自从MongoDB 3.0首次引入WiredTiger以来，已经投入了大量的精力来改进和优化针对不同工作负载的WiredTiger，因此，我强烈建议您使用最新的生产版本系列进行测试以获得最佳结果.如果您对架构设计和存储增长有特定疑问，建议您在DBA StackExchange上发布详细信息以进行讨论.
There has been significant investment in improving and tuning WiredTiger for different workloads since it was first introduced in MongoDB 3.0, so I would strongly encourage testing with the latest production release series for the best outcome. If you have a specific question about schema design and storage growth, I'd suggest posting details on DBA StackExchange for discussion.
I also learned that WiredTiger in MongoDB 3.6 added the capability to store deltas rather than re-writing the entire document (https://jira.mongodb.org/browse/DOCS-11416). What does this mean, exactly?
这是一个实现细节，对于某些用例，它改进了WiredTiger的内部数据结构.特别是，与以前的版本相比，MongoDB 3.6+中的WiredTiger在处理大型文档的较小更改时可以更高效.只要开放内部会话(如前所述，MVCC)使用了WiredTiger缓存，它们就必须能够返回多个版本的文档，因此对于具有较小更新的大型文档，存储增量列表可能更为有效.但是，如果累积了太多的增量(或者这些增量改变了文档中的大多数字段)，则此方法的性能可能会低于维护整个文档的多个副本.
This is an implementation detail that improves WiredTiger's internal data structures for some use cases. In particular, WiredTiger in MongoDB 3.6+ can be more efficient about working with small changes to large documents (as compared to previous releases). The WiredTiger cache needs to be able to return multiple versions of documents as long as they are used by open internal sessions (MVCC, as mentioned earlier), so for large documents with small updates it could be more efficient to store a list of deltas. However, if too many deltas accumulate (or the deltas are changing most of the fields in a document) this approach could be less performant than maintaining multiple copies of the full document.
通过检查点将数据提交到磁盘后，仍需要编写文档的完整版本.如果您想了解有关内部结构的更多信息，可以使用MongoDB 交易路径系列开发支持MongoDB 4.0中多文档事务的功能后的视频.
When data is committed to disk via a checkpoint, a full version of the document still needs to be written. If you want to learn more about some of the internals, there's a MongoDB Path To Transactions series of videos following the development of features to support multi-document transactions in MongoDB 4.0.
我不了解的是，当今大多数(如果不是全部)硬盘驱动器的扇区大小为4096字节，因此您不能仅向硬盘驱动器写入4个字节(例如)，而必须写入完整的字节4096字节的块(因此，请先读取它，更新其中的4个字节，然后再写入).由于大多数文档通常< 4096字节确实意味着在任何情况下都必须重写整个文档(即使使用MMAP).我想念什么?
Also what I don't understand is that nowadays most (if not all) hard drives have a sector size of 4096 bytes, so you cannot write to the hard drive only 4 bytes (for example) but instead must write the full block of 4096 bytes (so read it first, update the 4 bytes in it and then write it). As most document are often < 4096 bytes does this mean that re-writing the whole document is necessary in any case (even with MMAP). What did I miss?
在不深入介绍实现细节并试图解释所有涉及的移动部件的情况下，请考虑将不同的方法应用于正在更新许多文档(而不是在单个文档级别)的工作负载以及对内存的影响用法(在将文档写入磁盘之前).取决于文档大小和压缩等因素，单个I/O块可以代表从文档的一小部分(最大16MB)到多个文档的任何地方.
Without getting too far into implementation details and trying to explain all the moving parts involved, consider how the different approaches apply to workloads where many documents are being updated (rather than at the single document level) as well as the impact on memory usage (before documents are written to disk). Depending on factors like document size and compression, a single block of I/O can represent anywhere from a fraction of a document (max size 16MB) to multiple documents.
在MongoDB中，一般流程是在内存视图(例如WiredTiger缓存)中更新文档，并以快速仅追加日记格式将更改持久保存到磁盘，然后再
In MongoDB the general flow is that documents are updated in an in-memory view (for example, the WiredTiger cache) with changes persisted to disk in a fast append-only journal format before being periodically flushed to the data files. If the O/S only has to write blocks of data that have changed, touching fewer blocks of data requires less overall I/O.
这篇关于WiredTiger和就地更新的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
