寻求解释/信息:Windows使用"fsync"来写I/O性能. (FlushFileBuffers) [英] Explanation/information sought: Windows write I/O performance with "fsync" (FlushFileBuffers)

问题描述

此问题是我之前发布的先前问题的补充: Windows fsync (FlushFileBuffers)对大文件的性能.在哪里找到了可能的解决方案，但又提出了新的问题.

This question is in follow up of an earlier question I posted: Windows fsync (FlushFileBuffers) performance with large files. Where I found a possible solution but also new questions.

在对fsync写入的不同场景进行基准测试时，我发现了许多令人惊讶的结果.我希望有人能帮助您解释或指出解释这些结果的信息方向.

While benchmarking different scenarios for fsynced writes, I found a number of surprising results. I am hoping someone can help explain or point me in the direction of information that explains these results.

此基准测试的工作是将8个页面(32 K)的批处理顺序随机地(大小为4096字节的大页面)写入文件，然后刷新写入.它总共写入200000页，总计800 MB和25000次刷新.在开始写操作之前，文件的大小被设置为其最终长度.

What this benchmark does is writing random blocks (pages 4096 bytes large) to a file sequentially in batches of 8 pages (32 K) and then flushing the writes. It writes a total of 200000 pages, amounting to a total of 800 MB and 25000 flushes. The file's size is set to its final length before beginning the writes.

它总共支持4个选项，其中所有组合都可以运行:

It supports a total of 4 options, of which all combinations are run:

• 在写入批处理或常规刷新(NS)之后执行"fsync"/FlushFileBuffers操作(FS).
• 在开始写入(LB)或将文件留空(E)之前将单个字节写入文件的最后一个位置.
• 要使用普通的缓冲写入(B)或无缓冲/直写(WT)写入(使用FILE_FLAG_NO_BUFFERING和FILE_FLAG_WRITE_THROUGH).
• 要直接写入文件流，即通过文件句柄(F)或使用内存映射(MM)间接写入文件.

• To perform an "fsync"/FlushFileBuffers operation (FS) after writing a batch or a normal flush (NS).
• To write a single byte to the last position of the file before starting to write (LB) or leave the file empty (E).
• To use normal buffered writes (B) or unbuffered/writethrough (WT) writes (using FILE_FLAG_NO_BUFFERING and FILE_FLAG_WRITE_THROUGH).
• To write directly to the file stream, i.e through the file handle (F) or write indirectly to the file using a memory map (MM).

下表总结了我在系统(带有慢速主轴磁盘的64位Win 7笔记本电脑)上对这些选项的所有组合的发现.

The table below summarizes my findings on my system (64 bit Win 7 laptop with slow spindle disk) for all combinations of these options.

我发现，"fsync"缓冲写入的性能随着文件的大小呈指数下降，从而达到难以置信的低吞吐量，这使得在与大文件结合使用时不可行.如果文件的最后一个字节被写入(选项LB)，则吞吐量甚至会更低，因此我担心在随机而不是顺序写入的情况下，性能会更加出色.

What I found is that the performance of "fsynced" buffered writes decreases exponentially with the size of the file to an incredibly low throughput that makes doing this not feasible in combination with large files. If the file had its last byte written (option LB), the throughput is even lower, so I fear that in random instead of sequential write scenarios the performance will be even far more dramatic.

但是令人惊讶的是，使用无缓冲/直写式I/O时，吞吐量保持恒定，而与文件大小无关.最初(前100-200 MB)，其吞吐量低于缓冲写入，但此后，平均吞吐量迅速赶上，并且完成写入800 MB的速度大大提高.更令人惊讶的是，如果文件的最后一个字节被写入，那么吞吐量将增加2倍.

Surprisingly however is that with unbuffered/writethrough I/O the throughput remains constant, independent of the size of the file. Initially (the first 100-200 MB) it is at a lower throughput than the buffered writes, but after that the average throughput catches up quickly and it finishes writing the 800 MB substantially quicker. Even more surprising is that if the file had its last byte written, the throughput increases by a factor of 2.

当通过内存映射文件写入文件时，在以无缓冲/直写标志打开文件的情况下，也会看到同样的性能下降.同样，如果文件的最后一个字节写入了字节，性能也会变差.

When writing to the file through a memory mapped file, the same exponential decrease in performance is seen, also in the case where the file was opened with unbuffered/writethrough flags. And here as well, performance is worse if the file had a byte written to its last position.

更新 根据霍华德的解释此处和

UPDATE Based upon Howard's explanations here and here, I reran the test without creating a new file before starting the writes (i.e. opening the existing, fully written file and overwriting it). I have updated the code in my original question to reflect the changes made for this test. The results are partly in line with his explanation and findings on Linux. But there are some notable exceptions. The table below provides the results, red highlights significant changes, blue highlights where changes did not occur and this is surprising (i.e. not in line with expectations if the effects mentioned in Howard's explanation were the only ones at play).

对于使用"fsync"刷新进行的缓冲写入文件(即不通过memmap)，现在性能已从指数衰减变为恒定趋势.但是，与以前的测试方案相比，现在所需的时间要长得多.吞吐率恒定为1.5 MB/s，在此之前，吞吐率以20 MB/s的速度开始呈指数下降至1.5 MB/s.似乎可能的解释是，每次刷新时文件元数据也会被刷新，从而引起磁盘旋转以寻找元数据的位置.

For the buffered writes to file (i.e. not through memmap) with an "fsync" flush, the performance has now changed from exponentially decaying to a constant trend. However, it now takes much longer than in the previous test scenarios. The throughput is a constant 1.5 MB/s, where before it started at around 20 MB/s to exponentially decay to around 1.5 MB/s. It would appear that a possible explanation is that the file metadata also gets flushed on each flush, causing a full disk revolution to seek for the location of the metadata.

对于直写"到文件方案，写入最后一个字节或不写入最后一个字节的结果现在相同，这与霍华德的解释所期望的一致.

For the "write through" to file scenarios, the results for writing the last byte or not, are now identical, in line with what is expected from Howard's explanation.

但是，除了一个值得注意的例外，对内存映射的写操作并没有真正改变，这令人惊讶.它们在写入性能上仍然显示出相同的指数衰减(从大约20 MB/s开始下降到1.8 MB/s).这表明正在发挥一种不同的机制.一个值得注意的例外是如果创建的基础文件没有FILE_FLAG_WRITE_THROUGH并且执行了"fsync"刷新.现在，此方案显示了稳定的(较差)性能，吞吐量约为1.6 MB/s.由于我有一些疑问，因此我多次重试了此方案，每次都给出相同的结果.

The writes to the memory map however, with one notable exception, have not really changed, and this is surprising. They still show the same exponential decay in write performance (starting at around 20 MB/s decaying to 1.8 MB/s). This would suggest that a different mechanism is at play. The one notable exception is if the underlying file was created without FILE_FLAG_WRITE_THROUGH and "fsync" flushes are performed. This scenario now shows a constant (poor) performance with a throughput of around 1.6 MB/s. Since I had some doubts, I reran this scenario multiple times, giving the same result each time.

要弄清楚一点，我还使用一个较小的文件(50000页，总计200 MB)重新运行了该测试，以确认fsync性能(用于缓冲I/O)实际上取决于文件大小.结果如下所示，其中需要特别注意的部分以红色突出显示.

To figure out a bit more, I also reran this test using a smaller file (50000 pages, amounting to 200 MB), to confirm, that the fsync performance (for buffered I/O) actually does depend on file size. The results are shown below, with those that deserve special attention highlighted in red.

这些结果与较大文件中看到的结果很好相关.值得注意的变化是，与突出显示的写入相比，写入的性能要好一些，在这些写入中似乎达到了约7 MB/s的限制.

These results correlate well with what was seen for the larger file. The notable changes are that writes are a bit more performant for those that are highlighted, where they seem to hit a limit of around 7 MB/s.

根据到目前为止对我的系统的观察，总结为高度投机的结论:

Summarizing as highly speculative conclusions based on observations on my sytem so far:

带有已缓冲IO(即没有FILE_FLAG_WRITE_THROUGH标志)的文件在Windows上的
• "fsync"性能随着已写入文件的字节数呈指数下降.原因似乎是每次都需要刷新文件元数据，这会导致磁盘搜索到文件的开头.
写入内存映射文件时，Windows上的
• "fsync"性能也显示出指数级降低的性能.我目前还没有引起这种情况的确切机制的解释.

• "fsync" performance on windows on files with buffered IO (i.e. without FILE_FLAG_WRITE_THROUGH flags) is exponentially decreasing with the number of bytes already written to the file. The reason seems to be the need to flush file metadata every time, which causes a disk seek to the start of the file.
• "fsync" performance on windows when writing to a memory mapped file also shows exponentially decreasing performance. I do not currently have an explanation for the exact mechanism(s) causing this.

鉴于这种观察到的性能，至少在我的用例中，这两个I/O选项不能代表可行的解决方案.

Given this observed performance, at least for my use case, these two I/O options would not represent feasible solutions.

按照 Greg的建议，我将在Windows磁盘缓存关闭的情况下重新运行测试，并且还将运行Howard提供的基准代码，以排除由于我自己的错误而导致结果出现偏差的可能性.

As per Greg's suggestion I will rerun the test with windows disk caching turned off, and I will also run Howard's provided benchmark code to exclude the possibility that results are skewed due to errors in my own.

更新2 我已经完成了测试，目前正在汇总结果.为了不写的完整历史"，我将用结果摘要，发现和一些结论来代替当前问题的内容.霍华德(Howard)对这个问题的回答以及在.NET代码旁边运行他的c基准代码的能力非常有用.这些结果与应用程序相关性很好. Rlb的回答使我对与磁盘相关的合理数字"有了更好的了解.谢谢.

UPDATE 2 I have completed the tests and am currently compiling the results. In order not to write "the full history of" I will be replacing the current contents of this question with a summary of the results, findings and some conclusions. The answers by Howard on this question, and the ability to run his c benchmark code next to the .NET code has been most useful. The results of those to applications correlate quite well. Rlb's answer has helped me to get a better feeling for what are "reasonable numbers" related to disks. Thanks.

部分问题仍未得到解答.尤其与在写入内存映射时观察到的性能下降(取决于文件大小)有关.它可能与搜寻/元数据刷新有关，但我尚不清楚原因/方式.

A part of the question remains unanswered. Particularly related to observed decreasing (and file size dependent) performance when writing to a memory map. It may be related to seeks/metadata flushes, but it is not yet clear to me why/how.

推荐答案

您看到同步运行的速度呈指数级下降，因为您认为这些并不是纯粹的顺序工作负载.由于您每次都从一个新文件开始，因此您的写入使文件不断增长，并且元数据需要在文件系统中进行更新.这需要多次查找，并且随着文件的增长，从文件末尾到元数据的查找将花费越来越长的时间.我还错误地将此问题发布在您的其他问题上，请参阅此处的完整答案: https://stackoverflow.com/a/18429712/894520

You're seeing an exponential decrease in speed on the sync runs because these aren't purely sequential workloads as you believe. Since you're starting with a new file each time, your writes are growing the file and the metadata needs to be updated in the filesystem. That requires multiple seeks, and as the file grows the seeks from the end of the file to the metadata take longer and longer. I also posted this on your other question by mistake, see the full answer there: https://stackoverflow.com/a/18429712/894520

这篇关于寻求解释/信息:Windows使用"fsync"来写I/O性能. (FlushFileBuffers)的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！