如何从磁盘获得良好的并发读取性能 [英] How to obtain good concurrent read performance from disk
问题描述
我想问一个问题，然后用我自己的答案来跟踪它，也看看其他人有什么答案。
我们有两个大文件我们想同时从两个独立的线程读取。一个线程将顺序读取fileA，而另一个线程将顺序读取fileB。线程之间没有任何锁定或通信，它们都是按照它们的速度顺序读取，并且都立即丢弃它们读取的数据。
我们使用这个设置的经验在Windows上很差。两个线程的总吞吐量在2-3 MiB / sec左右。这个驱动似乎花费了大部分时间在这两个文件之间寻找反向和转发，据推测在每次搜索后读取很少。
如果我们禁用其中一个线程暂时看一个单线程的性能，然后我们得到更好的带宽（本机约45 MiB /秒）。所以很显然，双线程的糟糕表现是操作系统磁盘调度器的一个问题。
有没有什么可以提高并发线程读取性能？ / strong>也许通过使用不同的API或以某种方式调整操作系统磁盘调度程序参数。
一些细节：
在2GiB内存的机器上，每个文件的大小都是2 GiB。为了这个问题的目的，我们认为他们不被缓存和完全碎片整理。我们已经使用碎片整理工具，并重新启动，以确保这种情况。
我们没有使用特殊的API来读取这些文件。这个行为可以在不同的bog标准API中重复使用，比如Win32的CreateFile，C的fopen，C ++的std :: ifstream，Java的FileInputStream等等。每个线程都在一个循环打电话给阅读功能。我们已经从1KiB到128MiB之间的值从API中请求的每个字节的数量。改变这一点已经没有效果了，所以在每次磁盘寻道之后操作系统在物理上读取的数量不是由这个数字决定的。这正是应该预料的。单线程和双线程性能之间的巨大差异是可重复的Windows 2000，Windows XP（32位和64位） Windows Server 2003以及硬件RAID5和硬件RAID5。这个问题似乎在Windows I / O调度政策。根据我发现的这里有很多方法的操作系统安排磁盘请求。虽然Linux和其他人可以在不同的策略之间进行选择，但是在Vista Windows被锁定在单个策略之前：一个FIFO队列，其中所有的请求都在64 KB的块中分割。我相信这个策略是你遇到的问题的原因：调度程序会混合来自两个线程的请求，造成磁盘不同区域之间的连续查找。
现在，好消息是根据 Windows Vista Kernel 」这里和这里，Vista引入了更智能的磁盘调度程序，您可以在其中设置请求的优先级，并为您的进程分配一个最小的带宽。
坏消息是，我发现无法在以前的Windows版本中更改磁盘策略或缓冲区大小。另外，即使提高进程的磁盘I / O优先级也会提高对其他进程的性能，但是仍然存在线程相互竞争的问题。
我可以建议的是修改例如，你可以在你的线程B中使用这样的策略（类似于线程A）：
如果THREAD A从磁盘读取数据，则等待THREAD A停止读取或等待X ms
读取X ms（或Y MB）
再次停止读取和检查线程A的状态
您可以使用信号量进行状态检查，也可以使用perfmon计数器来获取实际磁盘队列的状态。
X和/或Y的值也可以通过检查实际的转移率并慢慢修改它们来自动调整，从而在应用程序运行在不同的机器和/或O.S.时最大化吞吐量。您可以发现缓存，内存或RAID级别会以某种方式影响它们，但是通过自动调整，您将始终在各种情况下获得最佳性能。
I'd like to ask a question then follow it up with my own answer, but also see what answers other people have.
We have two large files which we'd like to read from two separate threads concurrently. One thread will sequentially read fileA while the other thread will sequentially read fileB. There is no locking or communication between the threads, both are sequentially reading as fast as they can, and both are immediately discarding the data they read.
Our experience with this setup on Windows is very poor. The combined throughput of the two threads is in the order of 2-3 MiB/sec. The drive seems to be spending most of its time seeking backwards and forwards between the two files, presumably reading very little after each seek.
If we disable one of the threads and temporarily look at the performance of a single thread then we get much better bandwidth (~45 MiB/sec for this machine). So clearly the bad two-thread performance is an artefact of the OS disk scheduler.
Is there anything we can do to improve the concurrent thread read performance? Perhaps by using different APIs or by tweaking the OS disk scheduler parameters in some way.
Some details:
The files are in the order of 2 GiB each on a machine with 2GiB of RAM. For the purpose of this question we consider them not to be cached and perfectly defragmented. We have used defrag tools and rebooted to ensure this is the case.
We are using no special APIs to read these files. The behaviour is repeatable across various bog-standard APIs such as Win32's CreateFile, C's fopen, C++'s std::ifstream, Java's FileInputStream, etc.
Each thread spins in a loop making calls to the read function. We have varied the number of bytes requested from the API each iteration from values between 1KiB up to 128MiB. Varying this has had no effect, so clearly the amount the OS is physically reading after each disk seek is not dictated by this number. This is exactly what should be expected.
The dramatic difference between one-thread and two-thread performance is repeatable across Windows 2000, Windows XP (32-bit and 64-bit), Windows Server 2003, and also with and without hardware RAID5.
The problem seems to be in Windows I/O scheduling policy. According to what I found here there are many ways for an O.S. to schedule disk requests. While Linux and others can choose between different policies, before Vista Windows was locked in a single policy: a FIFO queue, where all requests where splitted in 64 KB blocks. I believe that this policy is the cause for the problem you are experiencing: the scheduler will mix requests from the two threads, causing continuous seek between different areas of the disk.
Now, the good news is that according to here and here, Vista introduced a smarter disk scheduler, where you can set the priority of your requests and also allocate a minimum badwidth for your process.
The bad news is that I found no way to change disk policy or buffers size in previous versions of Windows. Also, even if raising disk I/O priority of your process will boost the performance against the other processes, you still have the problems of your threads competing against each other.
What I can suggest is to modify your software by introducing a self-made disk access policy.
For example, you could use a policy like this in your thread B (similar for Thread A):
if THREAD A is reading from disk then wait for THREAD A to stop reading or wait for X ms
Read for X ms (or Y MB)
Stop reading and check status of thread A again
You could use semaphores for status checking or you could use perfmon counters to get the status of the actual disk queue. The values of X and/or Y could also be auto-tuned by checking the actual trasfer rates and slowly modify them, thus maximizing the throughtput when the application runs on different machines and/or O.S. You could find that cache, memory or RAID levels affect them in a way or the other, but with auto-tuning you will always get the best performance in every scenario.
这篇关于如何从磁盘获得良好的并发读取性能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
