由于额外的读取，网络上的文件读取速度很慢 [英] Reading file over network slow due to extra reads
问题描述
我正在读取文件，我要么读取一行数据(1600次顺序读取17个字节)，要么读取一列数据(1600次读取17个字节，以1600 * 17 = 27,200字节分隔).该文件位于本地驱动器或远程驱动器上.我读取了10次，因此我希望在每种情况下都可以读取272,000字节的数据.
I'm reading a file and I either read a row of data (1600 sequential reads of 17 bytes) or a column of data (1600 reads of 17 bytes separated by 1600*17=27,200 bytes). The file is either on a local drive or a remote drive. I do the reads 10 times so I expect in each case to read in 272,000 bytes of data.
在本地驱动器上，我可以看到期望的结果.在远程驱动器上，按顺序读取时，我也会看到期望的结果，但是在读取列时，我看到大量的额外读取正在完成.它们的长度为32,768字节，似乎没有被使用，但是它们使读取的数据量从272,000字节跃升至79 MB到106 MB之间的任何位置.这是使用Process Monitor的输出:
On the local drive, I see what I expect. On the remote drive when reading sequentially I also see what I expect but when reading a column, I see a ton of extra reads being done. They are 32,768 bytes long and don't seem to be used but they make the amount of data being read jump from 272,000 bytes to anywhere from 79 MB to 106 MB. Here is the output using Process Monitor:
1:39:39.4624488 PM DiskSpeedTest.exe 89628 ReadFile \\BCCDC01\BCC-raid3\SeisWareInc Temp Dir\BPepers_Temp\Projects\PT_4\Horizons\BaseName3D_1\RR_AP SUCCESS Offset: 9,390,069, Length: 17
1:39:39.4624639 PM DiskSpeedTest.exe 89628 FASTIO_CHECK_IF_POSSIBLE \\BCCDC01\BCC-raid3\SeisWareInc Temp Dir\BPepers_Temp\Projects\PT_4\Horizons\BaseName3D_1\RR_AP SUCCESS Operation: Read, Offset: 9,390,069, Length: 17
1:39:39.4624838 PM DiskSpeedTest.exe 89628 ReadFile \\BCCDC01\BCC-raid3\SeisWareInc Temp Dir\BPepers_Temp\Projects\PT_4\Horizons\BaseName3D_1\RR_AP SUCCESS Offset: 9,388,032, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
1:39:39.4633839 PM DiskSpeedTest.exe 89628 ReadFile \\BCCDC01\BCC-raid3\SeisWareInc Temp Dir\BPepers_Temp\Projects\PT_4\Horizons\BaseName3D_1\RR_AP SUCCESS Offset: 9,417,269, Length: 17
1:39:39.4634002 PM DiskSpeedTest.exe 89628 FASTIO_CHECK_IF_POSSIBLE \\BCCDC01\BCC-raid3\SeisWareInc Temp Dir\BPepers_Temp\Projects\PT_4\Horizons\BaseName3D_1\RR_AP SUCCESS Operation: Read, Offset: 9,417,269, Length: 17
1:39:39.4634178 PM DiskSpeedTest.exe 89628 ReadFile \\BCCDC01\BCC-raid3\SeisWareInc Temp Dir\BPepers_Temp\Projects\PT_4\Horizons\BaseName3D_1\RR_AP SUCCESS Offset: 9,444,469, Length: 17
1:39:39.4634324 PM DiskSpeedTest.exe 89628 FASTIO_CHECK_IF_POSSIBLE \\BCCDC01\BCC-raid3\SeisWareInc Temp Dir\BPepers_Temp\Projects\PT_4\Horizons\BaseName3D_1\RR_AP SUCCESS Operation: Read, Offset: 9,444,469, Length: 17
1:39:39.4634529 PM DiskSpeedTest.exe 89628 ReadFile \\BCCDC01\BCC-raid3\SeisWareInc Temp Dir\BPepers_Temp\Projects\PT_4\Horizons\BaseName3D_1\RR_AP SUCCESS Offset: 9,441,280, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
1:39:39.4642199 PM DiskSpeedTest.exe 89628 ReadFile \\BCCDC01\BCC-raid3\SeisWareInc Temp Dir\BPepers_Temp\Projects\PT_4\Horizons\BaseName3D_1\RR_AP SUCCESS Offset: 9,471,669, Length: 17
1:39:39.4642396 PM DiskSpeedTest.exe 89628 FASTIO_CHECK_IF_POSSIBLE \\BCCDC01\BCC-raid3\SeisWareInc Temp Dir\BPepers_Temp\Projects\PT_4\Horizons\BaseName3D_1\RR_AP SUCCESS Operation: Read, Offset: 9,471,669, Length: 17
1:39:39.4642582 PM DiskSpeedTest.exe 89628 ReadFile \\BCCDC01\BCC-raid3\SeisWareInc Temp Dir\BPepers_Temp\Projects\PT_4\Horizons\BaseName3D_1\RR_AP SUCCESS Offset: 9,498,869, Length: 17
1:39:39.4642764 PM DiskSpeedTest.exe 89628 FASTIO_CHECK_IF_POSSIBLE \\BCCDC01\BCC-raid3\SeisWareInc Temp Dir\BPepers_Temp\Projects\PT_4\Horizons\BaseName3D_1\RR_AP SUCCESS Operation: Read, Offset: 9,498,869, Length: 17
1:39:39.4642922 PM DiskSpeedTest.exe 89628 ReadFile \\BCCDC01\BCC-raid3\SeisWareInc Temp Dir\BPepers_Temp\Projects\PT_4\Horizons\BaseName3D_1\RR_AP SUCCESS Offset: 9,498,624, Length: 32,768, I/O Flags: Non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal
请注意，如果I/O标志设置为非缓存，分页I/O，同步分页I/O，优先级:正常，则需要额外读取32,768.这些多余的读取会使它从272 KB增大到106 MB，并导致速度变慢.从本地文件读取或我正在读取一行时它们不会发生，因此它们都是顺序的.
Notice the extra reads of 32,768 with I/O Flags set to non-cached, Paging I/O, Synchronous Paging I/O, Priority: Normal. These extra reads are what take it from 272 KB to 106 MB and are causing the slowness. They don't happen when reading from a local file or if I'm reading a row so it's all sequential.
我尝试设置FILE_FLAG_RANDOM_ACCESS，但似乎无济于事.关于什么原因导致这些额外读取以及如何使它们停止的任何想法?
I've tried setting the FILE_FLAG_RANDOM_ACCESS but it doesn't seem to help. Any ideas on what is causing these extra reads and how to make them stop???
测试正在Vista 64位系统上运行.我可以为演示问题的程序提供源代码，也可以为进行测试的控制台程序提供源代码.
The tests are being run on a Vista 64 bit system. I can provide source code for a program to demonstrate the problem as well as a console program that does the tests.
推荐答案
我找到了答案.Windows确实通过页面缓存读取文件，因此当我读取17个字节时，它首先必须传输32K的完整页面，然后才能从页面缓存中复制我想要的17个字节.令人讨厌的表现！
I found the answer to this. Windows does file reads through the page cache so when I read 17 bytes, it first has to transfer a full page of 32K over and then can copy the 17 bytes I want out of the page cache. Nasty result on performance!
第一次在本地文件上进行读取实际上发生了相同的事情，因为在这种情况下，它仍然一次仍将整个页面加载到页面缓存中.但是第二次我在本地运行测试时，文件都已经在页面缓存中了，所以我看不到它.而且如果SuperFetch已打开并且我已经进行了一段时间的测试，Windows会在之前开始将文件加载到缓存中，甚至我还运行我的测试应用程序，所以我再也看不到该页面了读取完成.
The same thing is actually happening the first time the reads are done on a local file since in that case it does still load a full page at a time into the page cache. But the second time I run the test locally, the files are all already in the page cache so I don't see it. And if SuperFetch is turned on and I've been doing these tests for a while, Windows will start loading the file into the cache before I even run my test application so again I don't see the page reads being done.
因此，操作系统在幕后做了很多事情，这使得很难完成良好的性能测试！
So the operating system is doing a lot of things behind the scenes that makes it tough to get good performance testing done!
这篇关于由于额外的读取，网络上的文件读取速度很慢的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
