自定义预取 [英] Custom Prefetch

问题描述

任何编程技巧，可移植或特定于 NT 和 Linux 更快？我在'提前'之前，先前，无论你更喜欢称之为机制，我可以控制代码中的两个操作系统在一个问题。

每个文件必须全部处理，即完整地处理其内容。我不知道Win32（NT）API类似于 madvise（）。

然而，我会建议一种方法。

首先，将Win32标志 FILE_FLAG_SEQUENTIAL_SCAN 传递给 CreateFile（）。这将允许Windows操作系统在打开文件时执行更好的缓冲。

使用 FILE_FLAG_SEQUENTIAL_SCAN ，一旦文件在内存中，文件解析器可以更快地运行。与Linux上的 madvise（）不同，由于使用了Win32标志，该文件不会在早期开始加载到内存中。

接下来，我们需要触发文件开始加载。通过使用 OVERLAPPED 结构和结构调用 ReadFileEx（）来异步读取文件的第一页> FileIOCompletionRoutine 函数。
$ b 您的 FileIOCompletionRoutine 可以简单地返回，或者您可以设置事件在重叠的结构中 - 阅读MSDN详细信息 ReadFileEx 获取详细信息。

因为它不会是严重的失败，如果预取没有完成，当你真的从文件中读取，最简单的实现将是火灾和遗忘 - 执行重叠的文件读取，然后从不检查结果。不过，请确保您将数据读取到有效的缓冲区中！

如果在读取前一个文件时对文件执行此操作，结果应该是下一个文件将开始分页。

请注意，这可能会降低您的表现。当下一个文件开始导入时，访问该文件的磁盘I / O将与当前正在解析的文件的磁盘I / O进行竞争。如果这两个文件在同一个磁盘上彼此物理距离很远，则预取的结果可能会随着驱动器磁头的寻找而延长。虽然现代驱动器有巨大的缓冲区，缓解了这一点，排队新文件的第一页可能会导致一个头部寻找。

bdonlan建议预取从处理异步加载文件的线程也是Win32的可行解决方案。

Any programmatic techniques, portable or specific to NT and Linux that get the result of number of large files loading faster? I am after a 'ahead of time', a prior, whatever you prefer to call it mechanisms that I can control in code for two OS in a question.

Each file has to be processed in full, i.e. completely in size and sequentially for its contents. The aim is to speed up some batch file processing.

解决方案

I am not aware of a Win32 (NT) API similar to madvise().

However, I would suggest an approach.

First, pass the Win32 flag FILE_FLAG_SEQUENTIAL_SCAN to CreateFile(). This will allow the Windows operating system to perform better buffering of the file once you have opened it.

With FILE_FLAG_SEQUENTIAL_SCAN, your file parser may operate more quickly once the file is in memory. Unlike madvise() on Linux, the file will not begin loading into memory any earlier due to the use of the Win32 flag.

Next, we need to trigger the file to begin loading. Asynchronously read the first page of the file by calling ReadFileEx() with an OVERLAPPED structure and a FileIOCompletionRoutine function.

Your FileIOCompletionRoutine can simply return, or you can set the event in the overlapped structure -- read the MSDN details of ReadFileEx for details.

Since it would not be a critical failure if the pre-fetch hasn't completed when you actually read from the file, the easiest implementation would be to "fire and forget" -- execute the overlapped file read and then never check the result of it. Be sure that you read the data into valid buffers, though!

If you perform this operation for a file while reading the previous file, the result should be that the next file will commence paging in.

Be aware that this may slow your performance. As the next file begins to page in, the disk I/O to access that file will compete with disk I/O for the file you are currently parsing. If the two files are physically distant from each other on the same disk, the result of pre-fetching might be additional delay as the drive head seeks. Although modern drives have huge buffers which mitigate this, queuing the first page of a new file is likely to cause a head seek.

bdonlan's suggestion of a 'pre-fetch' thread which loads the files asynchronously from the processing would be a workable solution for Win32, also.

这篇关于自定义预取的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！