如何让 CreateFile 尽可能快 [英] How to make CreateFile as fast as possible

问题描述

我需要在启动时读取数千个小文件的内容.在linux上，只使用fopen，读取速度非常快.在 Windows 上，这发生得非常缓慢.

I need to read the contents of several thousands of small files at startup. On linux, just using fopen and reading is very fast. On Windows, this happens very slowly.

我已改用使用 ReadFileEx 的重叠 I/O(异步 I/O)，当数据准备好读取时，Windows 会执行回调.

I have switched to using Overlapped I/O (Asynchronous I/O) using ReadFileEx, where Windows does a callback when data is ready to read.

然而，实际上对 CreateFile 本身的数千次调用仍然是一个瓶颈.请注意，我提供了自己的缓冲区，打开了 NO_BUFFERING 标志，提供了 SERIAL 提示等.但是，对 CreateFile 的调用需要几十秒，而在 linux 上，一切都完成得更快.

However, the actual thousands of calls to CreateFile itself are still a bottleneck. Note that I supply my own buffers, turn on the NO_BUFFERING flag, give the SERIAL hint, etc. However, the calls to CreateFile take several 10s of seconds, whereas on linux everything is done much faster.

有什么办法可以让这些文件准备好以便更快地阅读吗?

Is there anything that can be done to get these files ready for reading more quickly?

对 CreateFile 的调用是:

The call to CreateFile is:

            hFile = CreateFile(szFullFileName,
                GENERIC_READ,
                FILE_SHARE_READ | FILE_SHARE_WRITE,
                NULL,
                OPEN_EXISTING,
                FILE_ATTRIBUTE_NORMAL | FILE_FLAG_OVERLAPPED | FILE_FLAG_NO_BUFFERING | FILE_FLAG_SEQUENTIAL_SCAN,
                NULL);

推荐答案

kernel32.dll中的

CreateFile与内核系统调用NtCreateFile<相比有一些额外的开销ntdll.dll 中的/code>.这是CreateFile 调用来请求内核打开文件的真正函数.如果您需要打开大量文件，NtOpenFile 将通过避免 Win32 具有的特殊情况和路径转换来提高效率——这些东西不适用于目录中的一堆文件无论如何.

CreateFile in kernel32.dll has some extra overhead compared to the kernel syscall NtCreateFile in ntdll.dll. This is the real function that CreateFile calls to ask the kernel to open the file. If you need to open a large number of files, NtOpenFile will be more efficient by avoiding the special cases and path translation that Win32 has-- things that wouldn't apply to a bunch of files in a directory anyway.

NTSYSAPI NTSTATUS NTAPI NtOpenFile(OUT HANDLE *FileHandle, IN ACCESS_MASK DesiredAccess, IN OBJECT_ATTRIBUTES *ObjectAttributes, OUT IO_STATUS_BLOCK *IoStatusBlock, IN ULONG ShareAccess, IN ULONG OpenOptions);

HANDLE Handle;
OBJECT_ATTRIBUTES Oa = {0};
UNICODE_STRING Name_U;
IO_STATUS_BLOCK IoSb;

RtlInitUnicodeString(&Name_U, Name);

Oa.Length = sizeof Oa;
Oa.ObjectName = &Name_U;
Oa.Attributes = CaseInsensitive ? OBJ_CASE_INSENSITIVE : 0;
Oa.RootDirectory = ParentDirectoryHandle;

Status = NtOpenFile(&Handle, FILE_READ_DATA, &Oa, &IoSb, FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE, FILE_SEQUENTIAL_ONLY);

主要缺点:Microsoft 不支持在用户模式下使用此 API.也就是说，等效的功能 已记录用于内核模式，并且自 1993 年第一次发布 Windows NT 以来没有改变.

Main downside: this API is not supported by Microsoft for use in user mode. That said, the equivalent function is documented for kernel mode use and hasn't changed since the first release of Windows NT in 1993.

NtOpenFile 还允许您打开相对于现有目录句柄(示例中的 ParentDirectoryHandle)的文件，这应该会减少定位目录时的一些文件系统开销.

NtOpenFile also allows you to open a file relative to an existing directory handle (ParentDirectoryHandle in the example) which should cut down on some of the filesystem overhead in locating the directory.

最后，正如凯里·格雷戈里所说，NTFS 在处理包含大量文件的目录时可能太慢了.

In the end, NTFS may just be too slow in handling directories with large numbers of files as Carey Gregory said.

这篇关于如何让 CreateFile 尽可能快的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！