如何使CreateFile尽可能快 [英] How to make CreateFile as fast as possible

问题描述

我需要在启动时读取几千个小文件的内容。在Linux上，只使用fopen和阅读速度非常快。在Windows上，这种情况非常缓慢。

我已经使用ReadFileEx切换到使用重叠I / O（异步I / O），其中Windows在数据准备就绪时执行回调阅读。

然而，实际上成千上万的CreateFile调用本身仍然是一个瓶颈。请注意，我提供我自己的缓冲区，打开NO_BUFFERING标志，给SERIAL提示，等等。然而，到的CreateFile呼叫采取的若干秒10秒，而在Linux上一切都更快地完成。

有什么办法可以让这些文件更快地读取？

对CreateFile的调用是：

'pre> HFILE =的CreateFile（szFullFileName，
GENERIC_READ，
FILE_SHARE_READ | FILE_SHARE_WRITE，
NULL，
OPEN_EXISTING ，
FILE_ATTRIBUTE_NORMAL | FILE_FLAG_OVERLAPPED | FILE_FLAG_NO_BUFFERING | FILE_FLAG_SEQUENTIAL_SCAN，
NULL）;


解决方案

CreateFile kernel32.dll 中，$ c>与内核syscall有一些额外的开销 NtCreateFile 在 NTDLL.DLL 。这是 CreateFile 调用请求内核打开文件的真实函数。如果你需要打开大量的文件，避免Win32的特殊情况和路径转换，那么 NtOpenFile 会更有效率 - 这些东西不适用于束在反正一个目录中的文件的

  NTSYSAPI NTSTATUS NTAPI NtOpenFile（OUT HANDLE *的FileHandle，IN ACCESS_MASK DesiredAccess，IN OBJECT_ATTRIBUTES * ObjectAttributes ，OUT IO_STATUS_BLOCK * IoStatusBlock，IN ULONG ShareAccess，IN ULONG OpenOptions）; 
 
手柄; 
 OBJECT_ATTRIBUTES Oa = {0}; 
 UNICODE_STRING Name_U; 
 IO_STATUS_BLOCK IoSb; 
 
 RtlInitUnicodeString（& Name_U，Name）; 
 
 Oa.Length = sizeof Oa; 
 Oa.ObjectName =& Name_U; 
 Oa.Attributes = CaseInsensitive？ OBJ_CASE_INSENSITIVE：0; 
 Oa.RootDirectory = ParentDirectoryHandle; 
 
状态= NtOpenFile（安培;手柄，FILE_READ_DATA，&安培; OA，&安培; IOSB，FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE，FILE_SEQUENTIAL_ONLY）; 
  

主要缺点：Microsoft不支持此API在用户模式下使用。这就是说，同等功能的被记录在案用于内核模式使用，自1993年第一次发布Windows NT以来并没有改变。

NtOpenFile 还允许你打开一个相对于现有目录句柄的文件（在这个例子中是ParentDirectoryHandle），这样可以减少查找目录中的一些文件系统开销。

最后，如同Carey Gregory所说的那样，NTFS在处理大量文件的目录时可能会太慢。

I need to read the contents of several thousands of small files at startup. On linux, just using fopen and reading is very fast. On Windows, this happens very slowly.

I have switched to using Overlapped I/O (Asynchronous I/O) using ReadFileEx, where Windows does a callback when data is ready to read.

However, the actual thousands of calls to CreateFile itself are still a bottleneck. Note that I supply my own buffers, turn on the NO_BUFFERING flag, give the SERIAL hint, etc. However, the calls to CreateFile take several 10s of seconds, whereas on linux everything is done much faster.

Is there anything that can be done to get these files ready for reading more quickly?

The call to CreateFile is:

            hFile = CreateFile(szFullFileName,
                GENERIC_READ,
                FILE_SHARE_READ | FILE_SHARE_WRITE,
                NULL,
                OPEN_EXISTING,
                FILE_ATTRIBUTE_NORMAL | FILE_FLAG_OVERLAPPED | FILE_FLAG_NO_BUFFERING | FILE_FLAG_SEQUENTIAL_SCAN,
                NULL);

解决方案

CreateFile in kernel32.dll has some extra overhead compared to the kernel syscall NtCreateFile in ntdll.dll. This is the real function that CreateFile calls to ask the kernel to open the file. If you need to open a large number of files, NtOpenFile will be more efficient by avoiding the special cases and path translation that Win32 has-- things that wouldn't apply to a bunch of files in a directory anyway.

NTSYSAPI NTSTATUS NTAPI NtOpenFile(OUT HANDLE *FileHandle, IN ACCESS_MASK DesiredAccess, IN OBJECT_ATTRIBUTES *ObjectAttributes, OUT IO_STATUS_BLOCK *IoStatusBlock, IN ULONG ShareAccess, IN ULONG OpenOptions);

HANDLE Handle;
OBJECT_ATTRIBUTES Oa = {0};
UNICODE_STRING Name_U;
IO_STATUS_BLOCK IoSb;

RtlInitUnicodeString(&Name_U, Name);

Oa.Length = sizeof Oa;
Oa.ObjectName = &Name_U;
Oa.Attributes = CaseInsensitive ? OBJ_CASE_INSENSITIVE : 0;
Oa.RootDirectory = ParentDirectoryHandle;

Status = NtOpenFile(&Handle, FILE_READ_DATA, &Oa, &IoSb, FILE_SHARE_READ | FILE_SHARE_WRITE | FILE_SHARE_DELETE, FILE_SEQUENTIAL_ONLY);

Main downside: this API is not supported by Microsoft for use in user mode. That said, the equivalent function is documented for kernel mode use and hasn't changed since the first release of Windows NT in 1993.

NtOpenFile also allows you to open a file relative to an existing directory handle (ParentDirectoryHandle in the example) which should cut down on some of the filesystem overhead in locating the directory.

In the end, NTFS may just be too slow in handling directories with large numbers of files as Carey Gregory said.

这篇关于如何使CreateFile尽可能快的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！