在Delphi中读取巨大文件的最快方法是什么？ [英] What is the fastest way for reading huge files in Delphi?
问题描述
程序内部使用TMemoryStream存储和处理从文件读取的块。通过TFileStream读取数据，如下所示：
FileStream.Position：= Offset;
MemoryStream.CopyFrom（FileStream，Size）;
这样做不错，但不幸的是，随着文件变大，变得越来越慢。文件大小从几兆字节开始，但是经常达到数十GB。读取的块大小约为100千字节。
该文件的内容只能由我的程序读取。这是当时访问该文件的唯一程序。此外，这些文件存储在本地，因此这不是一个网络问题。
我在Windows XP盒子上使用Delphi 2007。
我可以如何加快此文件访问速度？
编辑：
- 无论文件的哪一部分都被读取，大文件的文件访问速度很慢。
- 程序通常不会顺序读取文件。大块的顺序是用户驱动的，无法预测。
- 从大文件读取一个块，而不是从一个小文件中读取一个相当大的块，总是较慢。 li>
- 我正在谈论从文件中读取一个块的性能，而不是处理整个文件所需的整体时间。对于较大的文件来说，后者显然需要更长时间，但这不是问题。
我需要向大家道歉： 根据建议使用内存映射文件实现文件访问后，原来没有太大的区别。但是在我添加了更多的时间码之后，也没有文件访问减慢了程序。无论文件大小如何，文件访问实际上几乎保持不变。用户界面的某些部分（我还没有识别）似乎有大量数据的性能问题，但是当我第一次对进程进行定时时，我无法看到差异。
我很抱歉确定瓶颈。
如果您打开帮助主题 CreateFile （）WinAPI函数，你会发现有趣的如FILE_FLAG_NO_BUFFERING和FILE_FLAG_RANDOM_ACCESS。您可以与他们一起玩，以获得一些表演。
接下来，复制文件数据（甚至大小为100Kb）是一个额外的步骤，可以减慢操作速度。最好使用创建文件映射和MapViewOfFile函数获取准备使用的数据指针。这样避免复制，也可能获得某些性能优势（但您需要仔细测量速度）。
My program needs to read chunks from a huge binary file with random access. I have got a list of offsets and lengths which may have several thousand entries. The user selects an entry and the program seeks to the offset and reads length bytes.
The program internally uses a TMemoryStream to store and process the chunks read from the file. Reading the data is done via a TFileStream like this:
FileStream.Position := Offset;
MemoryStream.CopyFrom(FileStream, Size);
This works fine but unfortunately it becomes increasingly slower as the files get larger. The file size starts at a few megabytes but frequently reaches several tens of gigabytes. The chunks read are around 100 kbytes in size.
The file's content is only read by my program. It is the only program accessing the file at the time. Also the files are stored locally so this is not a network issue.
I am using Delphi 2007 on a Windows XP box.
What can I do to speed up this file access?
edit:
- The file access is slow for large files, regardless of which part of the file is being read.
- The program usually does not read the file sequentially. The order of the chunks is user driven and cannot be predicted.
- It is always slower to read a chunk from a large file than to read an equally large chunk from a small file.
- I am talking about the performance for reading a chunk from the file, not about the overall time it takes to process a whole file. The latter would obviously take longer for larger files, but that's not the issue here.
I need to apologize to everybody: After I implemented file access using a memory mapped file as suggested it turned out that it did not make much of a difference. But it also turned out after I added some more timing code that it is not the file access that slows down the program. The file access takes actually nearly constant time regardless of the file size. Some part of the user interface (which I have yet to identify) seems to have a performance problem with large amounts of data and somehow I failed to see the difference when I first timed the processes.
I am sorry for being sloppy in identifying the bottleneck.
If you open help topic for CreateFile() WinAPI function, you will find interesting flags there such as FILE_FLAG_NO_BUFFERING and FILE_FLAG_RANDOM_ACCESS . You can play with them to gain some performance.
Next, copying the file data, even 100Kb in size, is an extra step which slows down operations. It is a good idea to use CreateFileMapping and MapViewOfFile functions to get the ready for use pointer to the data. This way you avoid copying and also possibly get certain performance benefits (but you need to measure speed carefully).
这篇关于在Delphi中读取巨大文件的最快方法是什么？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
