什么是C ++中高性能顺序文件I / O的最快方法？ [英] What is the Fastest Method for High Performance Sequential File I/O in C++?

问题描述

假设...

输出：

文件已打开... br />
数据被流到磁盘。内存中的数据位于大型连续缓冲区中。它以其原始形式直接从该缓冲区写入磁盘。缓冲区的大小是可配置的，但在流的持续时间内是固定的。缓冲区一个接一个地写入文件。

...文件已关闭。

输入： / b>

一个大文件（如上所述顺序写入）从头到尾从磁盘读取。

---

在C ++中实现最快的顺序文件I / O是否有普遍接受的准则？

• 选择最佳缓冲区大小的指南


• 太抽象以暴露特定平台的复杂性，或者可以假设它们是最优的？


• 异步I / O始终优于同步？如果应用程序不受CPU限制，该怎么办？

我意识到这将有平台特定的注意事项。我欢迎一般准则以及特定平台的一般准则。

（我对Win x64的最直接兴趣，但我也对Solaris和Linux上的评论感兴趣）

解决方案

有没有普遍接受的指导方针来实现C ++中最快的顺序文件I / / p>

规则0：测量。使用所有可用的分析工具，并了解他们。在编程中几乎是一个命令，如果你没有测量它，你不知道它有多快，而对于I / O来说更是如此。如果可能，请务必在实际工作条件下测试。对于I / O系统没有竞争的过程可以被过度优化，对于在实际负载下不存在的条件进行微调。

1. 使用映射内存，而不是写入文件。这并不总是更快，但它允许通过避免不必要的复制，并利用操作系统对磁盘实际使用的知识的优势，以特定于操作系统但相对便携的方式优化I / O。 （如果您使用包装程序，而不是操作系统特定的API调用，则为可移植。）




2. 尽量使输出线性化。不得不在存储器周围找到要写入的缓冲区在优化的条件下可以具有显着的效果，因为缓存线，分页和其他存储器子系统问题将开始重要。如果您有大量缓冲区，请查看是否支持尝试为您进行线性化的 scatter-gather I / O 。

一些可能的注意事项：

• 选择最佳缓冲区大小的指南

启动器的页面大小，但可以从中调整。

• 像boost :: asio这样的可移植库过于抽象，无法暴露特定平台的复杂性
  ，假设是最佳的？

不要认为它是最佳的。这取决于图书馆在您的平台上的运行情况，以及开发人员为了加快速度所付出的努力。已经说过，便携式I / O库可以非常快，因为在大多数系统上存在快速抽象，并且通常可以得到涵盖许多基础的通用API。 Boost.Asio是我最有限的知识，相当精细调整其特定的平台上：有一个整个家庭的操作系统和操作系统变体特定的API的快速异步I / O（例如 epoll ， / dev / epoll ， kqueue ， Windows重叠I / O ），以及Asio将其全部

• 异步I / O总是优于同步？如果应用程序不受CPU限制，该怎么办？

异步I /在原始意义上比同步I / O。异步I / O的作用是确保代码不会浪费时间等待I / O完成。它在一般的方式比不浪费时间，即使用线程的其他方法更快，因为它会回调到您的代码，当I / O准备好，而不是之前。对于需要终止的空闲线程，没有错误的启动或疑虑。

Assuming the following for...
Output:
The file is opened...
Data is 'streamed' to disk. The data in memory is in a large contiguous buffer. It is written to disk in its raw form directly from that buffer. The size of the buffer is configurable, but fixed for the duration of the stream. Buffers are written to the file, one after another. No seek operations are conducted.
...the file is closed.

Input:
A large file (sequentially written as above) is read from disk from beginning to end.

---

Are there generally accepted guidelines for achieving the fastest possible sequential file I/O in C++?

Some possible considerations:

• Guidelines for choosing the optimal buffer size
• Will a portable library like boost::asio be too abstracted to expose the intricacies of a specific platform, or can they be assumed to be optimal?
• Is asynchronous I/O always preferable to synchronous? What if the application is not otherwise CPU-bound?

I realize that this will have platform-specific considerations. I welcome general guidelines as well as those for particular platforms.
(my most immediate interest in Win x64, but I am interested in comments on Solaris and Linux as well)

解决方案

Are there generally accepted guidelines for achieving the fastest possible sequential file I/O in C++?

Rule 0: Measure. Use all available profiling tools and get to know them. It's almost a commandment in programming that if you didn't measure it you don't know how fast it is, and for I/O this is even more true. Make sure to test under actual work conditions if you possibly can. A process that has no competition for the I/O system can be over-optimized, fine-tuned for conditions that don't exist under real loads.

1. Use mapped memory instead of writing to files. This isn't always faster but it allows the opportunity to optimize the I/O in an operating system-specific but relatively portable way, by avoiding unnecessary copying, and taking advantage of the OS's knowledge of how the disk actually being used. ("Portable" if you use a wrapper, not an OS-specific API call).

2. Try and linearize your output as much as possible. Having to jump around memory to find the buffers to write can have noticeable effects under optimized conditions, because cache lines, paging and other memory subsystem issues will start to matter. If you have lots of buffers look into support for scatter-gather I/O which tries to do that linearizing for you.

Some possible considerations:

• Guidelines for choosing the optimal buffer size

Page size for starters, but be ready to tune from there.

• Will a portable library like boost::asio be too abstracted to expose the intricacies of a specific platform, or can they be assumed to be optimal?

Don't assume it's optimal. It depends on how thoroughly the library gets exercised on your platform, and how much effort the developers put into making it fast. Having said that a portable I/O library can be very fast, because fast abstractions exist on most systems, and it's usually possible to come up with a general API that covers a lot of the bases. Boost.Asio is, to the best of my limited knowledge, fairly fine tuned for the particular platform it is on: there's a whole family of OS and OS-variant specific APIs for fast async I/O (e.g. epoll, /dev/epoll, kqueue, Windows overlapped I/O), and Asio wraps them all.

• Is asynchronous I/O always preferable to synchronous? What if the application is not otherwise CPU-bound?

Asynchronous I/O isn't faster in a raw sense than synchronous I/O. What asynchronous I/O does is ensure that your code is not wasting time waiting for the I/O to complete. It is faster in a general way than the other method of not wasting that time, namely using threads, because it will call back into your code when I/O is ready and not before. There are no false starts or concerns with idle threads needing to be terminated.

这篇关于什么是C ++中高性能顺序文件I / O的最快方法？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！