使用stream用于缓冲输出以获得性能 [英] Using ofstream for buffered output to gain performance

问题描述

我需要编写一个程序，它会在输出文件中写入许多字符。
我的程序还需要写一个换行更好的格式。
我理解 ofstream 是一个缓冲流，如果我们使用一个缓冲流文件io，我们获得性能。但是，如果我们使用 std :: endl ，则输出将被刷新，并且由于缓冲输出，我们将丢失任何潜在的性能增益。

我想如果我使用'\\\
'换行，输出只会被刷新，当我们 std :: endl 。它是否正确？在文件输出期间有没有可以用来获得性能增益的技巧？

注意：我想在文件写操作完成时刷新缓冲输出。我认为这样我可以最小化文件I / O，从而可以获得性能。

解决方案

如果需要最大性能，则不应该混乱流的刷新：当流满时，流在内部刷新它们的缓冲器。这实际上比等待直到所有输出就绪，特别是对于大文件更有效：缓冲的数据被写入，而它仍然可能在内存中。如果你创建一个巨大的缓冲区，只有写入一次虚拟内存系统将把部分数据放到光盘上，而不是文件。

关于 std :: endl 的主要观点，是人们滥用它的行结束，这导致缓冲区冲刷，他们不知道性能的影响。 std :: endl 的目的是让人们控制在合理的点刷新文件。为了使这一点有效，他们需要知道他们在做什么。可悲的是，有太多的人不知道什么 std :: endl 是谁宣传它的使用作为一个行结束，以便它被用于许多地方，它是明显错误。

也就是说，以下是一些您可能想要提高性能的事情。我假设你需要格式化的输出（使用 std :: ofstream :: write（）不会给你）。

• 显然，除非你必须使用 std :: endl 。如果写代码已经存在并且在许多地方使用 std :: endl ，其中一些可能在您的控制之外，您可以使用过滤流缓冲区，其使用其内部缓冲区合理的大小，并且不将其 sync（）的调用转发到基础流缓冲器。虽然这涉及一个额外的副本，这是比一些杂乱的冲洗，因为这些是数量级更昂贵。


• 虽然它不应该影响 std :: ofstream ，调用 std :: ios_base :: sync_with_stdio（false）用于影响某些实现的性能。如果这有效果，你可以考虑使用不同的IOstream实现，因为在性能方面可能有更多的错误。


• 确保你使用 std :: locale 其 std :: codecvt< ...> 返回 true 调用 always_noconv（）时。这可以很容易地通过使用 std :: use_facet< std :: codecvt< char，char，stdd :: mbstate_t> >（out.get_loc（））。always_noconv（）。您可以使用 std :: locale（C）来获取 std :: locale


• 一些locale实现使用非常低效的数字facet实现，即使它们是相当不错的，默认实现 std :: num_put< ; char> facet仍然可以做你不需要的事情。特别是如果你的数字格式化是相当简单的，即你不继续改变格式化标志，你没有替换字符映射（即你不使用有趣的 std :: ctype< char> facet）等，使用自定义 std :: num_put< char> facet可能是合理的：创建快速但很容易用于整数类型的简单格式化函数和用于不在内部使用 snprintf（）的浮点的良好格式化函数。

有些人建议使用内存映射文件，但这只有在目标文件的大小预先知道时才有效。如果这是这种情况下，这是一个伟大的方式，还提高性能否则不值得麻烦。请注意，您可以通过创建一个自定义的 std :: streambuf ，使用内存映射文件（或更一般地，与任何类型的输出接口）的流格式化，映射接口。我发现内存映射有时在使用 std :: istream 时有效。在很多情况下，差异并不重要。

很久以前，我写了自己的IOStreams和locales实现，没有遇到一些性能问题（它可从我的网站，但它是一个有点陈旧，我没有触及它近10年）。有很多事情，可以改进这个实现仍然，但我没有一个最新的实现，我准备发布在某个地方。很快，希望 - 从近10年来，我一直在思考，虽然...

I need to write a program which will write many characters in a output file. My program will also need to write newline for better formatting. I understand ofstream is a buffered stream and if we use a buffered stream for file io, we gain performance. However, if we use std::endl the output will be flushed and we will loose any potential performance gain due to the buffered output.

I suppose if I use '\n' for new line the output will be only flushed when we will std::endl. Is this correct? And are there any tricks that can be used to get performance gain during file output?

Note: I want to flush the buffered output at the completion of the file write operations. I think in this way I can minimize file I/O and thus can gain performance.

解决方案

Generally, the user of stream classes shouldn't mess with the stream's flushing if maximum performance is wanted: the streams internally flush their buffer when it is full. This is actually more efficient than waiting until all output is ready, especially with large files: the buffered data is written while it is still likely to be in memory. If you create a huge buffer and only write it once the virtual memory system will have put parts of the data onto disc but not the file. It would need to be read from disc and written again.

The main point with respect to std::endl is that people abuse it a line ending which causes the buffer to flush and they are unaware of the performance implications. The intention of std::endl is that people are given control to flush files at reasonable points. For this to be effective they need to know what they are doing. Sadly, there were too many people ignorant of what std::endl does who advertised its use as a line ending such that it is used in many places where it is plain wrong.

That said, below are a number of things you might want to try to improve performance. I assume you need formatted output (which the use of std::ofstream::write() won't give you).

• Obviously, don't use std::endl unless you have to. If the writing code already exists and uses std::endl in many places, some of which possibly outside your control, you can use a filtering stream buffer which uses its internal buffer of reasonable size and which doesn't forward calls to its sync() function to the underlying stream buffer. Although this involves an extra copy, this is better than some spurious flushes as these are orders of magnitude more expensive.
• Although it shouldn't have an effect on std::ofstreams, calling std::ios_base::sync_with_stdio(false) used to affect the performance on some implementations. You'd want to look at using a different IOstream implementation if this has an effect because there are probably more things wrong with respect to performance.
• Make sure you are using a std::locale whose std::codecvt<...> returns true when calling its always_noconv(). This can easily be checked by using std::use_facet<std::codecvt<char, char, stdd::mbstate_t> >(out.get_loc()).always_noconv(). You can use std::locale("C") to get hold of an std::locale for which this should be true.
• Some locale implementations use very inefficient implementations of their numeric facets and even even if they are reasonably good, the default implementation of the std::num_put<char> facet may still do things you don't really need. Especially if your numeric formatting is reasonably simple, i.e. you don't keep changing formatting flags, you haven't replace mapping of characters (i.e. you don't use a funny std::ctype<char> facet), etc. it may be reasonable to use a custom std::num_put<char> facet: It is fairly easy to create a fast but simple formatting function for integer types and a good formatting function for floating points which doesn't use snprintf() internally.

Some people have suggested the use of memory mapped files but this only works reasonable when the size of the target file is known in advance. If this is the case this is a great way to also improve performance otherwise it isn't worth the bother. Note that you can use the stream formatting with memory mapped files (or, more generally, with any kind of output interface) by creating a custom std::streambuf which uses the memory mapping interface. I found memory mapping sometimes effective when using them with std::istreams. In many cases the differences don't really matter much.

A long time ago I wrote my own IOStreams and locales implementation which doesn't suffer from some of the performance problems mentioned above (it is available from my site but it is a bit stale and I haven't touched it for nearly 10 years now). There are lots of things which can be improved over this implementation still but I haven't an up to date implementation which I'd be ready to post somewhere. Soon, hopefully - something I keep thinking since nearly 10 years, though...

这篇关于使用stream用于缓冲输出以获得性能的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！