fwrite对于大量小写的效率 [英] efficiency of fwrite for massive numbers of small writes

问题描述

我有一个程序，使用 fwrite 保存许多大文件> 1GB它工作正常，但不幸的是由于数据的性质每次调用 fwrite 只能写入1到4个字节。结果是写可能需要一个小时，大部分时间似乎是由于syscall开销（或至少在fwrite的库函数中）。我有一个类似的问题与 fread 。

I have a program that saves many large files >1GB using fwrite It works fine, but unfortunately due to the nature of the data each call to fwrite only writes 1-4bytes. with the result that the write can take over an hour, with most of this time seemingly due to the syscall overhead (or at least in the library function of fwrite). I have a similar problem with fread.

有人知道任何现有的/库函数将缓冲这些写

Does anyone know of any existing / library functions that will buffer these writes and reads with an inline function, or is this another roll your own?

推荐答案

首先， fwrite（）是一个库而不是系统调用。其次，它已经缓冲了数据。

First of all, fwrite() is a library and not a system call. Secondly, it already buffers the data.

您可能想尝试增加缓冲区的大小。这是通过使用 setvbuf（） 。在我的系统上，这只能帮助一点点，但是YMMV。

You might want to experiment with increasing the size of the buffer. This is done by using setvbuf(). On my system this only helps a tiny bit, but YMMV.

如果 setvbuf（）可以做你自己的缓冲，并且一旦你已经积累了足够的数据，只调用 fwrite（）。这涉及更多的工作，但几乎肯定会加快写作，因为你自己的缓冲可以做得更轻量的 fwrite（）的。

If setvbuf() does not help, you could do your own buffering and only call fwrite() once you've accumulated enough data. This involves more work, but will almost certainly speed up the writing as your own buffering can be made much more lightweight that fwrite()'s.

编辑：如果任何人告诉你这是问题的纯粹数量 fwrite（）看见证据。更好的是，做你自己的性能测试。在我的计算机上，使用 fwrite（）的500,000,000个双字节写入需要11秒。这等同于大约90MB / s的吞吐量。

edit: If anyone tells you that it's the sheer number of fwrite() calls that is the problem, demand to see evidence. Better still, do your own performance tests. On my computer, 500,000,000 two-byte writes using fwrite() take 11 seconds. This equates to throughput of about 90MB/s.

最后但并非最不重要的是，我的测试中的11秒和您的问题中提到的一小时之间的巨大差异暗示了可能性你的代码中还有其他的东西，导致非常差的性能。

Last but not least, the huge discrepancy between 11 seconds in my test and one hour mentioned in your question hints at the possibility that there's something else going on in your code that's causing the very poor performance.

这篇关于fwrite对于大量小写的效率的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！