读取顺序文件 - 压缩文件与解压缩 [英] Read sequential file - Compressed file vs Uncompresses

问题描述

我正在寻找以最快的方式从磁盘读取序列文件。
我读一些帖子，如果我压缩文件使用例如，lz4，我可以实现更好的性能比读tje平面文件，因为我将最小化I / O操作。

I am looking for the fastest way to read a sequential file from disk. I read in some posts that if i compressed the file using for example, lz4, i could achieve better performance than read tje flat file, becausa i will minimize the i/o operations.

但是当我尝试这种方法，扫描lz4压缩文件给我一个很差的执行扫描平面文件。我没有尝试上面的lz4demo，但寻找它，我的代码是非常相似的。

But when i try this approach, scanning a lz4 compresses file give me a poor performqnce than scan the flat file. I did no try the lz4demo above, but looking for it, my code is very similar.

我发现这个基准：
http://skipperkongen.dk/2012/02/28/uncompressed-versus-compressed-read/
http：// code.google.com/p/lz4/source/browse/trunk/lz4demo.c?r=75

这是真的可以提高性能阅读a在未压缩的文件上压缩顺序文件？我做错了什么？

Is it really possible to improve performance reading a compresses sequential file over an uncompressed one? What am i doing wrong?

我

推荐答案

可以通过使用压缩来提高磁盘读取。

Yes, it is possible to improve disk read by using compression.

如果使用多线程读取器，这种效果最有可能发生：当一个线程从磁盘读取压缩数据时，另一个解码存储器内的先前压缩块。

This effect is most likely to happen if you use a multi-threaded reader : while one thread reads compressed data from disk, the other one decode the previous compressed block within memory.

考虑到LZ4的速度，解码操作可能在另一个线程完成读取下一个块之前完成。这样，您将实现带宽的改进，与测试文件的压缩率成正比。

Considering the speed of LZ4, the decoding operation is likely to finish before the other thread complete reading the next block. This way, you'll achieved a bandwidth improvement, proportional to the compression ratio of the tested file.

显然，在进行基准测试时还有其他的影响要考虑。例如，HDD的搜索次数比SSD大几个数量级，并且在不利的情况下，它可以成为定时的主要部分，将任何带宽优势降低到零。

Obviously, there are other effects to consider when benchmarking. For example, seek times of HDD are several order of magnitude larger than SSD, and under bad circumstances, it can become the dominant part of the timing, reducing any bandwidth advantage to zero.

这篇关于读取顺序文件 - 压缩文件与解压缩的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！