为什么“啜饮”一个文件不是一个好习惯？ [英] Why is "slurping" a file not a good practice?

问题描述

为什么啜饮文件不是普通文本文件I / O的好习惯，何时有用呢？

Why is "slurping" a file not a good practice for normal text-file I/O, and when is it useful?

例如，为什么不应该'我使用这些吗？

For example, why shouldn't I use these?

File.read('/path/to/text.txt').lines.each do |line|
  # do something with a line
end

或

File.readlines('/path/to/text.txt').each do |line|
  # do something with a line
end

推荐答案

我们一次又一次地看到有关阅读文本文件以逐行处理的问题，使用读取或 readlines ，它通过一个动作将整个文件拉入内存。

Again and again we see questions asking about reading a text file to process it line-by-line, that use variations of read, or readlines, which pull the entire file into memory in one action.

读取 的文档说：

The documentation for read says:

打开文件，可选地寻找给定的偏移量，然后返回长度字节（默认为文件的其余部分）。 [...]

Opens the file, optionally seeks to the given offset, then returns length bytes (defaulting to the rest of the file). [...]

readlines 的文档说：

The documentation for readlines says:

将name指定的整个文件读取为单独的行，并返回数组中的这些行。 [...]

Reads the entire file specified by name as individual lines, and returns those lines in an array. [...]

拉入一个小文件并不是什么大问题，但有一点需要在内存中进行内存处理当传入数据的缓冲区增长，并占用CPU时间。此外，如果数据占用太多空间，操作系统必须参与以保持脚本运行并开始假脱机到磁盘，这将使程序瘫痪。在HTTPd（Web主机）或需要快速响应的东西上，它会削弱整个应用程序。

Pulling in a small file is no big deal, but there comes a point where memory has to be shuffled around as the incoming data's buffer grows, and that eats CPU time. In addition, if the data consumes too much space, the OS has to get involved just to keep the script running and starts spooling to disk, which will take a program to its knees. On a HTTPd (web-host) or something needing fast response it'll cripple the entire application.

Slurping通常基于对文件速度的误解I / O或者认为最好先读取缓冲区而不是一次读取一行。

Slurping is usually based on a misunderstanding of the speed of file I/O or thinking that it's better to read then split the buffer than it is to read it a single line at a time.

这里有一些测试代码来演示由此引起的问题啜。

Here's some test code to demonstrate the problem caused by "slurping".

将此保存为test.sh：

Save this as "test.sh":

echo Building test files...

yes "abcdefghijklmnopqrstuvwxyz 123456890" | head -c 1000       > kb.txt
yes "abcdefghijklmnopqrstuvwxyz 123456890" | head -c 1000000    > mb.txt
yes "abcdefghijklmnopqrstuvwxyz 123456890" | head -c 1000000000 > gb1.txt
cat gb1.txt gb1.txt > gb2.txt
cat gb1.txt gb2.txt > gb3.txt

echo Testing...

ruby -v

echo
for i in kb.txt mb.txt gb1.txt gb2.txt gb3.txt
do
  echo
  echo "Running: time ruby readlines.rb $i"
  time ruby readlines.rb $i
  echo '---------------------------------------'
  echo "Running: time ruby foreach.rb $i"
  time ruby foreach.rb $i
  echo
done

rm [km]b.txt gb[123].txt 

它会创建五个大小不一的文件。 1K文件易于处理，非常常见。过去曾经认为1MB文件很大，但现在它们很常见。 1GB在我的环境中很常见，并且会定期遇到超过10GB的文件，因此了解1GB及以上的情况非常重要。

It creates five files of increasing sizes. 1K files are easily processed, and are very common. It used to be that 1MB files were considered big, but they're common now. 1GB is common in my environment, and files beyond 10GB are encountered periodically, so knowing what happens at 1GB and beyond is very important.

将其另存为readlines.rb 。它没有做任何事情，只是在内部逐行读取整个文件，并将其附加到一个然后返回的数组中，看起来它很快，因为它全部用C语言编写：

Save this as "readlines.rb". It doesn't do anything but read the entire file line-by-line internally, and append it to an array that is then returned, and seems like it'd be fast since it's all written in C:

lines = File.readlines(ARGV.shift).size
puts "#{ lines } lines read"

将此保存为foreach.rb：

Save this as "foreach.rb":

lines = 0
File.foreach(ARGV.shift) { |l| lines += 1 }
puts "#{ lines } lines read"

正在运行<$我的笔记本电脑上有c $ c> sh ./test.sh ：

Building test files...
Testing...
ruby 2.1.2p95 (2014-05-08 revision 45877) [x86_64-darwin13.0]

读取1K文件：

Running: time ruby readlines.rb kb.txt
28 lines read

real    0m0.998s
user    0m0.386s
sys 0m0.594s
---------------------------------------
Running: time ruby foreach.rb kb.txt
28 lines read

real    0m1.019s
user    0m0.395s
sys 0m0.616s

读取1MB文件：

Running: time ruby readlines.rb mb.txt
27028 lines read

real    0m1.021s
user    0m0.398s
sys 0m0.611s
---------------------------------------
Running: time ruby foreach.rb mb.txt
27028 lines read

real    0m0.990s
user    0m0.391s
sys 0m0.591s

读取1GB文件：

Running: time ruby readlines.rb gb1.txt
27027028 lines read

real    0m19.407s
user    0m17.134s
sys 0m2.262s
---------------------------------------
Running: time ruby foreach.rb gb1.txt
27027028 lines read

real    0m10.378s
user    0m9.472s
sys 0m0.898s

读取2GB文件：

Running: time ruby readlines.rb gb2.txt
54054055 lines read

real    0m58.904s
user    0m54.718s
sys 0m4.029s
---------------------------------------
Running: time ruby foreach.rb gb2.txt
54054055 lines read

real    0m19.992s
user    0m18.765s
sys 0m1.194s

读取3GB文件：

Running: time ruby readlines.rb gb3.txt
81081082 lines read

real    2m7.260s
user    1m57.410s
sys 0m7.007s
---------------------------------------
Running: time ruby foreach.rb gb3.txt
81081082 lines read

real    0m33.116s
user    0m30.790s
sys 0m2.134s

注意 readlines如何每次文件大小增加时运行速度慢两倍，并且使用 foreach 线性减速。在1MB时，我们可以看到影响啜饮I / O的东西不会影响逐行读取。而且，由于1MB文件现在很常见，如果我们不提前考虑，很容易看到它们会在程序的整个生命周期内减慢文件的处理速度。这里或几秒钟它们发生一次并不多，但如果它们每分钟发生多次，则会在一年结束时产生严重的性能影响。

Notice how readlines runs twice as slow each time the file size increases, and using foreach slows linearly. At 1MB, we can see there's something affecting the "slurping" I/O that doesn't affect reading line-by-line. And, because 1MB files are very common these days, it's easy to see they'll slow the processing of files over the lifetime of a program if we don't think ahead. A couple seconds here or there aren't much when they happen once, but if they happen multiple times a minute it adds up to a serious performance impact by the end of a year.

多年前我在处理大型数据文件时遇到了这个问题。我正在使用的Perl代码会在加载文件时重新分配内存时会定期停止。重写代码以不污染数据文件，而是逐行读取和处理它，从超过五分钟的速度提高到小于一的速度，并教会了我一个重要的教训。

I ran into this problem years ago when processing large data files. The Perl code I was using would periodically stop as it reallocated memory while loading the file. Rewriting the code to not slurp the data file, and instead read and process it line-by-line, gave a huge speed improvement from over five minutes to run to less than one and taught me a big lesson.

啜饮文件有时很有用，特别是如果你必须跨行界限做一些事情，但是，如果你不得不这样做，那么花一些时间思考阅读文件的其他方法是值得的。例如，考虑维护从最后n行构建的小缓冲区并进行扫描。这样可以避免因尝试读取和保存整个文件而导致的内存管理问题。这在Perl相关博客 Perl Slurp-Eaze 它涵盖了whens和whys以证明使用完整的文件读取，并且适用于Ruby。

"slurping" a file is sometimes useful, especially if you have to do something across line boundaries, however, it's worth spending some time thinking about alternate ways of reading a file if you have to do that. For instance, consider maintaining a small buffer built from the last "n" lines and scan it. That will avoid memory management issues caused by trying to read and hold the entire file. This is discussed in a Perl-related blog "Perl Slurp-Eaze" which covers the "whens" and "whys" to justify using full file-reads, and applies well to Ruby.

出于其他出色的理由，不要啜饮你的文件，阅读如何搜索模式的文件文本并将其替换为给定值。

For other excellent reasons not to "slurp" your files, read "How to search file text for a pattern and replace it with a given value".

这篇关于为什么“啜饮”一个文件不是一个好习惯？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！