Ruby - 批量读取文件 [英] Ruby - Read file in batches

问题描述

我正在读取一个 10mb 大小的文件，其中包含一些 ID.我将它们读入 ruby​​ 列表.我担心将来可能会导致内存问题，因为文件中的 id 数量可能会增加.有没有有效的批量读取大文件的方法?

I am reading a file that is 10mb in size and which contains some id's. I read them into a list in ruby. I am concerned that it might cause memory issues in the future, when the number of id's in file might increase. Is there a effective way of reading a large file in batches?

谢谢

推荐答案

没有通用的方法.

1) 可以分块读取文件:

1) you can read file by chunks:

File.open('filename','r') do |f|
  chunk = f.read(2048)
  ...
end

缺点:如果它在块之间，您可能会错过子字符串，即您查找SOME_TEXT"，但SOME_"是第一个 2048 字节块的最后 5 个字节，并且"TEXT" 是第 2 个块的 4 个字节

disadvantage: you can miss a substring if it'd be between chunks, i.e. you look for "SOME_TEXT", but "SOME_" is a last 5 bytes of 1st 2048-byte chunk, and "TEXT" is a 4 bytes of 2nd chunk

2) 可以逐行读取文件

2) you can read file line-by-line

File.open('filename','r') do |f|
  line = f.gets
  ...
end

缺点:这样会比第一种方法慢 2..5 倍

disadvantage: this way it'd be 2x..5x slower than first method

这篇关于Ruby - 批量读取文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！