将数据写入文件的最有效方法 [英] most efficient way to write data into a file

问题描述

我想将2TB数据写入一个文件，将来可能会达到PB级.

I want to write 2TB data into one file, in the future it might be a petabyte.

数据由所有'1'组成.例如，由"1111111111111......11111"组成的2TB数据(每个字节用'1'表示).

The data is composed of all '1'. For example, 2TB data consisting of "1111111111111......11111" (each byte is represented by '1').

以下是我的方式:

File.open("data",File::RDWR||File::CREAT) do |file|
  2*1024*1024*1024*1024.times do
  file.write('1')
  end
end

这意味着File.write被称为2TB倍.从Ruby的角度来看，有没有更好的方法来实现它?

That means, File.write is called 2TB times. From the point of Ruby, is there a better way to implement it?

推荐答案

您遇到了一些问题:

1. File::RDWR||File::CREAT始终取值为File::RDWR.您的意思是File::RDWR|File::CREAT(|而不是||).

1. File::RDWR||File::CREAT always evaluates to File::RDWR. You mean File::RDWR|File::CREAT (| rather than ||).

2*1024*1024*1024*1024.times do运行循环1024次，然后将循环的结果乘以左侧的内容.您是说(2*1024*1024*1024*1024).times do.

2*1024*1024*1024*1024.times do runs the loop 1024 times then multiplies the result of the loop by the stuff on the left. You mean (2*1024*1024*1024*1024).times do.

关于您的问题，通过一次写入1024个字节，我得到了显着的加速:

Regarding your question, I get significant speedup by writing 1024 bytes at a time:

File.open("data",File::RDWR|File::CREAT) do |file|
  buf = "1" * 1024
  (2*1024*1024*1024).times do
    file.write(buf)
  end
end

您可能会进行实验，找到比1024更佳的缓冲区大小.

You might experiment and find a better buffer size than 1024.

这篇关于将数据写入文件的最有效方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！