以块的形式写入文件 [英] Writing to file in chunks

问题描述

嗨


我正在尝试将字节写入chunkSize块中的文件。这对于大型文件来说非常慢，例如9MB，我需要找到一种方法让它更快。

Hi
I'm trying to write bytes to a file in chunks of chunkSize. This is really slow with large files, for example 9MB, and I need to find a way to make it much faster.

For i As Integer = 0 To If(remainder = 0, 0, 1) + ((contents.Length - remainder) / blockSize)
    If contents.Length < i * blockSize Then
        IO.File.WriteAllBytes(outputPath, contents.Skip(i * blockSize).Take(blockSize).ToArray())
    Else
        IO.File.WriteAllBytes(outputPath, contents.Skip(i * blockSize).Take(contents.Length - (i * blockSize)).ToArray())
    End If
Next

想法是写入blockSize（4096）字节，或者最后留下的任何内容字节数不是blockSize的完美倍数。



我尝试过：



这个函数的各种形式

The idea is to either write blockSize (4096) bytes, or whatever's left at the end if the number of bytes isn't a perfect multiple of blockSize.

What I have tried:

Various forms of this function

推荐答案

你为什么这样做？

你所做的只是放慢了整个事情可怕 - 因为你每次编写时都必须分配一个新数组，并且你销毁文件中的所有先前数据： ;什么WriteAllBytes做的！

Why are you doing that?
All you are doing is slowing the whole thing down horribly - because you have to allocate a new array each time you write, and you destroy all previous data in the file: that;s what WriteAllBytes does!

Quote：

创建一个新文件，将指定的字节数组写入文件，然后关闭文件。如果目标文件已经存在，则会被覆盖。

Creates a new file, writes the specified byte array to the file, and then closes the file. If the target file already exists, it is overwritten.

File.WriteAllBytes Method（String，Byte []）（System.IO） [ ^ ]

既然你的块size是4096字节，这意味着你要分配超过2000个新数组才能做到这一点！



首先，对于.NET，9MB不是那么大的文件：一个带有9MB阵列的WriteAllBytes将与文件系统可以处理的速度一样快（而且速度非常快，它比块大小和缓存知道的要多得多！）。

如果你想以块的形式写，然后在循环外打开一个流，并使用 FileStream.Write方法（Byte []，Int32，Int32）（System.IO） [ ^ ]输出每个块 - 它为此提供偏移和长度参数，而不需要为每个块分配内存。但是......它仍然比一次写下来慢！

File.WriteAllBytes Method (String, Byte[]) (System.IO)[^]
Since your block size is 4096 bytes, that means you allocate over 2000 new arrays in order to do this!

First off, for .NET, 9MB is not that large a file: a single WriteAllBytes with a 9MB array will be as quick as the file system can cope with (and that's pretty fast, it knows a lot more about block sizes and caches than you do!).
Second if you want to write in chunks, then open a single stream outside the loop, and use FileStream.Write Method (Byte[], Int32, Int32) (System.IO)[^] to output each chunk - it provides offset and length parameters for just that purpose without needing to allocate memory for each chunk. But ... it'll still be slower than writing it all at once!

这篇关于以块的形式写入文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！