加快写入文件的速度. [英] Speeding up writes to file.

问题描述

我一直在从事一个项目，该项目需要以高速率将大量数据写入RAID驱动器.每个文件大小最大为50GB.我们没有达到足够高的数据速率，因此我们使用了4MB缓冲区大小的无缓冲写入，这是系统可能的最大大小.这样可以将写入速度提高到我们所需的速度.我们还尝试了预分配较大的文件大小，但这没有什么区别(我们尝试了50GB的整个文件大小，也尝试了1MB的较小文件，我们认为这是最大的预分配大小，这会有所不同). br/>
我不明白为什么大缓冲区比预分配有更大的差异，因为我认为预分配大小将意味着对磁盘的写入是在更大的块中完成的，而较大的缓冲区大小则意味着可以分配更多的数据在写入开始之前将其写入缓冲区.如果写作使事情变慢，那么预分配应该发挥最大作用.

有人可以解释一下吗?

谢谢，

解决方案

无缓冲写入意味着您没有使用任何操作系统文件系统缓冲技术，而是将原始文件流直接写入硬件. br/>
块大小越大，意味着您对写入数据的调用次数就越少.

看钱的一种方法是与金钱进行比较.

在您的左手中，有一堆硬币，块大小是硬币的面额，小块大小是一分钱，大块大小是1个硬币.

1个硬币.

您必须将10美元放入

一台机器.哪一个会更快的便士或美元硬币?

现在，介绍一个钱包/钱包，将其作为缓冲区，如果打开缓冲区，则必须先将每种硬币放入钱包中，然后才能取出并放入机器中.

当您使用较大的文件和较大的块大小时，缓冲对您没有帮助.如果您使用的是块大小小的文件，则缓冲区将使硬盘处理每个文件的时间变慢，并提供更快的软件来卸载其工作以保持文件移动.

有帮助吗?

看一下我的两篇文章，您将看到现实中有关块大小影响的示例；
QNAP NAS内存升级，硬件更改和性能优势 [ ^ ]


混合替代方案-它有所作为吗? [ 解决方案

Unbuffered writes means that you are not using any of the operating systems file systems buffering techniques and writing the the raw file stream straight to the hardware.

The larger the block size means that you are making less frequent calls for data to be written.

One way to look at it is by comparing to money.

In your left hand you have a pile of coins, the block size is the denomination of the coin, a small block size is a penny, a large block size is a

1 coin.

You have to put

10 dollars into a machine. Which will do it quicker pennies or dollar coins?

Now, introduce a purse/wallet, that it your buffer, if you turn buffering on, each coin must be put into the purse before it can be then taken out and put into the machine.

As you are working with a large file and large block sizes, buffering doesn''t help you. If you were using small files with small block sizes, the buffer allows the slow hard disk time to process each file and provides the faster software to offload its work to keep it moving.

Does that help?

Look at my two articles and you will see real world examples of the effect of block size;
QNAP NAS Memory Upgrade, Hardware Change and Performance Benefits[^]


The Hybrid Alternative - Did it make a difference?[^]

If you look at the graphs, the block size is along the X-axis, and in both benchmarks the buffering is disabled.

这篇关于加快写入文件的速度.的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！