Perl:写速度之谜? [英] Perl: write speed mystery?

问题描述

输出速率怎么能高于硬盘写入速率?

更新 1:我更改了以下内容:

1. 关闭杀毒软件.没有变化.

2. 插入新的物理磁盘并使用第一个分区为测试.(用于初始测试的磁盘位于最后一个分区，与系统分区分开，但是在同一个物理磁盘上.).结果:有相同循环模式，但系统不再测试期间无响应.写入速度为稍高(可能是由于使用了第一个分区和/或不再干扰系统划分).初步结论:有某种来自系统分区的干扰.

3. 安装了 64 位 Perl.循环消失了一切都在 2 秒的时间范围内稳定:55% CPU单核，写入速度约 65 MB/s.

4. 在带有 64 位 Perl 的原始驱动器上尝试过.结果:中间某个地方.8 秒的周期，CPU 20-50%，35 - 65 MB/秒(而不是 0-100% 的深循环，0 -120 MB/秒).系统只是轻微地没有反应.写入速度为 50 MB/秒.这支持了干涉理论.

5. 在 Perl 脚本中刷新.还没试过.

<小时>

好的，我克服了

这是怎么回事?

<小时>

完整的Perl 脚本和BAT 驱动程序脚本(HTML 格式为预标签).如果两个环境变量 MBSIZE 和OUTFILE 设置然后 Perl 脚本应该能够运行在 Windows 以外的其他平台上保持不变.

平台:来自 ActiveState 的 Perl 5.10.0；(最初是 32 位，后来是 64 位)；构建 1004.Windows XP x64 SP2，无页面文件，8 GB RAM，AMD 四核 CPU，500 GB Green Caviar 硬盘(写入速度 85MB/s?).

解决方案

我和其他所有人都说问题是缓冲区填充然后清空.尝试打开 autoflush 以避免使用缓冲区(在 Perl 中):

#!/usr/bin/perl使用严格；使用警告；使用 IO::Handle;我的 $filename = "output.txt";打开我的 $numbers_outfile, ">", $filename或者死无法打开 $filename: $!";$numbers_outfile->autoflush(1);#每次通过循环应该是1演出对于 (1 .. 20) {#每次虽然循环应该是1兆对于 (1 .. 1024) {#print 1 meg Zs打印 {$numbers_outfile} "Z" x (1024*1024)}}

如果您要打印一点，做一些工作，打印一点点，做一些工作等，缓冲区可能会很好.但如果您只是将数据爆破到磁盘上，它们可能会导致奇怪的行为.您可能还需要禁用文件系统正在执行的任何写缓存.

How can the output rate be higher than hard disk write rate?

Update 1: I have changed the following:

1. Turned off antivirus. No change.

2. Inserted new physical disk and used the first partition for the test. (The disk for the initial test was on the last partition, separate from the system partition, but on the same physical disk.). Result: there is the same cyclic pattern, but the system is no longer unresponsive during the test. The write speed is somewhat higher (could be due to using the first partition and/or no longer interference with the system partition). Preliminary conclusion: there was some kind of interference from the system partition.

3. Installed 64 bit Perl. The cycles are gone and everything is stable on a 2 second timescale: 55% CPU on the single core, write speed about 65 MB/s.

4. Tried on the original drive with 64 bit Perl. Result: somewhere in-between. Cycles of 8 seconds, CPU 20-50%, 35 - 65 MB/sec (instead of deep cycles of 0-100%, 0 - 120 MB/sec). The system is only mildly unresponsive. Write speed is 50 MB/sec. This supports the interference theory.

5. Flushing in the Perl script. Not tried yet.

---

OK, I got past the first hurdle. I have written a Perl script that can generate a very large text file (e.g. 20 GB) and is essentially just a number of:

print NUMBERS_OUTFILE $line;

where $line is a long string with a "\n" at the end.

When the Perl script starts the write rate is about 120 MB/s (consistent between what is computed by the script, Process Explorer and "IO Write Bytes/sec" for process Perl in Performance Monitor.) and 100% CPU on the single core it is running on. This rate is, I believe, higher than write speed of the hard disk.

Then after some time (e.g. 20 seconds and 2.7 GB written) the whole system becomes very unresponsive and CPU drops to 0%. This last for e.g. 30 seconds. The average write speed over these two phases is consistent with the write speed of the hard disk. The times and sizes mentioned in this paragraph varies a lot from run to run. The range 1 GB to 4.3 GB for the first phase has been observed so far. Here is a transcript for the run with 4.3 GB.

There are several of these cycles for a 9.2 GB text file generated in the test:

What is going on?

---

Full Perl script and BAT driver script (HTML formatted with the pre tag). If the two environment variables MBSIZE and OUTFILE are setup then the Perl script should be able to run unchanged on other platforms than Windows.

Platform: Perl 5.10.0 from ActiveState; (initially 32 bit, later 64 bit); build 1004. Windows XP x64 SP2, no page file, 8 GB RAM, AMD quad core CPU, 500 GB Green Caviar hard disks (write speed 85 MB/s?).

解决方案

I am with everyone else who is saying that the problem is buffers filling and then emptying. Try turning on autoflush to avoid having a buffer (in Perl):

#!/usr/bin/perl

use strict;
use warnings;

use IO::Handle;

my $filename = "output.txt";

open my $numbers_outfile, ">", $filename
    or die "could not open $filename: $!";

$numbers_outfile->autoflush(1);

#each time through the loop should be 1 gig
for (1 .. 20) {
    #each time though the loop should be 1 meg
    for (1 .. 1024) {
        #print 1 meg of Zs
        print {$numbers_outfile} "Z" x (1024*1024)
    }
}

Buffers can be good if you are going to print a little, do so work, print a litte, do some work, etc. But if you are just going to be blasting data onto disk, they can cause odd behavior. You may also need to disable any write caching your filesystem is doing.

这篇关于Perl:写速度之谜?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！