为什么我的 Perl 程序不能在 Windows 上创建超过 4 GB 的文件? [英] Why can't my Perl program create files over 4 GB on Windows?

问题描述

为什么输出到一个文件时文件的大小上限为 4 GB文件使用打印?我希望流媒体输出应该可以生成任意大小的文件.

Why is the size of files capped at 4 GB when outputting to a file using print? I would expect that with streaming output it should be possible to generate files of arbitrary size.

更新:ijw 和 查斯.欧文斯是对的.我以为 F: 驱动器是 NTFS 格式的，但实际上它使用的是 FAT32 文件系统.一世在另一个驱动器上试了一下，我可以生成一个 20 GB 的文本文件.在这种情况下没有限制.向大家道歉.

Update: ijw and Chas. Owens were correct. I thought the F: drive was NTFS formatted, but in fact it used the FAT32 filesystem. I tried it on another drive and I could generate a 20 GB text file. There are no limits in this case. Apologies to all.

详细信息:在研究回答问题时Stack Overflow 我需要衡量的性能使用 Perl 读取一个非常大的文本文件.为了测试阅读我需要一个大的文本文件，我写了一个小Perl 脚本生成文本文件并遇到意想不到的问题.输出文件增长直到达到 4GB.根据 Windows 资源管理器，在一次运行中的大小脚本为 4294967269 字节(磁盘上为 4294967296 字节).脚本继续运行，但文件不再增长.

Details: while researching for answering a question here on Stack Overflow I needed to measure the performance of reading a very large text file using Perl. In order to test the reading I needed a large text file and I wrote a small Perl script to generate the text file and ran into an unexpected problem. The output file grows until it reach 4 GB. According to Windows Explorer the size in one run of the script was 4294967269 bytes (and 4294967296 bytes on disk). The script continues, but the file no longer grows.

重要的是它只是一些:

print NUMBERS_OUTFILE $line;

其中 $line 是一个长字符串，末尾带有\n".这线的长度可以配置，对于这个问题;例如250 个字符或 34000 个字符.NUMBERS_OUTFILE 是使用以下方式创建的文件句柄:

where $line is a long string with a "\n" at the end. The length of the line can be configured and is not critical for this problem; e.g. 250 characters or 34000 characters. NUMBERS_OUTFILE is a file handle created with:

open ( NUMBERS_OUTFILE,">F:\temp2\out1.txt")

驱动器 F:是 NTFS 格式并且在单独的物理上硬盘从装有操作系统的磁盘.

Drive F: is NTFS formatted and is on a separate physical hard disk from the disk with the operating system.

原因是什么，是否有解决方法?

What is the reason and is there a work-around?

完整的Perl 脚本和BAT 驱动程序脚本(HTML 格式为预标签).如果两个环境变量 MBSIZE 和OUTFILE 设置然后 Perl 脚本应该能够运行在 Windows 以外的其他平台上保持不变.

Full Perl script and BAT driver script (HTML formatted with the pre tag). If the two environment variables MBSIZE and OUTFILE are setup then the Perl script should be able to run unchanged on other platforms than Windows.

平台:来自 ActiveState 的 Perl 5.10.0；32 位；构建 1004.Windows XP x64 SP2，8 GB RAM，500 GB Green Caviar硬盘.

Platform: Perl 5.10.0 from ActiveState; 32 bit; build 1004. Windows XP x64 SP2, 8 GB RAM, 500 GB Green Caviar hard disks.

perl -V 说:

Summary of my perl5 (revision 5 version 10 subversion 0) configuration:
  Platform:
    osname=MSWin32, osvers=5.00, archname=MSWin32-x86-multi-thread
    uname=''
    config_args='undef'
    hint=recommended, useposix=true, d_sigaction=undef
    useithreads=define, usemultiplicity=define
    useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
    use64bitint=undef, use64bitall=undef, uselongdouble=undef
    usemymalloc=n, bincompat5005=undef
  Compiler:
    cc='cl', ccflags ='-nologo -GF -W3 -MD -Zi -DNDEBUG -O1 -DWIN32 -D_CONSOLE -DNO_ST
RICT -DHAVE_DES_FCRYPT -DUSE_SITECUSTOMIZE -DPRIVLIB_LAST_IN_INC -DPERL_IMPLICIT_CONTE
XT -DPERL_IMPLICIT_SYS -DUSE_PERLIO -DPERL_MSVCRT_READFIX',
    optimize='-MD -Zi -DNDEBUG -O1',
    cppflags='-DWIN32'
    ccversion='12.00.8804', gccversion='', gccosandvers=''
    intsize=4, longsize=4, ptrsize=4, doublesize=8, byteorder=1234
    d_longlong=undef, longlongsize=8, d_longdbl=define, longdblsize=10
    ivtype='long', ivsize=4, nvtype='double', nvsize=8, Off_t='__int64', lseeksize=8
    alignbytes=8, prototype=define
  Linker and Libraries:
    ld='link', ldflags ='-nologo -nodefaultlib -debug -opt:ref,icf  -libpath:"D:\Perl\
lib\CORE"  -machine:x86'
    libpth=\lib
    libs=  oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib  comdlg32.lib a
dvapi32.lib shell32.lib ole32.lib oleaut32.lib  netapi32.lib uuid.lib ws2_32.lib mpr.l
ib winmm.lib  version.lib odbc32.lib odbccp32.lib msvcrt.lib
    perllibs=  oldnames.lib kernel32.lib user32.lib gdi32.lib winspool.lib  comdlg32.l
ib advapi32.lib shell32.lib ole32.lib oleaut32.lib  netapi32.lib uuid.lib ws2_32.lib m
pr.lib winmm.lib  version.lib odbc32.lib odbccp32.lib msvcrt.lib
    libc=msvcrt.lib, so=dll, useshrplib=true, libperl=perl510.lib
    gnulibc_version=''
  Dynamic Linking:
    dlsrc=dl_win32.xs, dlext=dll, d_dlsymun=undef, ccdlflags=' '
    cccdlflags=' ', lddlflags='-dll -nologo -nodefaultlib -debug -opt:ref,icf  -libpat
h:"D:\Perl\lib\CORE"  -machine:x86'


Characteristics of this binary (from libperl):
  Compile-time options: MULTIPLICITY PERL_DONT_CREATE_GVSV
                        PERL_IMPLICIT_CONTEXT PERL_IMPLICIT_SYS
                        PERL_MALLOC_WRAP PL_OP_SLAB_ALLOC USE_ITHREADS
                        USE_LARGE_FILES USE_PERLIO USE_SITECUSTOMIZE
  Locally applied patches:
        ActivePerl Build 1004 [287188]
        33741 avoids segfaults invoking S_raise_signal() (on Linux)
        33763 Win32 process ids can have more than 16 bits
        32809 Load 'loadable object' with non-default file extension
        32728 64-bit fix for Time::Local
  Built under MSWin32
  Compiled at Sep  3 2008 13:16:37
  @INC:
    D:/Perl/site/lib
    D:/Perl/lib
.

推荐答案

嗯，这很奇怪.至少在 OS X 和 Linux 上，限制是由文件系统强加的.也许 Win32 上的 Activestate Perl 没有编译大文件支持?你能把运行 perl -V 的结果贴出来吗?

Hmm, that is odd. At least on OS X and Linux, the limit is imposed by the filesystem. Perhaps Activestate Perl on Win32 is not compiled with largefile support? Could you post the result of running perl -V?

我们关心的输出部分是

Platform:
osname=MSWin32, osvers=5.00, archname=MSWin32-x86-multi-thread
uname=''
config_args='undef'
hint=recommended, useposix=true, d_sigaction=undef
useithreads=define, usemultiplicity=define
useperlio=define, d_sfio=undef, uselargefiles=define, usesocks=undef
use64bitint=undef, use64bitall=undef, uselongdouble=undef
usemymalloc=n, bincompat5005=undef

具体来说，uselargefiles=define.这个特性被定义(即打开)的事实意味着 Perl 将使用一个无符号的 64 位整数作为文件偏移量.从理论上讲，这可以支持高达 16 艾字节(17,179,869,184 GB)的文件；但是，文件系统限制通常会在您达到该限制之前发挥作用.

Specifically, uselargefiles=define. The fact that this feature is defined (i.e. turned on) means that Perl will use an unsigned 64 bit integer for file offsets. This, theoretically, enables files up to 16 exabytes (17,179,869,184 gigabytes); however, filesystem limits often come into play before you reach that limit.

这篇关于为什么我的 Perl 程序不能在 Windows 上创建超过 4 GB 的文件?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！