什么是最好的Python Zip模块处理大文件？ [英] What Is The Best Python Zip Module To Handle Large Files?

问题描述

编辑：具体的压缩和提取速度。

Specifically compression and extraction speeds.

任何建议？

b $ b

推荐答案

所以我制作了一个随机大的zip文件：

So I made a random-ish large zipfile:

$ ls -l *zip
-rw-r--r--  1 aleax  5000  115749854 Nov 18 19:16 large.zip
$ unzip -l large.zip | wc
   23396   93633 2254735

即，116 MB包含23.4K文件，事情：

i.e., 116 MB with 23.4K files in it, and timed things:

$ time unzip -d /tmp large.zip >/dev/null

real    0m14.702s
user    0m2.586s
sys         0m5.408s

是系统提供的命令行unzip二进制 - 无疑是微调和优化为纯C可执行文件可以。然后（清理后/ tmp; - ）...：

this is the system-supplied commandline unzip binary -- no doubt as finely-tuned and optimized as a pure C executable can be. Then (after cleaning up /tmp;-)...:

$ time py26 -c'from zipfile import ZipFile; z=ZipFile("large.zip"); z.extractall("/tmp")'

real    0m13.274s
user    0m5.059s
sys         0m5.166s

...这是使用标准库的Python - 对CPU时间要求更高一些，但实际速度提高了10％以上， 。

...and this is Python with its standard library - a bit more demanding of CPU time, but over 10% faster in real, that is, elapsed time.

当然，你可以重复这样的测量（在你的特定平台上 - 如果CPU性能很差，例如ARM芯片慢，那么Python的额外需求CPU时间可能会使它变慢 - 和你的特定zip文件感兴趣，因为每个大zip文件将有一个非常不同的组合，很可能是性能）。但是对我来说这是什么意思，没有太多的空间来构建一个Python扩展比好旧的 zipfile - 因为Python使用它击败纯C ，system-included unzip！ - ）

You're welcome to repeat such measurements of course (on your specific platform -- if it's CPU-poor, e.g a slow ARM chip, then Python's extra demands of CPU time may end up making it slower -- and your specific zipfiles of interest, since each large zipfile will have a very different mix and quite possibly performance). But what this suggests to me is that there isn't that much space to build a Python extension much faster than good old zipfile -- since Python using it beats the pure-C, system-included unzip!-)

这篇关于什么是最好的Python Zip模块处理大文件？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！