什么是最好的Python Zip模块处理大文件? [英] What Is The Best Python Zip Module To Handle Large Files?
问题描述
编辑:具体的压缩和提取速度。
Specifically compression and extraction speeds.
任何建议?
b $ b
推荐答案
所以我制作了一个随机大的zip文件:
So I made a random-ish large zipfile:
$ ls -l *zip
-rw-r--r-- 1 aleax 5000 115749854 Nov 18 19:16 large.zip
$ unzip -l large.zip | wc
23396 93633 2254735
即,116 MB包含23.4K文件,事情:
i.e., 116 MB with 23.4K files in it, and timed things:
$ time unzip -d /tmp large.zip >/dev/null
real 0m14.702s
user 0m2.586s
sys 0m5.408s
是系统提供的命令行unzip二进制 - 无疑是微调和优化为纯C可执行文件可以。然后(清理后/ tmp; - )...:
this is the system-supplied commandline unzip binary -- no doubt as finely-tuned and optimized as a pure C executable can be. Then (after cleaning up /tmp;-)...:
$ time py26 -c'from zipfile import ZipFile; z=ZipFile("large.zip"); z.extractall("/tmp")'
real 0m13.274s
user 0m5.059s
sys 0m5.166s
...这是使用标准库的Python - 对CPU时间要求更高一些,但实际速度提高了10%以上, 。
...and this is Python with its standard library - a bit more demanding of CPU time, but over 10% faster in real, that is, elapsed time.
当然,你可以重复这样的测量(在你的特定平台上 - 如果CPU性能很差,例如ARM芯片慢,那么Python的额外需求CPU时间可能会使它变慢 - 和你的特定zip文件感兴趣,因为每个大zip文件将有一个非常不同的组合,很可能是性能)。但是对我来说这是什么意思,没有太多的空间来构建一个Python扩展比好旧的 zipfile
- 因为Python使用它击败纯C ,system-included unzip! - )
You're welcome to repeat such measurements of course (on your specific platform -- if it's CPU-poor, e.g a slow ARM chip, then Python's extra demands of CPU time may end up making it slower -- and your specific zipfiles of interest, since each large zipfile will have a very different mix and quite possibly performance). But what this suggests to me is that there isn't that much space to build a Python extension much faster than good old zipfile
-- since Python using it beats the pure-C, system-included unzip!-)
这篇关于什么是最好的Python Zip模块处理大文件?的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!