更快地替代 Python 的 zipfile 模块? [英] Faster alternative to Python's zipfile module?

问题描述

是否有比 Python 2.7.4 zipfile 模块(带有 ZIP_DEFLATED)明显更快的替代方案，用于将大量文件压缩为单个 zip 文件?我看了一下 czipfile https://pypi.python.org/pypi/czipfile/1.0.0，但这似乎侧重于更快的解密(而不是压缩).

Is there a noticeably faster alternative to Python 2.7.4 zipfile module (with ZIP_DEFLATED) for zipping a large number of files into a single zip file? I had a look at czipfile https://pypi.python.org/pypi/czipfile/1.0.0, but that appears to be focused on faster decrypting (not compressing).

我经常需要处理大量图像文件(约 12,000 个 .exr 和 .tiff 文件的组合文件)，每个文件的大小在约 1MB - 6MB 之间(所有文件约 9 GB)成一个单独的 zip 文件以便装运.处理此压缩过程需要大约 90 分钟(在 Windows 7 64 位上运行).

I am routinely having to process a large number of image files (~12,000 files of a combination of .exr and .tiff files) with each file between ~1MB - 6MB in size (and ~9 GB for all the files) into a single zip file for shipment. This zipping takes ~90 minutes to process (running on Windows 7 64bit).

如果有人可以推荐一个不同的 python 模块(或者一个 C/C++ 库，甚至一个独立的工具)，它能够在比 zipfile 模块更短的时间内将大量文件压缩到一个 .zip 文件中，这将不胜感激(任何接近约 5-10%(或更多)的速度都会非常有帮助).

If anyone can recommend a different python module (or alternatively a C/C++ library or even a standalone tool) that would be able to compress a large number of files into a single .zip file in less time than the zipfile module, that would be greatly appreciated (anything close to ~5-10% faster (or more) would be very helpful).

推荐答案

正如 Patashu 提到的，外包给 7-zip 可能是最好的主意.

As Patashu mentions, outsourcing to 7-zip might be the best idea.

以下是一些示例代码，可帮助您入门:

Here's some sample code to get you started:

import os
import subprocess

path_7zip = r"C:\Program Files\7-Zip\7z.exe"
path_working = r"C:\temp"
outfile_name = "compressed.zip"
os.chdir(path_working)

ret = subprocess.check_output([path_7zip, "a", "-tzip", outfile_name, "*.txt", "*.py", "-pSECRET"])

正如martineau 提到的，您可以尝试使用压缩方法.本页提供了一些关于如何更改命令行参数的示例.

As martineau mentioned you might experiment with compression methods. This page gives some examples on how to change the command line parameters.

这篇关于更快地替代 Python 的 zipfile 模块?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！