使用 Python 提取 tar 文件的最快方法 [英] Fastest way to extract tar files using Python
问题描述
我必须提取数百个 tar.bz 文件,每个文件大小为 5GB.所以尝试了以下代码:
I have to extract hundreds of tar.bz files each with size of 5GB. So tried the following code:
import tarfile
from multiprocessing import Pool
files = glob.glob('D:\\*.tar.bz') ##All my files are in D
for f in files:
tar = tarfile.open (f, 'r:bz2')
pool = Pool(processes=5)
pool.map(tar.extractall('E:\\') ###I want to extract them in E
tar.close()
但是代码有类型错误:类型错误:map() 至少需要 3 个参数(给定 2 个)
But the code has type error: TypeError: map() takes at least 3 arguments (2 given)
我该如何解决?还有其他加速提取的想法吗?
How can I solve it? Any further ideas to accelerate extracting?
推荐答案
您需要将 pool.map(tar.extractall('E:\\')
更改为类似 pool 的内容.map(tar.extractall(),"list_of_all_files")
You need to change pool.map(tar.extractall('E:\\')
to something like pool.map(tar.extractall(),"list_of_all_files")
请注意,map()
需要 2 个参数,第一个是函数,第二个是可迭代的,然后将函数应用于可迭代的每个项目并返回结果列表.
Note that map()
takes 2 argument first is a function , second is a iterable , and Apply function to every item of iterable and return a list of the results.
您需要将 TarInfo
对象传递给另一个进程:
Edit : you need to pass a TarInfo
object into the other process :
def test_multiproc():
files = glob.glob('D:\\*.tar.bz2')
pool = Pool(processes=5)
result = pool.map(read_files, files)
def read_files(name):
t = tarfile.open (name, 'r:bz2')
t.extractall('E:\\')
t.close()
>>>test_multiproc()
这篇关于使用 Python 提取 tar 文件的最快方法的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!