如何在python中并行遍历任意数量的文件? [英] How to iterate over arbitrary number of files in parallel in python?

问题描述

我在名为paths

我希望能够遍历每个文件的第一行，对这n个元组的数据进行处理，然后再在每个文件的第二行进行移动. path中的文件对象数是任意的.

I'd like to be able to go through and read the first line of each file, do something with this n-tuple of data, then move on the second line of each file. The number of file objects in path is arbitrary.

这可能吗?

推荐答案

这取决于它的实际任意性".只要数量小于操作系统的限制，那么itertools.izip应该就可以正常工作了(或适当地itertools.izip_longest).

This depends on how "arbitrary" it actually is. As long as the number is less than the limit of your OS, then itertools.izip should work just fine (or itertools.izip_longest as appropriate).

files = [open(f) for f in filenames]
for lines in itertools.izip(*files):
    # do something

for f in files:
    f.close()

如果您拥有的文件数量超过操作系统允许打开的文件数量，那么您就不走运了(至少就简单的解决方案而言).

If you can have more files than your OS will allow you to open, then you're out of luck (at least as far as an easy solution is concerned).

这篇关于如何在python中并行遍历任意数量的文件?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！