Python中的C#Parallel.Foreach等效项 [英] C# Parallel.Foreach equivalent in Python

问题描述

我有96个txt文件需要处理.现在，我正在使用一个for循环并一次执行一次，此过程非常缓慢.生成的96个文件不需要合并.有没有办法让它们并行运行，在C#中为ala Parallel.foreach? 当前代码:

I have 96 txt files that have to be processed. Right now I am using a for loop and doing them one at a time, this process is very slow. The resulting 96 files, do not need to be merged. Is there a way to make them run in parallel, ala Parallel.foreach in C#? Current code:

for src_name in glob.glob(source_dir+'/*.txt'):
   outfile = open (...)
   with open(...) as infile:
      for line in infile:
         --PROCESS--
   for --condition--:
      outfile.write(...)
   infile.close()
   outfile.close()

希望此过程针对source_dir中的所有文件并行运行.

Want this process to run in parallel for all files in source_dir.

推荐答案

假定限制因素确实是处理过程而不是I/O，则可以使用

Assuming that the limiting factor is indeed the processing and not the I/O, you can use joblib to easily run your loop on multiple CPUs.

文档中的简单示例:

>>> from math import sqrt
>>> from joblib import Parallel, delayed
>>> Parallel(n_jobs=2)(delayed(sqrt)(i ** 2) for i in range(10))
[0.0, 1.0, 2.0, 3.0, 4.0, 5.0, 6.0, 7.0, 8.0, 9.0]

这篇关于Python中的C#Parallel.Foreach等效项的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！