multiprocessing AttributeError模块对象没有属性'__path__' [英] multiprocessing AttributeError module object has no attribute '__path__'
问题描述
我有一个长脚本,最后需要对一个庞大列表中的所有项目运行一个函数,这需要很长时间,例如:
i have a long script which at the end needs to run a function to all items of huge list which takes a long time,consider for example:
input_a= [1,2,3,4] # a lengthy computation on some data
print('test.1') # for testing how the script runs
input_b= [5,6,7,8] # some other computation
print('test.2')
def input_analyzer(item_a): # analyzing item_a using input_a and input_b
return(item_a * input_a[0]*input_b[2])
from multiprocessing import Pool
def analyzer_final(input_list):
pool=Pool(7)
result=pool.map(input_analyzer, input_list)
return(result)
my_list= [10,20,30,40,1,2,2,3,4,5,6,7,8,9,90,1,2,3] # a huge list of inputs
if __name__=='__main__':
result_final=analyzer_final(my_list)
print(result_final)
return(result)
此代码的输出随运行而变化,但所有运行的共同点是整个脚本的多次运行,似乎通过将7分配为Pool,整个脚本将运行约8次!
the output of this codes, varies run to run but what all the runs has in common is several running of whole script, it seems that by assigning 7 as Pool, the whole script will run about 8 times!
我不确定我是否能很好地理解多处理的概念,但是我认为它应该做的只是使用多个CPU运行函数"input_analyzer",而不是多次运行整个脚本.如果是我的真实代码,它的时间太长了,它给了我一个奇怪的错误:
im not sure if i got the concept of multiprocessing well, but i thought what it should do is just running the function 'input_analyzer' using several CPUs and not running the whole script several times. in case of my real code, its so long and it gives me a strange error :
我不使用多重处理就可以很好地运行此代码,我不知道自己在这里做错什么,尤其是出现错误"AttributeError模块对象没有属性' path '"时,我非常感谢.
without using multiprocessing i run this code just fine, i dont know what im doing wrong here especially with error "AttributeError module object has no attribute 'path'"i appreciate any help.
推荐答案
from multiprocessing import Pool as ThreadPool
import requests
API_URL = 'http://example.com/api'
pool = ThreadPool(4) # Hint...
def foo(x):
params={'x': x}
r = requests.get(API_URL, params=params)
return r.json()
if __name__ == '__main__':
num_iter = [1,2,3,4,5]
out = pool.map(foo, num_iter)
print(out)
提示的答案:这就是为什么引发异常的原因.池定义为外部 if __name__ == '__main__'
Hint's Answer: This is why the exception is raised. Pool definition is outside if __name__ == '__main__'
已修复...
from multiprocessing import Pool as ThreadPool
import requests
API_URL = 'http://example.com/api'
def foo(x):
params={'x': x}
r = requests.get(API_URL, params=params)
return r.json()
if __name__ == '__main__':
pool = ThreadPool(4) # Hint...
num_iter = [1,2,3,4,5]
out = pool.map(foo, num_iter)
print(out)
The python docs touch on this scenario as well: https://docs.python.org/2/library/multiprocessing.html#using-a-pool-of-workers
当我完全使用multiprocessing.dummy时,我并不认为这是个问题.
I did not find this to be an issue when using multiprocessing.dummy at all.
这篇关于multiprocessing AttributeError模块对象没有属性'__path__'的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!