多重处理会导致Python崩溃,并在调用fork()时在另一个线程中发出错误提示 [英] Multiprocessing causes Python to crash and gives an error may have been in progress in another thread when fork() was called
问题描述
我是Python的新手,并尝试为我的for循环实现一个Multiprocessing模块.
I am relatively new to Python and trying to implement a Multiprocessing module for my for loop.
我有一个存储在img_urls中的图像URL数组,我需要下载并应用一些Google视觉.
I have an array of Image url's stored in img_urls which I need to download and apply some Google vision.
if __name__ == '__main__':
img_urls = [ALL_MY_Image_URLS]
runAll(img_urls)
print("--- %s seconds ---" % (time.time() - start_time))
这是我的runAll()方法
This is my runAll() method
def runAll(img_urls):
num_cores = multiprocessing.cpu_count()
print("Image URLS {}",len(img_urls))
if len(img_urls) > 2:
numberOfImages = 0
else:
numberOfImages = 1
start_timeProcess = time.time()
pool = multiprocessing.Pool()
pool.map(annotate,img_urls)
end_timeProcess = time.time()
print('\n Time to complete ', end_timeProcess-start_timeProcess)
print(full_matching_pages)
def annotate(img_path):
file = requests.get(img_path).content
print("file is",file)
"""Returns web annotations given the path to an image."""
print('Process Working under ',os.getpid())
image = types.Image(content=file)
web_detection = vision_client.web_detection(image=image).web_detection
report(web_detection)
当我运行它并导致python崩溃时,我将其作为警告
I am getting this as the warning when I run it and python crashes
objc[67570]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[67570]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
objc[67567]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[67567]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
objc[67568]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[67568]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
objc[67569]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[67569]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
objc[67571]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[67571]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
objc[67572]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called.
objc[67572]: +[__NSPlaceholderDate initialize] may have been in progress in another thread when fork() was called. We cannot safely call it or ignore it in the fork() child process. Crashing instead. Set a breakpoint on objc_initializeAfterForkError to debug.
推荐答案
发生此错误是由于增加了安全性,以限制Mac OS High Sierra中的多线程.我知道这个答案有点晚了,但是我使用以下方法解决了这个问题:
This error occurs because of added security to restrict multithreading in Mac OS High Sierra. I know this answer is a bit late, but I solved the problem using the following method:
设置环境变量.bash_profile,以允许新的Mac OS High Sierra安全规则下的多线程应用程序或脚本.
Set an environment variable .bash_profile to allow multithreading applications or scripts under the new Mac OS High Sierra security rules.
打开终端:
$ nano .bash_profile
在文件末尾添加以下行:
Add the following line to the end of the file:
export OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
保存,退出,关闭终端,然后重新打开终端.检查是否已设置环境变量:
Save, exit, close terminal and re-open the terminal. Check to see that the environment variable is now set:
$ env
您将看到类似于以下内容的输出:
You will see output similar to:
TERM_PROGRAM=Apple_Terminal
SHELL=/bin/bash
TERM=xterm-256color
TMPDIR=/var/folders/pn/vasdlj3ojO#OOas4dasdffJq/T/
Apple_PubSub_Socket_Render=/private/tmp/com.apple.launchd.E7qLFJDSo/Render
TERM_PROGRAM_VERSION=404
TERM_SESSION_ID=NONE
OBJC_DISABLE_INITIALIZE_FORK_SAFETY=YES
您现在应该可以在多线程中运行python脚本了.
You should now be able to run your python script with multithreading.
这篇关于多重处理会导致Python崩溃,并在调用fork()时在另一个线程中发出错误提示的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!