如何处理__main__.py中的python3多处理 [英] How to deal with python3 multiprocessing in __main__.py

问题描述

我提出的问题，我不明白问题的真正原因(这似乎与我在其中一个子进程中使用烧瓶有关).

Il posed question, I did not understand the ture cause of the issue (it seems to have been related to my usage of flask in one of the subprocesses).

请忽略此(由于赏金无法删除)

PLEASE IGNORE THIS (can't delete due to bounty)

本质上，当将 python 库作为模块运行时，我必须启动一些进程和/或池.

Essentially, I have to start some Processes and or a pool when running a python library as a module.

但是，由于 __name__ == '__main__' 在 __main__.py 中始终为真，这证明是一个问题(请参阅多处理文档:https://docs.python.org/3/library/multiprocessing.html)

However, since __name__ == '__main__' is always true in __main__.py this proves to be an issue (see multiprocessing docs: https://docs.python.org/3/library/multiprocessing.html)

我尝试了多种解决方案，从:pytgquabr.com:8182/58288945/using-multiprocessing-with-runpy 到基于文件的互斥体，只允许 ma​​in 的内容运行一次但是 multiprocessing 的行为仍然很奇怪(例如，进程在没有错误日志的情况下几乎一开始就死了).

I've attempted multiple solutions ranging from: pytgquabr.com:8182/58288945/using-multiprocessing-with-runpy to a file-based mutext to only allow the contents of main to run once but multiprocessing still behaves strangely (e.g. Processes die almost as soon as they start with no error logs).

知道处理此问题的正确"方式是什么吗?

Any idea of what the "proper" way of going about this is ?

推荐答案

保护 __main__ 模块仅在 __main__定义对象时才需要> 用于另一个进程.查找这个定义是导致子进程执行__main__的原因.

Guarding the __main__ module is only needed if an object defined inside __main__ is used in another process. Looking up this definition is what causes the execution of __main__ in the subprocess.

使用 __main__.py 时，将所有与 multiprocessing 一起使用的定义限制到其他模块.__main__.py 应该只导入和使用这些.

When using a __main__.py, restrict all definitions used with multiprocessing to other modules. __main__.py should only import and use these.

# my_package/some_module.py
def module_print(*args, **kwargs):
    """Function defined in some module - fine for use inside multiprocess"""
    print(*args, **kwargs)

# my_package/__main__.py
import multiprocessing                 # imports are allowed
from .some_module import module_print

def do_multiprocess():
    """Function defined in __main__ module - fine for use wrapping multiprocess"""
    with multiprocessing.Pool(processes=12) as pool:
        pool.map(module_print, range(20))  # multiprocessing external function is allowed

do_multiprocess()  # directly calling __main__ function is allowed

这篇关于如何处理__main__.py中的python3多处理的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！