Windows中的Python多处理池奇怪行为 [英] Python multiprocessing Pool strange behavior in Windows
问题描述
Python多处理池在Linux和Windows之间具有不同的行为.
Python multiprocessing Pool have different behavior between Linux and Windows.
在按工作人员数量运行方法映射时，在Linux中，该过程是在您指定为参数的特定功能范围内运行的， 但是在Windows中，每个工作程序都在父进程的范围内运行，并再次使用不应有的代码.
When running method map by number of workers, in Linux it's ran the process on the scope of the specific function you gave as parameter, But in Windows, every worker run at the scope of the parent process and use again code that it's not should be.
例如:(烧瓶仅用于使其类似于我的代码)
For example: (The flask it's only for making it similar to my code)
from multiprocessing import Pool, Event
from flask import Flask
print(">>> This code running for every each worker")
app = Flask(__name__)
terminating = None
def f(**kwargs):
print("f()")
x = kwargs.pop("x", 1)
print(x * x)
return x * x
def worker_warpper(arg):
func, kwargs = arg
return func(**kwargs)
def initializer(terminating_):
global terminating
terminating = terminating_
@app.route('/check', methods=['GET'])
def check():
with Pool(processes=3) as pool:
ls = [(f, {"x": 2}), (f, {"x": 5}), (f, {"x": 6})]
pool_map = pool.map(worker_warpper, ls)
return "Finished"
if __name__ == "__main__":
print("Listening...")
app.run(port=5151, host='0.0.0.0')
此代码段应在3个不同的进程中并行运行函数"f"(仅函数"f")3次.
This chunk of code should be run the function "f" (only function "f") 3 times at 3 different process in parallel.
但是它再次在顶部运行打印. (这并不完全是针对每个流程的，但是运行"f"的次数与要再次运行的顶部打印次数之间存在相关性.)
But it runs the print at the top again. (it's not exactly for every process again - but there is relation between the number of times to run "f" and the number of the print at the top to run again)
print(">>> This code running for every each worker")
仅在Windows中，仅在Linux中再次运行"f".
Only in Windows, in Linux only "f" running again.
输出:(Linux)
>>> This code running for new worker (not all of the workers)
Listening
...
* Running on http://0.0.0.0:5151/ (Press CTRL+C to quit)
f()
4
f()
25
f()
36
127.0.0.1 - -
[29/Jan/2017 11:46:26] "GET /check HTTP/1.1" 200 -
输出:(Windows)
Output: (Windows)
>>> This code running for new worker (not all of the workers)
Listening
...
* Running on http://0.0.0.0:5151/ (Press CTRL+C to quit)
>>> This code running for new worker (not all of the workers)
f()
4
f()
25
f()
36
127.0.0.1 - -
[29/Jan/2017 11:46:26] "GET /check HTTP/1.1" 200 -
为什么Linux和Windows之间有不同的行为? 那我该怎么办?
Why there is different behavior between linux and windows? And what I can do about it?
如果不清楚，请告诉我，我将尝试另一种方式.
If it's not clear tell me and i will try in a different way.
谢谢！
推荐答案
Windows和Linux之间的区别在于子进程的启动方式.在Linux上，子进程是使用fork()启动的:新进程以与父进程相同的状态启动:python代码已被解释，并获得了父进程的内存副本.
The difference between Windows and Linux is the way a child process is started. On Linux, child processes are started using fork(): The new process starts in the same state as the parent process: the python code is already interpreted and it gets a copy of the memory of the parent process.
在Windows上完全不同:进行了spawn处理:启动了新的python解释器，该解释器再次解析并执行python文件.这就是为什么您在顶部的print进程会再次执行的原因.
On Windows it's entirely different: Processes are spawned: a new python interpreter is started, which again parses the python file and executes it. That's why your print process at the top is executed again.
有关详细信息，请参见有关fork与vs.的文档产生.
For details see the docs about fork vs. spawn.
常见的陷阱是避免底部出现if __name__ == '__main__'.但是，由于您的代码中已经包含了此代码，因此您非常接近安全代码".
A common pitfall is to avoid the if __name__ == '__main__' at the bottom. But since you already have this in your code, you're pretty close to "safe code".
我该怎么办?
What can I do about it?
您可以使用 Threading 代替多处理.当启动新线程时，新线程使用与父线程相同的内存空间.缺点是由于python的全局解释器锁"，您只能使用一个CPU内核.
You can use Threading instead of Multiprocessing. When you start a new Thread, the new thread uses the same memory space as the parent thread. Downside is that you can only utilize one CPU core because of pythons "global interpreter lock".
有关详细信息，请参见此讨论
For details see this discussion
这篇关于Windows中的Python多处理池奇怪行为的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
