Python多处理-为什么使用functools.partial慢于默认参数? [英] Python multiprocessing - Why is using functools.partial slower than default arguments?

问题描述

考虑以下功能:

def f(x, dummy=list(range(10000000))):
    return x

如果使用multiprocessing.Pool.imap，则会得到以下计时:

If I use multiprocessing.Pool.imap, I get the following timings:

import time
import os
from multiprocessing import Pool

def f(x, dummy=list(range(10000000))):
    return x

start = time.time()
pool = Pool(2)
for x in pool.imap(f, range(10)):
    print("parent process, x=%s, elapsed=%s" % (x, int(time.time() - start)))

parent process, x=0, elapsed=0
parent process, x=1, elapsed=0
parent process, x=2, elapsed=0
parent process, x=3, elapsed=0
parent process, x=4, elapsed=0
parent process, x=5, elapsed=0
parent process, x=6, elapsed=0
parent process, x=7, elapsed=0
parent process, x=8, elapsed=0
parent process, x=9, elapsed=0

现在，如果我使用functools.partial而不是使用默认值:

Now if I use functools.partial instead of using a default value:

import time
import os
from multiprocessing import Pool
from functools import partial

def f(x, dummy):
    return x

start = time.time()
g = partial(f, dummy=list(range(10000000)))
pool = Pool(2)
for x in pool.imap(g, range(10)):
    print("parent process, x=%s, elapsed=%s" % (x, int(time.time() - start)))

parent process, x=0, elapsed=1
parent process, x=1, elapsed=2
parent process, x=2, elapsed=5
parent process, x=3, elapsed=7
parent process, x=4, elapsed=8
parent process, x=5, elapsed=9
parent process, x=6, elapsed=10
parent process, x=7, elapsed=10
parent process, x=8, elapsed=11
parent process, x=9, elapsed=11

为什么使用functools.partial的版本要慢得多?

Why is the version using functools.partial so much slower?

推荐答案

使用multiprocessing要求向工作进程发送有关要运行的函数的信息，而不仅仅是要传递的参数.该信息是通过酸洗在主流程中传输的，然后将其发送到工作进程，然后在那里解开进程.

Using multiprocessing requires sending the worker processes information about the function to run, not just the arguments to pass. That information is transferred by pickling that information in the main process, sending it to the worker process, and unpickling it there.

这导致了主要问题:

使用默认参数选择函数很便宜；它只会腌制函数的名称(加上使Python知道它是一个函数的信息)；工作进程仅查找名称的本地副本.他们已经有一个命名函数f可以找到，因此传递它几乎不需要花费任何费用.

Pickling a function with default arguments is cheap; it only pickles the name of the function (plus the info to let Python know it's a function); the worker processes just look up the local copy of the name. They already have a named function f to find, so it costs almost nothing to pass it.

但是拾取partial函数的过程涉及对基础函数(廉价)和 all 的默认参数进行酸洗(当默认参数为10M长时， expensive list).因此，每次在partial情况下分派任务时，都会对绑定的参数进行腌制，然后将其发送给工作进程，工作进程取消处理，然后最终完成实际"工作.在我的机器上，该泡菜的大小约为50 MB，这是一笔巨大的开销.在我的计算机上进行快速时序测试时，对1000万长的list的0进行酸洗和解酸大约需要620毫秒(而忽略了实际传输50 MB数据的开销).

But pickling a partial function involves pickling the underlying function (cheap) and all the default arguments (expensive when the default argument is a 10M long list). So every time a task is dispatched in the partial case, it's pickling the bound argument, sending it to the worker process, the worker process unpickles, then finally does the "real" work. On my machine, that pickle is roughly 50 MB in size, which is a huge amount of overhead; in quick timing tests on my machine, pickling and unpickling a 10 million long list of 0 takes about 620 ms (and that's ignoring the overhead of actually transferring the 50 MB of data).

partial必须以这种方式腌制，因为他们不知道自己的名字.腌制f之类的函数时，f(为def -ed)知道其限定名称(在交互式解释器中或在程序的主模块中为__main__.f)，因此远程端可以通过等效于from __main__ import f在本地重新创建它.但是partial不知道它的名字.当然，您已将其分配给g，但是pickle或partial本身都不知道它可以使用限定名称__main__.g来使用；它可以命名为foo.fred或一百万个其他名称.因此，它必须pickle必要的信息才能完全从头开始重新创建它.每次调用也要pickle(不是每个工作人员一次)，因为它不知道工作项之间的父级中可调用对象没有更改，并且它一直在尝试确保其发送最新状态.

partials have to pickle this way, because they don't know their own names; when pickling a function like f, f (being def-ed) knows its qualified name (in an interactive interpreter or from the main module of a program, it's __main__.f), so the remote side can just recreate it locally by doing the equivalent of from __main__ import f. But the partial doesn't know its name; sure, you assigned it to g, but neither pickle nor the partial itself know it available with the qualified name __main__.g; it could be named foo.fred or a million other things. So it has to pickle the info necessary to recreate it entirely from scratch. It's also pickle-ing for each call (not just once per worker) because it doesn't know that the callable isn't changing in the parent between work items, and it's always trying to ensure it sends up to date state.

您还有其他问题(仅在partial情况下安排list的创建以及调用partial包装函数与直接调用该函数的较小开销)，但是相对于每次呼叫开销的酸洗和取消partial的添加(list的初始创建增加了一次开销，其费用几乎是每个 酸洗/解开周期成本的一半；开销通过partial调用的时间不到一微秒).

You have other issues (timing creation of the list only in the partial case and the minor overhead of calling a partial wrapped function vs. calling the function directly), but those are chump change relative to the per-call overhead pickling and unpickling the partial is adding (the initial creation of the list is adding one-time overhead of a little under half what each pickle/unpickle cycle costs; the overhead to call through the partial is less than a microsecond).

这篇关于Python多处理-为什么使用functools.partial慢于默认参数?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！