与常规dict相比，Python manager.dict()非常慢 [英] Python manager.dict() is very slow compared to regular dict

问题描述

我有一个字典来存储对象:

I have a dict to store objects:

jobs = {}
job = Job()
jobs[job.name] = job

现在，我想将其转换为使用manager dict，因为我想使用多处理功能，并且需要在处理过程中共享此dict

now I want to convert it to use manager dict because I want to use multiprocessing and need to share this dict amonst processes

mgr = multiprocessing.Manager()
jobs = mgr.dict()
job = Job()
jobs[job.name] = job

仅通过转换为使用manager.dict()，事情就变得非常慢.

just by converting to use manager.dict() things got extremely slow.

例如，如果使用本机字典，则仅用0.65秒即可创建625个对象并将其存储到字典中.

For example, if using native dict, it only took .65 seconds to create 625 objects and store it into the dict.

同一任务现在需要126秒！

The very same task now takes 126 seconds!

任何优化方法都可以使manager.dict()与python {}保持一致?

Any optimization i can do to keep manager.dict() on par with python {}?

推荐答案

问题是由于某种原因，每个插入操作都非常慢(在我的计算机上慢117倍)，但是如果您使用常规命令更新manager.dict()，这将是一个快速的操作.

The problem is that each insert is quite slow for some reason (117x slower on my machine), but if you update your manager.dict() with a normal dict, it will be a single and fast operation.

jobs = {}
job = Job()
jobs[job.name] = job
# insert other jobs in the normal dictionary

mgr = multiprocessing.Manager()
mgr_jobs = mgr.dict()
mgr_jobs.update(jobs)

然后使用mgr_jobs变量.

另一种选择是使用被广泛采用的multiprocessing.Queue类.

Another option is to use the widely adopted multiprocessing.Queue class.

这篇关于与常规dict相比，Python manager.dict()非常慢的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！