“块大小" Python的multiprocessing.Pool.map中的参数 [英] "chunksize" parameter in Python's multiprocessing.Pool.map

问题描述

例如，如果我有一个带有2个处理器的池对象:

p=multiprocessing.Pool(2)

我想遍历目录中的文件列表并使用map函数

有人可以解释一下此函数的chunksize:

p.map(func, iterable[, chunksize])

例如，如果我将块大小设置为10，是否意味着每10个文件应使用一个处理器处理?

解决方案

查看 解决方案

Looking at the documentation for Pool.map it seems you're almost correct: the chunksize parameter will cause the iterable to be split into pieces of approximately that size, and each piece is submitted as a separate task.

So in your example, yes, map will take the first 10 (approximately), submit it as a task for a single processor... then the next 10 will be submitted as another task, and so on. Note that it doesn't mean that this will make the processors alternate every 10 files, it's quite possible that processor #1 ends up getting 1-10 AND 11-20, and processor #2 gets 21-30 and 31-40.

这篇关于“块大小" Python的multiprocessing.Pool.map中的参数的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！