Python的多重map_async [英] Python Multiprocessing map_async

问题描述

我想跳过从<返回的结果href=\"http://docs.python.org/2/library/multiprocessing.html#multiprocessing.pool.multiprocessing.Pool.map_async\"相对=nofollow> map_async 。他们渐渐在内存中，但我不需要他们。

I’d like to skip results that are returned from map_async. They are growing in memory but I don’t need them.

下面是一些code：

def processLine(line):
    #process something
    print "result"
pool = Pool(processes = 8)
for line in sys.stdin:
    lines.append(line)
    if len(lines) >= 100000:
        pool.map_async(processLine, lines, 2000)
pool.close()
pool.join()

当我必须处理文件，数亿行的，蟒蛇生长进程在内存中几个G的空间。我该如何解决？

When I have to process file with hundreds of millions of rows, the python process grows in memory to a few gigabytes. How can I resolve that?

感谢您的帮助：）

推荐答案

您code有一个错误：

Your code has a bug:

for line in sys.stdin:
    lines.append(line)
    if len(lines) >= 100000:
        pool.map_async(processLine, lines, 2000)

这是要等到行累积超过10万线。在此之后， pool.map_async 被称为100000+行的整个列表中的每增加行

This is going to wait until lines accumulates more than 100000 lines. After that, pool.map_async is being called on the entire list of 100000+ lines for each additional line.

目前还不清楚到底是什么你真的想这样做，但
如果你不想返回值，可以使用 pool.apply_async ，而不是 pool.map_async 。也许是这样的：

It is not clear exactly what you are really trying to do, but if you don't want the return value, use pool.apply_async, not pool.map_async. Maybe something like this:

import multiprocessing as mp

def processLine(line):
    #process something
    print "result"

if __name__ == '__main__':
    pool = mp.Pool(processes = 8)
    for line in sys.stdin:
        pool.apply_async(processLine, args = (line, ))
    pool.close()
    pool.join()

这篇关于Python的多重map_async的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！