使用multiprocessing.Pool打开的文件太多 [英] Too many files open with multiprocessing.Pool

问题描述

我有一个运行multiprocessing.Pool的python脚本来单独处理很多文件.我通常的cpu限制为8.我的问题是运行一段时间后，我总是收到"IOError:[Errno 24]打开的文件太多".每个子进程都会打开一些文件，这些文件只能通过file.open()进行读取.然后，将这些文件处理程序传递给多个函数以检索数据.在每个子进程的末尾，这些文件都使用file.close()关闭.我也尝试了with语句，但没有解决问题.有谁知道这是怎么回事.我四处搜寻，但找不到任何答案.我正在关闭文件，并且函数正常返回，因此保持文件处理程序不变的原因.

I have a python script that runs multiprocessing.Pool to process a lot of files separately. I usually have a cpu limit of 8. My problem is after running a while I always get "IOError: [Errno 24] Too many open files". Each child process opens a few files for reading only with file.open(). These file handlers are than passed to multiple functions to retrieve data. At the end of the each child process these files are closed with file.close(). I tried the with statement as well but did not fix the issue. Does any one have any idea whats wrong. I googled around but failed to find any answers. I am closing the files and the functions are returning properly so what keep the file handlers around.

我的设置是带有python 2.6的Mac 10.5

My settings are Mac 10.5 with python 2.6

谢谢

Ogan

    from custom import func1, func2
    # func1 and func2 only seek, read and return values form the file
    # however, they do not close the file
    import multiprocessing
    def Worker(*args):
        f1 = open("db1.txt")
        f2 = open("db2.txt")
        for each in args[1]:
            # do many stuff
            X = func1(f1)
            Y = func2(f2)

        f1.close()
        f2.close()
        return

    Data = {1:[2], 2:[3]}  
    JobP= multiprocessing.Pool(8) 
    jobP.map_async(Worker, Data.items()) 
    jobP.close()
    jobP.join()

推荐答案

要更改优胜美地(OS X 10.10)中打开文件的数量限制:

To change the number open files limit in Yosemite (OS X 10.10):

sudo launchctl limit maxfiles [number-of-files] unlimited

这篇关于使用multiprocessing.Pool打开的文件太多的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！