"WindowsError:[错误206]文件名或扩展名太长".用子进程运行程序很多次之后 [英] "WindowsError: [Error 206] The filename or extension is too long" after running a program very many times with subprocess

问题描述

我的python程序在Windows HPC 2008环境中准备输入，运行外部FORTRAN代码并处理输出.除非代码在1042-1045次之间执行外部程序(通常问题会更早收敛)，否则它会很好地工作.在这些情况下，我会得到一个例外:

My python program prepares inputs, runs an external FORTRAN code, and processes the outputs in a Windows HPC 2008 environment. It works great, unless the code executes the external program between 1042-1045 times (Usually the problem converges earlier). In these situations, I get an exception:

WindowsError:[错误206]文件名或扩展名太长

WindowsError: [Error 206] The filename or extension is too long

但是，文件名的路径不是 随时间增长.它只是清理目录并再次运行.

However, the path to the filename is not growing with time. It's just cleaning the directory and running again.

这是代码:

inpF = open(inName)
outF = open(localOutName,'w')
p = subprocess.Popen(pathToExe,shell=False,stdin=inpF,stdout=outF,cwd=runPath)
stdout, stderr = p.communicate()
outF.close()
inpF.close()

pathToExe是指向UNC位置的常量字符串(例如\\ server \ shared \ program.exe)，stdin是本地驱动器上只读模式下的打开文件，stdout是打开时处于写入模式下的打开文件本地驱动器，而cwd是C:\驱动器上的本地路径.根据

pathToExe is a constant string pointing to a UNC location (e.g. \\server\shared\program.exe), stdin is an open file in read-only mode on a local drive, stdout is an open file in write mode on a local drive, and cwd is a local path on the C:\ drive. I have confirmed that none of the arguments to subprocess are longer than 80 characters, even though the limit is supposed to be 32,768, according to this somewhat related post.

我做错了什么?不知何故，不断积累的东西只会在我跑上一千遍时才成为问题.

What am I doing wrong? Somehow something is accumulating that only becomes a problem when I run over a thousand times.

更新:

为了测试打开太多文件"的假设，我举了一个非常小的示例，该示例在使用其他可执行文件时可以非常快速地运行.这里的主要区别是stdin和stdout在这里只是空文件，而在以前的情况下，它们都是大文件.在这种情况下，代码可以很好地运行2000次，而较早的代码在〜1042处失败.因此，不仅仅是有那么多文件.也许打开了太多大文件?

To test the "too many files open" hypothesis, I made a very small example that runs very quickly with a different executable. The main difference here is that the stdin and stdout are just empty files here, whereas in the previous case, they're both large files. In this case, the code runs just fine for 2000 runs, whereas the earlier fails at ~1042. So it's not just that there are that many files. Maybe there are too many large files open?

import subprocess
for i in range(nRuns):
    if not (i % (nRuns/10.0)):
        print('{0:.2}% complete'.format(i/float(nRuns)*100))
    inpF=open('in.txt')
    outF=open('out.txt','w')
    p = subprocess.Popen('isotxsmerge.exe',shell=False,stdin=inpF,
                                 stdout=outF,cwd='.')
    stdout, stderr = p.communicate()
    outF.close()
    inpF.close()

推荐答案

hmmm....实际上，我认为错误消息文本是红色鲱鱼.我不确定，但是在我看来，正在发生的事情是您用尽了文件句柄.从各种来源看来，规范的文件句柄限制大约为2048个文件...这奇怪地是2 x 1042个子进程的附近.我不知道Windows python解释器的内部结构，但是我的猜测是，即使关闭文件，句柄也没有足够快地被垃圾回收.再说一次...这只是一个猜测...但是也许这是另一种思路，可能会使您产生更确定和更有成效的东西.

hmmm....actually, I think the error message text is a red herring. I don't know for sure, but it seems likely to me that what's happening is that you're running out of file handles. From various sources, it seems that the canonical file handle limit is around 2048 files...which is curiously in the neighborhood of 2 x your 1042 subprocesses. I don't know the internals of the windows python interpreter, but my guess is that the handles aren't being garbage collected fast enough, even though you're closing the files. Again...this is just a guess...but perhaps it's another line of thinking that might lead you to something more conclusive and productive.

同时，作为一种变通办法，您可以使用旧的备用方法，即使用一个调控器流程来生成一个流程，然后生成一个子流程.中间子进程在死亡之前具有确定的生命周期(例如，其产生的子进程不超过1000个).当中间子进程到期时，调控器进程将启动一个新的子进程.这是一种骇客……而且那是一种笨拙的行为……但确实有效. (IIRC，Apache Web服务器以前对子进程可以处理多少个请求有某种自毁限制.)

In the mean time, as a work around, you can use the old stand-by approach by having a governor process that spawns a process, that in turn spawns the subprocesses. The intermediate subprocess has a determined lifespan (say...no more than 1000 subprocesses that it spawns) before it dies. When the intermediate subprocess expires, the governor process starts a new one. This is a hack...and a clumsy one at that...but it does work. (IIRC, the apache web server used to have some kind of self-destruct limit on how many requests a sub-process could handle.)

无论如何...祝你好运和开心的编程.

Anyway...best of luck and happy coding.

这篇关于"WindowsError:[错误206]文件名或扩展名太长".用子进程运行程序很多次之后的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！