python 2.7 IOError: [Errno 24] 打开的文件太多: [英] python 2.7 IOError: [Errno 24] Too many open files:

问题描述

在工作中，我很不幸在 Python 中修复了由其他人完成的写得不好的 url 验证器脚本.这是一段非常混乱的代码，在尝试修复其中一个错误时，我发现了一些我不理解的行为.

At work I had the bad luck to have fix a badly written url validator script in python done by someone else. It's a really messy code, and trying to fix one of the bugs, I found some behavior that I don't understand.

脚本必须处理一个包含大约 10,000 个 url 的文件，它必须检查每个 url 以查看它是否有效，不仅在它的结构中还要检查它是否存在(为此使用 pycurl).在代码的一部分上，这是完成的:

The script has to process a file with around 10 thousand url's in it, it has to check each url to see if it's valid, not only in it's structure but also check if it exists (using pycurl for this). On one part of the code, this is done:

for li in lineas:
    liNew = "http://" + li
    parsedUrl = urlparse.urlparse(liNew)    

在这种情况下，错误是在该行的开头添加了http://"，就像之前在脚本中所做的那样.所以我把代码改成这样:

On this case the bug was the addition of "http://" at the beginning of the line as that was being done before on the script. So I changed the code to this:

for li in lineas:
    liNew = li
    parsedUrl = urlparse.urlparse(liNew)    

现在，使用相同的输入文件，脚本失败并显示错误:

Now, with the same input file the script fails with the error:

IOError: [Errno 24] Too many open files:/path/to/file/being/written/to.txt

使用 liNew = "http://" + li，文件描述符不会超过 1024 的默​​认限制，但将该行更改为 liNew = li会让他们超过 8000，为什么??

With liNew = "http://" + li, file descriptors don't go over the default limit of 1024, but changing that line to liNew = li will make them go over 8000, why ??

推荐答案

使用 liNew = "http://" + li，文件描述符不会超过 1024 的默​​认限制，但是将该行更改为 liNew = li 会使它们超过 8000，为什么??

With liNew = "http://" + li, file descriptors don't go over the default limit of 1024, but changing that line to liNew = li will make them go over 8000, why ??

• 之前:网址损坏 - 没有下载任何内容(没有打开文件)
• 之后:正确的网址 - 网址被保存到文件中(有 10K 网址)

同时下载数百个网址(带宽、磁盘)可能没有意义.确保下载后所有文件(sockets、磁盘文件)都被妥善处理(及时调用close()方法).

It probably doesn't make sense to download more that a few hundreds urls concurrently (bandwidth, disk). Make sure that all files (sockets, disk files) are properly disposed after the download (close() method is called in time).

默认限制 (1024) 很低，但除非您了解代码的作用，否则不要增加它.

Default limit (1024) is low but don't increase it unless you understand what the code does.

这篇关于python 2.7 IOError: [Errno 24] 打开的文件太多:的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！