在python中将stdout设置为非阻塞 [英] Setting stdout to non-blocking in python
问题描述
事先警告:出于好奇，我在这里四处闲逛.我没有特别的理由去做我在下面做的事情！
以下是在 MacOS 10.12.5
Python 2.7.13 上完成的我正在使用 python 进行黑客攻击，我认为如果我使 stdout 非阻塞，看看会发生什么会很有趣
fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)对fcntl 的调用肯定是成功的.然后我尝试写入大量数据(大于 OSX 上管道的最大缓冲区大小 - 65536 字节).我以各种方式做这件事并得到不同的结果，有时是例外，有时似乎是一个严重的失败.
案例 1
fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)尝试:sys.stdout.write("A" * 65537)除了作为 e 的例外:时间.睡眠(1)打印捕获:{}".format(e)# 安全睡眠，防止快速退出时间.睡眠(1)这总是会抛出异常Caught: [Errno 35] 资源暂时不可用.我觉得有道理.更高级别的文件对象包装器告诉我写调用失败.
案例 2
fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)尝试:sys.stdout.write("A" * 65537)除了作为 e 的例外:打印捕获:{}".format(e)# 安全睡眠，防止快速退出时间.睡眠(1)这有时会抛出异常 Caught: [Errno 35] 资源暂时不可用 或者有时没有捕获到异常，我看到以下输出:
在文件对象析构函数中关闭失败:sys.excepthook 丢失丢失的sys.stderr案例 3
fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)尝试:sys.stdout.write("A" * 65537)除了作为 e 的例外:打印捕获:{}".format(e)# 安全睡眠，防止快速退出时间.睡眠(1)打印睡觉"这有时会抛出异常Caught: [Errno 35] Resource暂时不可用，或者有时没有捕获到异常而我只看到Slept".似乎通过 printing "Slept" 我没有收到来自案例 2 的错误消息.
案例 4
fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)尝试:os.write(sys.stdout.fileno(),A"* 65537)除了作为 e 的例外:打印捕获:{}".format(e)# 安全睡眠，防止快速退出时间.睡眠(1)总是好的！
案例 5
fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)尝试:打印 os.write(sys.stdout.fileno(), "A" * 65537)除了作为 e 的例外:打印捕获:{}".format(e)# 安全睡眠，防止快速退出时间.睡眠(1)这有时没问题，有时会打印 close failed in file object destructor 错误消息.
我的问题是，为什么这会在 python 中失败?我在这里做了一些根本不好的事情 - 无论是使用 python 还是在系统级别?
似乎在写入已经失败时过早写入标准输出会导致错误消息.该错误似乎不是例外.不知道它来自哪里.
注意我可以用 C 编写等效的程序，它运行正常:
#include #include #include #include #include int main(int argc, const char * argv[]){const size_t NUM_CHARS = 65537;字符缓冲区[NUM_CHARS];//设置标准输出非阻塞fcntl(fileno(stdout), F_SETFL, O_NONBLOCK);//尝试写入大量数据memset(buf, 65, NUM_CHARS);size_t 写入 = fwrite(buf, 1, NUM_CHARS, stdout);//稍等片刻，让 stdout 有机会被读取使用睡眠(1000)；//这将被正确写入sprintf(buf, "\n我写了 %zd 个字节\n", 写了);fwrite(buf, 1, strlen(buf), stdout);返回0；} 这很有趣.到目前为止，我发现了一些东西:
案例 1
这是因为 sys.stdout.write 要么写入所有字符串，要么抛出异常，这在使用 O_NONBLOCK 时不是预期的行为.当对 write 的底层调用返回 EAGAIN(OS X 上的 Errno 35)时，应该使用剩余的数据再次尝试.应该使用 os.write 代替，并且应该检查返回值以确保所有数据都被写入.
此代码按预期工作:
fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)def stdout_write(s):写入 = 0而写 <镜片):尝试:写入 = 写入 + os.write(sys.stdout.fileno(), s[written:])除了作为 e 的 OSError:经过stdout_write("A" * 65537)案例 2
我怀疑此错误消息是由于https://bugs.python.org/issue11380:
在文件对象析构函数中关闭失败:sys.excepthook 丢失丢失的sys.stderr我不确定为什么有时会调用它.这可能是因为 except 语句中有一个 print 试图使用刚写入失败的相同 stdout.>
案例 3
这与案例 1 类似.此代码始终适用于我:
fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)def stdout_write(s):写入 = 0而写 <镜片):尝试:写入 = 写入 + os.write(sys.stdout.fileno(), s[written:])除了作为 e 的 OSError:经过stdout_write("A" * 65537)时间.睡眠(1)打印睡觉"案例 4
确保你检查了os.write的返回值，我怀疑没有成功写入完整的65537个字节.
案例 5
这类似于案例 2.
Prior warning: I'm hacking around here out of curiosity. I have no specific reason to do what I'm doing below!
Below is done on Python 2.7.13 on MacOS 10.12.5
I was hacking around with python and I thought it'd be interesting to see what happened if I made stdout nonblocking
fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)
The call to fcntl is definitely successful. I then try to write a large amount of data (bigger than the max buffer size of a pipe on OSX - which is 65536 bytes). I do it in a variety of ways and get different outcomes, sometimes an exception, sometimes what seems to be a hard fail.
Case 1
fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)
try:
sys.stdout.write("A" * 65537)
except Exception as e:
time.sleep(1)
print "Caught: {}".format(e)
# Safety sleep to prevent quick exit
time.sleep(1)
This always throws the exception Caught: [Errno 35] Resource temporarily unavailable. Makes sense I think. Higher level file object wrapper is telling me the write call failed.
Case 2
fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)
try:
sys.stdout.write("A" * 65537)
except Exception as e:
print "Caught: {}".format(e)
# Safety sleep to prevent quick exit
time.sleep(1)
This sometimes throws the exception Caught: [Errno 35] Resource temporarily unavailable or sometimes there is no exception caught and I see the following output:
close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr
Case 3
fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)
try:
sys.stdout.write("A" * 65537)
except Exception as e:
print "Caught: {}".format(e)
# Safety sleep to prevent quick exit
time.sleep(1)
print "Slept"
This sometimes throws the exception Caught: [Errno 35] Resource temporarily unavailable or sometimes there is no exception caught and I just see "Slept". It seems that by printing "Slept" I don't get the error message from Case 2.
Case 4
fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)
try:
os.write(sys.stdout.fileno(), "A" * 65537)
except Exception as e:
print "Caught: {}".format(e)
# Safety sleep to prevent quick exit
time.sleep(1)
Always okay!
Case 5
fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)
try:
print os.write(sys.stdout.fileno(), "A" * 65537)
except Exception as e:
print "Caught: {}".format(e)
# Safety sleep to prevent quick exit
time.sleep(1)
This is sometimes okay or sometimes prints the close failed in file object destructor error message.
My question is, why does this fail like this in python? Am I doing something fundamentally bad here - either with python or at the system level?
It seems like somehow that writing too soon to stdout when the write already failed causes the error message. The error doesn't appear to be an exception. No idea where it's coming from.
N.B. I can write the equivalent program in C and it works okay:
#include <stdio.h>
#include <stdlib.h>
#include <memory.h>
#include <sys/fcntl.h>
#include <unistd.h>
int main(int argc, const char * argv[])
{
const size_t NUM_CHARS = 65537;
char buf[NUM_CHARS];
// Set stdout non-blocking
fcntl(fileno(stdout), F_SETFL, O_NONBLOCK);
// Try to write a large amount of data
memset(buf, 65, NUM_CHARS);
size_t written = fwrite(buf, 1, NUM_CHARS, stdout);
// Wait briefly to give stdout a chance to be read from
usleep(1000);
// This will be written correctly
sprintf(buf, "\nI wrote %zd bytes\n", written);
fwrite(buf, 1, strlen(buf), stdout);
return 0;
}
This is interesting. There are a few things that I've found so far:
Case 1
This is because sys.stdout.write will either write all of the string or throw an exception, which isn't the desired behavior when using O_NONBLOCK. When the underlying call to write returns EAGAIN (Errno 35 on OS X), it should be tried again with the data that is remaining. os.write should be used instead, and the return value should be checked to make sure all the data is written.
This code works as expected:
fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)
def stdout_write(s):
written = 0
while written < len(s):
try:
written = written + os.write(sys.stdout.fileno(), s[written:])
except OSError as e:
pass
stdout_write("A" * 65537)
Case 2
I suspect that this error message is due to https://bugs.python.org/issue11380:
close failed in file object destructor:
sys.excepthook is missing
lost sys.stderr
I'm not sure why it's sometimes being called. It may be because there is a print in the except statement which is trying to use the same stdout that a write just failed on.
Case 3
This is similar to Case 1. This code always works for me:
fcntl.fcntl(sys.stdout.fileno(), fcntl.F_SETFL, os.O_NONBLOCK)
def stdout_write(s):
written = 0
while written < len(s):
try:
written = written + os.write(sys.stdout.fileno(), s[written:])
except OSError as e:
pass
stdout_write("A" * 65537)
time.sleep(1)
print "Slept"
Case 4
Make sure you check the return value of os.write, I suspect that the full 65537 bytes are not being successfully written.
Case 5
This is similar to Case 2.
这篇关于在python中将stdout设置为非阻塞的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
