为什么打印到标准输出这么慢?可以加快速度吗? [英] Why is printing to stdout so slow? Can it be sped up?
问题描述
我一直对使用print语句简单地输出到终端所花费的时间感到惊讶/沮丧.在最近经历了一些痛苦缓慢的日志记录之后，我决定进行调查，并且很惊讶地发现几乎所有的时间都花在等待终端处理结果上.
I've always been amazed/frustrated with how long it takes to simply output to the terminal with a print statement. After some recent painfully slow logging I decided to look into it and was quite surprised to find that almost all the time spent is waiting for the terminal to process the results.
可以以某种方式加快对stdout的写入速度吗?
Can writing to stdout be sped up somehow?
我编写了一个脚本(此问题底部的"print_timer.py")来比较将100k行写入stdout，文件以及将stdout重定向到/dev/null时的时序.这是计时结果:
I wrote a script ('print_timer.py' at the bottom of this question) to compare timing when writing 100k lines to stdout, to file, and with stdout redirected to /dev/null. Here is the timing result:
$ python print_timer.py
this is a test
this is a test
<snipped 99997 lines>
this is a test
-----
timing summary (100k lines each)
-----
print :11.950 s
write to file (+ fsync) : 0.122 s
print with stdout = /dev/null : 0.050 s
哇.为了确保python在幕后不做任何事情，例如认识到我将stdout重新分配给/dev/null之类的东西，我在脚本外部进行了重定向...
Wow. To make sure python isn't doing something behind the scenes like recognizing that I reassigned stdout to /dev/null or something, I did the redirection outside the script...
$ python print_timer.py > /dev/null
-----
timing summary (100k lines each)
-----
print : 0.053 s
write to file (+fsync) : 0.108 s
print with stdout = /dev/null : 0.045 s
所以这不是python的把戏，它只是终端.我一直都知道将输出转储到/dev/null会加快速度，但是从来没有想到它是如此重要！
So it isn't a python trick, it is just the terminal. I always knew dumping output to /dev/null sped things up, but never figured it was that significant!
我很惊讶tty这么慢.
It amazes me how slow the tty is. How can it be that writing to physical disk is WAY faster than writing to the "screen" (presumably an all-RAM op), and is effectively as fast as simply dumping to the garbage with /dev/null?
此链接讨论了终端如何阻止I/O，因此它可以解析[输入]，更新其帧缓冲区，与X服务器通信以滚动窗口等等". ...但是我没有完全理解它.可能要花这么长时间?
This link talks about how the terminal will block I/O so it can "parse [the input], update its frame buffer, communicate with the X server in order to scroll the window and so on"... but I don't fully get it. What can be taking so long?
我希望没有出路(缺少更快的tty实现?)，但无论如何我都会问.
I expect there is no way out (short of a faster tty implementation?) but figure I'd ask anyway.
更新:阅读了一些评论后，我想知道我的屏幕尺寸实际上对打印时间有多大影响，这确实有一定意义.上面最慢的数字是我的Gnome终端被炸毁为1920x1200.如果将其减小得很小，我会得到...
UPDATE: after reading some comments I wondered how much impact my screen size actually has on the print time, and it does have some significance. The really slow numbers above are with my Gnome terminal blown up to 1920x1200. If I reduce it very small I get...
-----
timing summary (100k lines each)
-----
print : 2.920 s
write to file (+fsync) : 0.121 s
print with stdout = /dev/null : 0.048 s
那肯定更好(〜4倍)，但不会改变我的问题.它只是对我的问题添加 ，因为我不明白为什么终端屏幕渲染会减慢应用程序向stdout的写入速度.为什么我的程序需要等待屏幕渲染继续?
That is certainly better (~4x), but doesn't change my question. It only adds to my question as I don't understand why the terminal screen rendering should slow down an application writing to stdout. Why does my program need to wait for screen rendering to continue?
不是所有创建的终端/tty应用程序都相等?我还没有实验.在我看来，终端确实应该能够缓冲所有传入的数据，不可见地进行解析/渲染，并且仅以合理的帧速率渲染在当前屏幕配置中可见的最新块.因此，如果我可以在约0.1秒内将+ fsync写入磁盘，则终端应该能够以该顺序完成相同的操作(在执行此操作时可能需要进行一些屏幕更新).
Are all terminal/tty apps not created equal? I have yet to experiment. It really seems to me like a terminal should be able to buffer all incoming data, parse/render it invisibly, and only render the most recent chunk that is visible in the current screen configuration at a sensible frame rate. So if I can write+fsync to disk in ~0.1 seconds, a terminal should be able to complete the same operation in something of that order (with maybe a few screen updates while it did it).
我仍然希望可以从应用程序端更改tty设置，以使程序员更好地实现此行为.如果严格来说这是终端应用程序问题，那么这可能甚至不属于StackOverflow吗?
I'm still kind of hoping there is a tty setting that can be changed from the application side to make this behaviour better for programmer. If this is strictly a terminal application issue, then this maybe doesn't even belong on StackOverflow?
我想念什么?
这是用于生成计时的python程序:
Here is the python program used to generate the timing:
import time, sys, tty
import os
lineCount = 100000
line = "this is a test"
summary = ""
cmd = "print"
startTime_s = time.time()
for x in range(lineCount):
print line
t = time.time() - startTime_s
summary += "%-30s:%6.3f s\n" % (cmd, t)
#Add a newline to match line outputs above...
line += "\n"
cmd = "write to file (+fsync)"
fp = file("out.txt", "w")
startTime_s = time.time()
for x in range(lineCount):
fp.write(line)
os.fsync(fp.fileno())
t = time.time() - startTime_s
summary += "%-30s:%6.3f s\n" % (cmd, t)
cmd = "print with stdout = /dev/null"
sys.stdout = file(os.devnull, "w")
startTime_s = time.time()
for x in range(lineCount):
fp.write(line)
t = time.time() - startTime_s
summary += "%-30s:%6.3f s\n" % (cmd, t)
print >> sys.stderr, "-----"
print >> sys.stderr, "timing summary (100k lines each)"
print >> sys.stderr, "-----"
print >> sys.stderr, summary
推荐答案
感谢所有评论！我最终在您的帮助下自行回答.不过，回答您自己的问题感觉很肮脏.
Thanks for all the comments! I've ended up answering it myself with your help. It feels dirty answering your own question, though.
问题1:为什么打印到标准输出的速度慢?
答案:打印到标准输出本质上不是很慢.这是您使用的终端，速度很慢.它与应用程序端的I/O缓冲(例如:python文件缓冲)的关系几乎为零.见下文.
Answer: Printing to stdout is not inherently slow. It is the terminal you work with that is slow. And it has pretty much zero to do with I/O buffering on the application side (eg: python file buffering). See below.
问题2:可以加快速度吗?
答案:是的，可以，但似乎不是从程序方面(将'printing'到stdout的那一侧).为了加快速度，请使用更快的其他终端模拟器.
Answer: Yes it can, but seemingly not from the program side (the side doing the 'printing' to stdout). To speed it up, use a faster different terminal emulator.
说明...
我尝试了一个自描述为轻量级"的终端程序，称为"c2>"，并获得了明显"的更好的结果.以下是在wterm中以1920x1200在同一系统上运行的测试脚本的输出(位于问题的底部)，该系统使用gnome-terminal在基本打印选项上花费了12s:
I tried a self-described 'lightweight' terminal program called wterm and got significantly better results. Below is the output of my test script (at the bottom of the question) when running in wterm at 1920x1200 in on the same system where the basic print option took 12s using gnome-terminal:
-----
timing summary (100k lines each)
-----
print : 0.261 s
write to file (+fsync) : 0.110 s
print with stdout = /dev/null : 0.050 s
0.26s比12s好得多！我不知道wterm是否更聪明地按照我的建议进行渲染(以合理的帧速率渲染可见的"尾巴)，或者是否仅比少" c5>.就我的问题而言，我已经找到了答案. gnome-terminal慢.
0.26s is MUCH better than 12s! I don't know whether wterm is more intelligent about how it renders to screen along the lines of how I was suggesting (render the 'visible' tail at a reasonable frame rate), or whether it just "does less" than gnome-terminal. For the purposes of my question I've got the answer, though. gnome-terminal is slow.
所以-如果您的脚本运行时间长，感觉很慢，并且会向stdout喷出大量文本，请尝试其他终端，看看效果是否更好！
So - If you have a long running script that you feel is slow and it spews massive amounts of text to stdout... try a different terminal and see if it is any better!
请注意，我几乎是从ubuntu/debian存储库中随机提取wterm的. 此链接可能是同一终端，但是我不确定.我没有测试任何其他终端模拟器.
Note that I pretty much randomly pulled wterm from the ubuntu/debian repositories. This link might be the same terminal, but I'm not sure. I did not test any other terminal emulators.
更新:因为必须要抓痒，所以我用相同的脚本和全屏(1920x1200)测试了一大堆其他终端模拟器.我的手动收集的统计信息在这里:
Update: Because I had to scratch the itch, I tested a whole pile of other terminal emulators with the same script and full screen (1920x1200). My manually collected stats are here:
wterm 0.3s
aterm 0.3s
rxvt 0.3s
mrxvt 0.4s
konsole 0.6s
yakuake 0.7s
lxterminal 7s
xterm 9s
gnome-terminal 12s
xfce4-terminal 12s
vala-terminal 18s
xvt 48s
记录的时间是手动收集的，但它们是相当一致的.我记录了最好的(ish)值. YMMV，很明显.
The recorded times are manually collected, but they were pretty consistent. I recorded the best(ish) value. YMMV, obviously.
作为奖励，这是对其中可用的各种终端模拟器中的一些的有趣的浏览！我很惊讶我的第一个替代"测试竟然是同类中最好的.
As a bonus, it was an interesting tour of some of the various terminal emulators available out there! I'm amazed my first 'alternate' test turned out to be the best of the bunch.
这篇关于为什么打印到标准输出这么慢?可以加快速度吗?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
