为 sys.stdin 设置较小的缓冲区大小? [英] Setting smaller buffer size for sys.stdin?

问题描述

我正在使用以下 bash 命令模式运行 memcached:

I'm running memcached with the following bash command pattern:

memcached -vv 2>&1 | tee memkeywatch2010098.log 2>&1 | ~/bin/memtracer.py | tee memkeywatchCounts20100908.log

尝试在整个平台范围内追踪无法匹配的密钥集.

to try and track down unmatched gets to sets for keys platform wide.

memtracer 脚本如下所示，可以正常工作，但有一个小问题.观察中间日志文件大小，直到 memkeywatchYMD.log，memtracer.py 才开始获取输入大小约为 15-18K.是否有更好的方法来读取 stdin 或将缓冲区大小减少到 1k 以下以加快响应时间?

The memtracer script is below and works as desired, with one minor issue. Watching the intermediate log file size, memtracer.py doesn't start getting input until memkeywatchYMD.log is about 15-18K in size. Is there a better way to read in stdin or perhaps a way to cut the buffer size down to under 1k for faster response times?

#!/usr/bin/python

import sys
from collections import defaultdict

if __name__ == "__main__":


    keys = defaultdict(int)
    GET = 1
    SET = 2
    CLIENT = 1
    SERVER = 2

    #if <
    for line in sys.stdin:
        key = None
        components = line.strip().split(" ")
        #newConn = components[0][1:3]
        direction = CLIENT if components[0].startswith("<") else SERVER

        #if lastConn != newConn:        
        #    lastConn = newConn

        if direction == CLIENT:            
            command = SET if components[1] == "set" else GET
            key = components[2]
            if command == SET:                
                keys[key] -= 1                                                                                    
        elif direction == SERVER:
            command = components[1]
            if command == "sending":
                key = components[3] 
                keys[key] += 1

        if key != None:
            print "%s:%s" % ( key, keys[key], )

推荐答案

您可以使用 python 的 -u 标志从 stdin/stdout 中完全删除缓冲:

You can completely remove buffering from stdin/stdout by using python's -u flag:

-u     : unbuffered binary stdout and stderr (also PYTHONUNBUFFERED=x)
         see man page for details on internal buffering relating to '-u'

和手册页澄清:

   -u     Force stdin, stdout and stderr to  be  totally  unbuffered.   On
          systems  where  it matters, also put stdin, stdout and stderr in
          binary mode.  Note that there is internal  buffering  in  xread-
          lines(),  readlines()  and  file-object  iterators ("for line in
          sys.stdin") which is not influenced by  this  option.   To  work
          around  this, you will want to use "sys.stdin.readline()" inside
          a "while 1:" loop.

除此之外，不支持更改现有文件的缓冲，但您可以使用与现有文件相同的底层文件描述符创建一个新文件对象，并且可能使用不同的缓冲，使用 os.fdopen.即，

Beyond this, altering the buffering for an existing file is not supported, but you can make a new file object with the same underlying file descriptor as an existing one, and possibly different buffering, using os.fdopen. I.e.,

import os
import sys
newin = os.fdopen(sys.stdin.fileno(), 'r', 100)

应该将 newin 绑定到一个文件对象的名称，该文件对象读取与标准输入相同的 FD，但一次仅缓冲大约 100 个字节(并且您可以继续 sys.stdin = newin 以使用新文件对象作为标准输入从那里开始).我说应该"是因为这个领域曾经在某些平台上存在许多错误和问题(提供具有完全通用性的跨平台功能非常困难)——我不确定它是什么状态是现在，但我绝对建议在所有感兴趣的平台上进行彻底的测试，以确保一切顺利.(-u，完全删除缓冲，应该可以在所有平台上减少问题，如果这可能满足您的要求).

should bind newin to the name of a file object that reads the same FD as standard input, but buffered by only about 100 bytes at a time (and you could continue with sys.stdin = newin to use the new file object as standard input from there onwards). I say "should" because this area used to have a number of bugs and issues on some platforms (it's pretty hard functionality to provide cross-platform with full generality) -- I'm not sure what its state is now, but I'd definitely recommend thorough testing on all platforms of interest to ensure that everything goes smoothly. (-u, removing buffering entirely, should work with fewer problems across all platforms, if that might meet your requirements).

这篇关于为 sys.stdin 设置较小的缓冲区大小?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！