tail -f 登录服务器，处理数据，然后通过twisted提供给客户端 [英] Tail -f log on server, process data, then serve to client via twisted

问题描述

目标:在客户端的 wxPython GUI 中显示来自服务器的数据

Goal: Show data from server in wxPython GUI on client

Twisted 的新手.我有一个在 Windows 7 客户端上运行的 wxPython GUI，我有一个在 Ubuntu 服务器上运行的程序可以生成日志.我目前的尝试是 tail -f 日志，将输出通过管道传输到扭曲的服务器，然后将满足我的正则表达式条件的任何数据提供给客户端.我已经打开了一个隧道，所以我不需要用 SSH 把事情复杂化.我已经运行了以下代码块，但它只提供输入中的第一行.我知道我需要不断检查换行符的输入，然后将其写入传输，但我不确定如何在不中断连接的情况下执行此操作.

Newcomer to Twisted. I have a wxPython GUI running on a Windows 7 client, and I have a program running on an Ubuntu server that produces a log. My current attempt is to tail -f the log, pipe the output to a twisted server, then serve any data that meets my regex conditions to the client. I already have a tunnel open, so I don't need to complicate things with SSH. I've gotten the following block of code running, but it only serves the first line in the input. I know I need to keep checking the input for a newline and then writing it to the transport, but I'm not sure how to do that without breaking the connection.

我找不到足够的信息来修补完整的解决方案.我也尝试过使用套接字和文件 IO 的各种其他方法，但我认为 Twisted 似乎是解决此问题的好工具.我在正确的轨道上吗?任何建议表示赞赏.谢谢

I haven't been able to find enough information to patch a full solution together. I have also tried various other methods using sockets and file IO, but I think Twisted seems to be a good tool for this issue. Am I on the right track? Any recommendations appreciated. Thanks

#! /usr/bin/python

import optparse, os, sys

from twisted.internet.protocol import ServerFactory, Protocol

def parse_args():
    usage = """usage: %prog [options]
"""

    parser = optparse.OptionParser(usage)

    help = "The port to listen on. Default to a random available port."
    parser.add_option('--port', type='int', help=help)

    help = "The interface to listen on. Default is localhost."
    parser.add_option('--iface', help=help, default='localhost')

    options =parser.parse_args()

    return options#, log_file

class LogProtocol(Protocol):
    def connectionMade(self):
        for line in self.factory.log:
            self.transport.write(line)

class LogFactory(ServerFactory):
    protocol = LogProtocol

    def __init__(self,log):
        self.log = log

def main():
    log = sys.stdin.readline()
    options, log_file = parse_args()

    factory = LogFactory(log)

    from twisted.internet import reactor

    port = reactor.listenTCP(options.port or 0, factory,
                             interface=options.iface)

    print 'Serving %s on %s.' % (log_file, port.getHost())

    reactor.run()


if __name__ == '__main__':
    main()

为了回答第一条评论，我还尝试从 Python 中读取日志，程序挂起.代码如下:

To answer the first comment, I have also tried to just read the log from within Python, program hangs. Code follows:

#! /usr/bin/python

import optparse, os, sys, time
from twisted.internet.protocol import ServerFactory, Protocol

def parse_args():
    usage = """ usage: %prog [options]"""

    parser = optparse.OptionParser(usage)

    help = "The port to listen on. Default to a random available port"
    parser.add_option('--port', type='int', help=help, dest="port")

    help = "The logfile to tail and write"
    parser.add_option('--file', help=help, default='log/testgen01.log',dest="logfile")

    options = parser.parse_args()
    return options

class LogProtocol(Protocol):
    def connectionMade(self):
        for line in self.follow():
            self.transport.write(line)
        self.transport.loseConnection()

    def follow(self):
        while True:
            line = self.factory.log.readline()
            if not line:
                time.sleep(0.1)
                continue
            yield line

class LogFactory(ServerFactory):
    protocol = LogProtocol

    def __init__(self,log):
        self.log = log

def main():
    options, log_file = parse_args()
    log = open(options.logfile)
    factory = LogFactory(log)

    from twisted.internet import reactor

    port = reactor.listenTCP(options.port or 0, factory)    #,interface=options.iface)

    print 'Serving %s on %s.' % (options.logfile, port.getHost())

    reactor.run()


if __name__ == '__main__':
    main()

推荐答案

您在这里尝试实现几个不同的易于分离的目标.首先说一下看日志文件.

You've got a few different easily separated goals you're trying to achieve here. First, I'll talk about watching the log file.

您的发电机有几个问题.其中之一很大 - 它调用 time.sleep(0.1).sleep 功能块传递给它的时间量.当它阻塞时，调用它的线程不能做任何其他事情(毕竟这大致就是阻塞"的意思).您在调用 LogProtocol.connectionMade 的同一线程中迭代生成器(因为 connectionMade 调用 follow).LogProtocol.connectionMade 在运行 Twisted 反应器的同一线程中调用，因为 Twisted 大致是单线程的.

Your generator has a couple problems. One of them is big - it calls time.sleep(0.1). The sleep function blocks for the amount of time passed to it. While it is blocking, the thread which called it can't do anything else (that's roughly what "blocking" means, after all). You're iterating over the generator in the same thread as LogProtocol.connectionMade is called in (since connectionMade calls follow). LogProtocol.connectionMade is called in the same thread as the Twisted reactor is running, because Twisted is roughly single threaded.

因此，您使用 sleep 调用阻塞了反应器.只要 sleep 阻塞了反应器，反应器就不能做任何事情——比如通过套接字发送字节.顺便说一下，阻塞是可传递的.所以 LogProtocol.connectionMade 是一个更大的问题:它无限地迭代、睡眠和阅读.所以它无限期地阻塞反应堆.

So, you're blocking the reactor with the sleep calls. As long as sleep is blocking the reactor, the reactor can't do anything - like send bytes over sockets. Blocking is transitive, by the way. So LogProtocol.connectionMade is an even bigger problem: it iterates indefinitely, sleeping and reading. So it blocks the reactor indefinitely.

您需要在不阻塞的情况下从文件中读取行.你可以通过轮询来做到这一点——这实际上是你现在正在采取的方法——但要避免睡眠调用.使用 reactor.callLater 来安排将来从文件中读取:

You need to read lines from the file without blocking. You can do this by polling - which is effectively the approach you're taking now - but avoiding the sleep call. Use reactor.callLater to schedule future reads from the file:

def follow(fObj):
    line = fObj.readline()
    reactor.callLater(0.1, follow, fObj)

follow(open(filename))

你也可以让 LoopingCall 处理使这个循环永远运行的部分:

You can also let LoopingCall deal with the part that makes this a loop that runs forever:

def follow(fObj):
    line = fObj.readline()

from twisted.internet.task import LoopingCall

loop = LoopingCall(follow, open(filename))
loop.start(0.1)

这两种方法都可以让您随着时间的推移从文件中读取新行，而不会阻塞反应器.当然，他们都只是在读完之后把那条线放在地板上.这让我想到了第二个问题......

Either of these will let you read new lines from the file over time without blocking the reactor. Of course, they both just drop the line on the floor after they read it. This leads me to the second problem...

您需要对文件中出现新行做出反应.大概你想把它写出来给你的连接.这并不太难:反应"很容易，它通常只意味着调用函数或方法.在这种情况下，最简单的方法是让 LogProtocol 设置日志跟随并提供回调对象来处理出现的行.考虑对上面的 follow 功能进行的这种轻微调整:

You need to react to the appearance of a new line in the file. Presumably you want to write it out to your connection. This isn't too hard: "reacting" is pretty easy, it usually just means calling a function or a method. In this case, it's easiest to have the LogProtocol set up the log following and supply a callback object to handle lines when they appear. Consider this slight adjustment to the follow function from above:

def follow(fObj, gotLine):
    line = fObj.readline()
    if line:
        gotLine(line)

def printLine(line):
    print line

loop = LoopingCall(follow, open(filename), printLine)
loop.start(0.1)

现在您可以非阻塞地轮询日志文件中的新行并了解新行何时实际出现.这很容易与 LogProtocol...

Now you can non-blockingly poll a log file for new lines and learn when one has actually shown up. This is simple to integrate with LogProtocol...

class LogProtocol(Protocol):
    def connectionMade(self):
        self.loop = LoopingCall(follow, open(filename), self._sendLogLine)
        self.loop.start()

    def _sendLogLine(self, line):
        self.transport.write(line)

最后一个细节是，您可能希望在连接丢失时停止观看文件:

One last detail is that you probably want to stop watching the file when the connection is lost:

    def connectionLost(self, reason):
        self.loop.stop()

因此，此解决方案通过使用 LoopingCall 而不是 time.sleep 来避免阻塞，并在使用简单的方法调用发现行时将行推送到协议.

So, this solution avoids blocking by using LoopingCall instead of time.sleep and pushes lines to the protocol when they're found using simple method calls.

这篇关于tail -f 登录服务器，处理数据，然后通过twisted提供给客户端的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！