开销最小的分布式事件日志库/框架? [英] Minimal overhead distributed event logging library/framework?

问题描述

我们希望保留系统中所有重大事件的记录.例如，在数据库可能存储当前用户状态的地方，事件日志应记录对该状态的所有更改以及更改发生的时间.

We'd like to keep records of all major events in our systems. For example, where the database might store the current user status, the event log should record all changes to that status along with when the changes occurred.

事件日志记录功能应尽可能使事件抛出者的开销接近零，应容纳结构化信息(与文本日志消息相对)，并应支持分布式部署(许多盒子抛出许多事件).

The event logging facility should be as close to zero overhead for the event thrower as possible, should accommodate structured information (as opposed to text log messages), and should support distributed deployment (many boxes throwing many events).

在过去的生活中，我们有一个基于UDP的系统，该系统运行良好，因为我们对系统有很好的控制权(最大限度地减少了数据包丢失).偶数抛出者将触发将被捕获并记录在其他设备上的UDP数据包.我正在寻找类似的，希望开源的，现成的并且可以在更通用的网络中部署的东西.另外，我也乐意接受有关如何构建类似内容的建议.

In a past life we had a UDP based system that worked well because we had great control over the system (minimized packet loss). The even throwers would fire off UDP packets that would be caught and journaled on other boxes. I'm looking for something similar, hopefully open source, off the shelf, and deployable in more general networks. Alternatively I'm open to suggestions for how to build something like this.

这应该适用于多种语言，但主要针对Java和Python.初始化(事件抛出)应用程序将有所不同；一些将是网络应用程序，另一些将是面向批处理的应用程序.结果可能会存在于Hadoop/HDFS/HBase中.

This should work across multiple languages, but will be primarily targeted for Java and Python. The pariticipating (event throwing) applications will vary; some will be web apps, others batch oriented apps. The results will likely live in Hadoop/HDFS/HBase.

推荐答案

如果您想断开UDP路由(您对此似乎很满意)，并且可以选择Java，那么请查看

If you want to go down the UDP route (as you seem happy with that), and Java is an option, then check out Log4j and its support for UDP transmission via the Log4j UDPAppender.

LoggingEvent将以java.lang.Object作为消息，因此它非常通用，您可以将所需的任何数据放入其中.如果您要跨网络访问，它应该(最有可能)是可序列化的，并且考虑到您想要UDP，它的大小应相称-64k或更小，然后取决于传输层).您只需要在服务器端拦截LoggingEvent，然后根据需要对其进行处理即可.

LoggingEvent will take a java.lang.Object as a message, so it's pretty generic and you can throw whatever data you want into that. If you're going across the network it should (most likely) be serialisable, and given that you want UDP, should be of a comensurate size - 64k or less, and then dependent on the transport layer). You'll simply have to intercept the LoggingEvent on the server side and then process it however you want.

请注意，UDP附加程序是Log4j的 companion 组件，您必须自己构建它.但这很简单.

Note that the UDP appender comes as a companion component to Log4j and you'll have to build it yourself. But that's trivial.

这篇关于开销最小的分布式事件日志库/框架?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！