聚合来自多个服务器的多个日志文件的最佳方法 [英] Best way to aggregate multiple log files from several servers

问题描述

我需要一种简单的方法来监视分布在许多HP-UX服务器上的多个文本日志文件.它们是来自多个分布式旧系统的文本和XML日志文件的混合.当前，我们仅使用SSH到服务器并使用 tail -f 和 grep ，但是当您需要跟踪许多日志时，这种方法无法扩展.

I need a simple way to monitor multiple text log files distributed over a number of HP-UX servers. They are a mix of text and XML log files from several distributed legacy systems. Currently we just ssh to the servers and use tail -f and grep, but that doesn't scale when you have many logs to keep track of.

由于日志的格式不同，只是文件夹中的文件(当它们达到特定大小时会自动旋转)，所以我既需要远程收集它们，又要分别解析每个文件.

Since the logs are in different formats and just files in folders (automatically rotated when they reach a certain size) I need to both collect them remotely and parse each one differently.

我最初的想法是做一个简单的守护进程，我可以使用每种文件类型的自定义文件读取器在每台服务器上运行，以将其解析为可以通过套接字通过网络导出的通用格式.另一个在本地运行的查看器程序将连接到这些套接字，并在某些简单的选项卡式GUI中显示已解析的日志，或者将其汇总到控制台中.

My initial thought was to make a simple daemon process that I can run on each server using a custom file reader for each file type to parse it into a common format that can be exported over the network via a socket. Another viewer program running locally will connect to these sockets and show the parsed logs in some simple tabbed GUI or aggregated to a console.

如果要以这种方式实现，应该尝试转换为哪种日志格式?

What log format should I try to convert to if I am to implement it this way?

还有其他更简单的方法吗?我应该尝试将日志文件转换为log4j格式以与链锯一起使用，还是有更好的日志?可以连接到远程套接字的查看器?是否可以按照大规模分布的系统，并且不会更改所有应用程序的当前日志记录实现不能选择使用UDP广播或将消息放在JMS队列上.

Is there some other easier way? Should I attempt to translate the log files to the log4j format to use with Chainsaw or are there better log viewers that can connect to remote sockets? Could I use BareTail as suggested in another log question? This is not a massivly distributed system and changing the current logging implementations for all applications to use UDP broadcast or put messages on a JMS queue is not an option.

推荐答案

选项:

1. 使用SocketAppender将所有日志直接发送到1个服务器. (这可能会严重影响性能并增加单点故障.)
2. 使用脚本来聚合数据.我使用scp，ssh和身份验证密钥来允许我的脚本从所有服务器获取数据，而无需任何登录提示.

这篇关于聚合来自多个服务器的多个日志文件的最佳方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！