从多个服务器聚合多个日志文件的最佳方法 [英] Best way to aggregate multiple log files from several servers

问题描述

我需要一种简单的方法来监控分布在多个 HP-UX 服务器上的多个文本日志文件.它们是来自多个分布式遗留系统的文本和 XML 日志文件的混合.目前我们只是通过 ssh 连接到服务器并使用 tail -f 和 grep，但是当您有很多日志要跟踪时，这不会扩展.

I need a simple way to monitor multiple text log files distributed over a number of HP-UX servers. They are a mix of text and XML log files from several distributed legacy systems. Currently we just ssh to the servers and use tail -f and grep, but that doesn't scale when you have many logs to keep track of.

由于日志的格式不同，而且只是文件夹中的文件(当它们达到一定大小时会自动旋转)，我需要远程收集它们并以不同的方式解析每个.

Since the logs are in different formats and just files in folders (automatically rotated when they reach a certain size) I need to both collect them remotely and parse each one differently.

我最初的想法是制作一个简单的守护进程，我可以在每台服务器上运行该进程，使用每个文件类型的自定义文件阅读器将其解析为可以通过套接字通过网络导出的通用格式.另一个在本地运行的查看器程序将连接到这些套接字并在一些简单的选项卡式 GUI 中显示解析的日志或聚合到控制台.

My initial thought was to make a simple daemon process that I can run on each server using a custom file reader for each file type to parse it into a common format that can be exported over the network via a socket. Another viewer program running locally will connect to these sockets and show the parsed logs in some simple tabbed GUI or aggregated to a console.

如果我要以这种方式实现它，我应该尝试转换成什么日志格式?

What log format should I try to convert to if I am to implement it this way?

还有其他更简单的方法吗?我应该尝试将日志文件转换为 log4j 格式以与 Chainsaw 一起使用还是有更好的日志可以连接到远程套接字的查看器?我可以按照 BareTail/18632/how-to-monitor-a-text-file-in-realtime#18690">另一个日志问题?这不是一个大规模分布式系统并改变所有应用程序的当前日志实现使用 UDP 广播或将消息放入 JMS 队列不是一种选择.

Is there some other easier way? Should I attempt to translate the log files to the log4j format to use with Chainsaw or are there better log viewers that can connect to remote sockets? Could I use BareTail as suggested in another log question? This is not a massivly distributed system and changing the current logging implementations for all applications to use UDP broadcast or put messages on a JMS queue is not an option.

推荐答案

选项:

1. 使用 SocketAppender 将所有日志直接发送到 1 个服务器.(这可能会严重影响性能并增加单点故障.)
2. 使用脚本聚合数据.我使用 scp、ssh 和身份验证密钥来允许我的脚本从所有服务器获取数据，而无需任何登录提示.

这篇关于从多个服务器聚合多个日志文件的最佳方法的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！