大规模分布系统中的日志文件 [英] Log files in massively distributed systems

问题描述

我在网格和HPC空间中进行了大量工作，而在数百个（有时甚至数千个）服务器中分布的系统所面临的最大挑战之一就是分析日志文件。

I do a lot of work in the grid and HPC space and one of the biggest challenges we have with a system distributed across hundreds (or in some case thousands) of servers is analysing the log files.

当前日志文件本地写入每个刀片上的磁盘，但我们也可以考虑使用UDP Appender发布日志信息并集中收集。

Currently log files are written locally to the disk on each blade but we could also consider publishing logging information using for example a UDP Appender and collect it centally.

鉴于目标是能够尽可能接近实时地发现问题，我们该怎么办？

Given that the objective is to be able to identify problems in as close to real time as possible, what should we do?

推荐答案

首先，使用NTP同步系统中的所有时钟。

First, synchronize all clocks in the system using NTP.

第二，如果要在单个位置收集日志（ （例如您提到的UDP附加程序），请确保日志具有足够的信息以提供实际帮助。我将至少包括生成日志的服务器，发生时间和消息。如果存在任何类型的事务处理ID或作业ID类型的概念，也应包括在内。

Second, if you are collecting the logs in a single location (like the UDP appender you mention) make sure the logs have enough information to actually help. I would include at least the server that generated the log, the time it happened, and the message. If there is any sort of transaction id, or job id type concept, include that also.

由于您提到了UDP Appender，所以我猜您正在使用log4j（或其中的一个）。 Log4j具有MDC类，该类允许通过处理线程传递额外的信息。它可以帮助收集一些额外的信息并将其传递。

Since you mentioned a UDP Appender I am guessing you are using log4j (or one of it's siblings). Log4j has an MDC class that allows extra information to be passed along through a processing thread. it can help collect some of the extra information and pass it along.

这篇关于大规模分布系统中的日志文件的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！