使用 Powershell 在非常大的日志文件的最后 x 行中搜索特定字符串 [英] Using Powershell to search for specific strings in the last x lines of very large log files

问题描述

我有一个需求，但到目前为止我还没有在论坛上找到解决方案或帮助...

I have a requirement that I have not found a solution or assistance for on the forums so far...

问题:我监控实时系统，它们会生成许多每天滚动的非常大的日志文件，1GB 并不少见，我所做的是查看(尾部)日志已知错误条件或我想要监控的其他字符串，我可能需要对其采取措施.

The issue: I monitor real time systems and these generate a number of very large log files that roll over on a daily basis, 1GB is not uncommon, and what I do is look through the (tail end of the) logs for known error conditions or other strings that I want to monitor for, which I may need to take action on.

由于这种工作既费时又乏味，而且很容易因人为错误而错过日志中的问题，因此我一直在自动化此日志文件监控.我使用名为 Servers Alive 的产品执行计划检查，并编写脚本来监视日志文件中出现的日志条目，我有兴趣查找这些条目可能表明服务存在问题，如果需要，我可以调用其他脚本来重新启动服务解决遇到的问题.

As this sort of work is both time consuming and tedious, and easy for human error to miss issues within logs, I have been automating this log file monitoring. I utilise a product called Servers Alive to perform scheduled checks and I write scripts to monitor log files for occurrences of log entries that I am interested in looking for which may indicate problems with services, and I can than call other scripts to restart services if necessary to resolve the issue that has been encountered.

我以前使用 Perl 为这些日志监控任务编写脚本，这些脚本非常快，但不一定是最干净的方法，我是管理员而不是程序员，所以我没有开发人员方法或可依赖的经验.

I have previously done the scripting for these log monitoring tasks using Perl and these scripts are very quick, but not necessarily the cleanest way to do this, I'm an admin rather that programmer so I don't have the developer methodologies or experience to rely on.

下面的 Perl 代码片段显示我打开"了一个日志文件 $logfile，然后从文件末尾向后搜索"给定数量的数据，然后我搜索从这一点到我感兴趣的日志条目的文件末尾的数据，在本例中，日志条目是No packet received from EISEC Client"

The Perl code snippet below shows that I 'open' a log file $logfile, and then 'seek' backwards from the end of the file for a given amount of data and then I search through the data from this point to the end of the file for the log entry I am interested in monitoring, in this example the log entry is "No packet received from EISEC Client"

在此示例中，我们正在监控的日志条目表明 EISEC 服务存在问题，并且该服务的简单重启通常可以解决该问题，所有这些我都会自动使用 Servers Alive 作为计划检查和警报机制.

In this example the log entry we are monitoring for indicates an issue with the EISEC service and a simple restart of the service normally resolves the issue, all of which I do automatically utilising Servers Alive as the scheduled check and alerting mechanism.

Perl 脚本函数

sub checkEisecSrvloggedon {

print "$logfile\n";

if (open (EISECSRV_LOGGEDON, $logfile)) {

    seek (EISECSRV_LOGGEDON, -40000, 2);

    $line = <EISECSRV_LOGGEDON>;

    $eisecsrvloggedon_ok = 0;

    while ($line = <EISECSRV_LOGGEDON>) {   

            if ($line =~/No packet received from EISEC Client/) {

                #increment counter
                ++$eisecsrvloggedon_ok; 

                }

            }
        }
    }

如果可能的话，我想为此使用 PowerShell 实施解决方案，因为我们已经转移到 Windows Server 2008 R2 和 Windows 7 客户端，但我找不到详细说明我如何高效、快速且没有任何大量内存开销.

I would like to implement a solution for this using PowerShell if possible now that we have moved on to Windows Server 2008 R2 and Windows 7 clients, but I cannot find details of how I could efficiently do this, quickly and without any large memory overhead.

我尝试过基于 Get-Content 的解决方案，但需要读取整个日志文件使得这些类型的解决方案无法使用，因为查询日志文件需要很长时间.我需要能够定期检查这些大型日志文件中的一些，在某些情况下每隔几分钟检查一次.我见过非常适合拖尾日志文件末尾的尾部类型解决方案，这些脚本使用 System.IO.File 方法.这确实获得了我希望在我的脚本中实现的性能/速度，但我对 PowerShell 不够熟悉，不知道如何使用这种方法快速到达大型日志文件的末尾，并且然后能够向后读取给定数量的数据，然后在此日志部分中搜索相关字符串.

I have tried Get-Content based solutions but the need to read the whole logfile makes these type of solutions unusable as it takes far too long to query the logfile. I need to be able to check a number of these large logfile on a very regular basis, once every few minutes in some cases. I have seen tail type solutions that are great for tailing the end of the logs files, and these scripts are using System.IO.File methods. This does get the performance / speed I would like to achieve in my scripts, but I am not familiar enough with PowerShell to know how to use this methodology to get quickly to the end of a large logfile, and then be able to read backwards for a given amount of data and then search for the relevant strings in this log section.

有人有什么想法吗?

推荐答案

如果您使用 PowerShell 3.0 或更新版本，您可以使用 get-content 和 select-string<的组合/code> commandlet 以获得相同的功能.从 3.0 版开始，get-content 支持 -tail 选项，该选项仅以有效的方式返回文件的最后 n 行.使用它，您可以使用以下内容(在最后 1000 行中搜索)重新实现上面的 Perl 脚本:

If you use PowerShell 3.0 or newer, you can use a combination of the get-content and select-string commandlets to get the same functionality. Since version 3.0, get-content support a -tail option that only returns the last n lines of a file in an efficient way. Using this, you can reimplement the Perl script above with the following (searching in the last 1000 lines):

# Returns the number of occurrences
(get-content logfile.txt -Tail 1000 | select-string -pattern "No packet received from EISEC Client").Length

这篇关于使用 Powershell 在非常大的日志文件的最后 x 行中搜索特定字符串的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！