即时分析和绘制从日志文件接收到的数据 [英] Parse and plot data received from log-file on-the-fly

问题描述

我想创建一个Perl(或Bash)脚本来动态创建和绘制数据.这意味着我想从日志文件 file.log 中提取数据，理想情况下不要编写tmp文件(如果只能使用tmp文件，也可以)，并且用Gnuplot进行绘图.随着日志文件的增长，我想在图中查看其他信息.

I want to create a Perl (or Bash) script to create and plot data on-the-fly. That means I want to extract data from a log-file file.log, ideally don't write a tmp-file (if it's only possible with a tmp-file, this would be fine too) and plot with Gnuplot. As the logfile is growing, I want to see the additional information in the plot.

有关类似问题，请参见此问题主题.

See this question for similar topic.

对于Perl，到目前为止，我的脚本是:

For Perl, what I have so far is this script:

#!/usr/bin/perl
use strict;
use warnings;

my $path = "file.log";   
my @grepped;
my $switch = "off";

open(INFILE,"< $path") or die $! \n";
while (my $line = <INFILE>) {

       if ($line =~ m{^Time = (\d+)}){
               push(@grepped,"$1\t");
       };

       if ($line =~ m{^Errors: local = (\d+), global = (\d+)}){
               push(@grepped,"$1\t");
               push(@grepped,"$2\n");
               $switch = "refresh";
       };


if ($switch eq "refresh"){

open(GP, "| gnuplot -persist") or die: $! \n";
print GP << "GNU_EOF";

plot "@grepped" u 2:1
pause 1; refresh; reread;

GNU_EOF

close(GP);
}


}
close(INFILE)};

我的第一个问题是Gnuplot的即时功能无法正常工作.更改 file.log 时，Gnuplot图形的刷新不起作用.我不知道-persist在这里是否正确，或者我是否必须使用replot选项而不是refresh.我尝试过，但是对我不起作用.

My first problem is that the on-the-fly functionality for Gnuplot isn't working. When the file.log gets changed, the refreshing of the Gnuplot-graph isn't working. I don't know if the -persist is correct here or if I have to use replot option instead of refresh. I tried it but it's not working for me.

第二个问题是将数据读取到数组 @grepped 中，并在一个脚本中的Gnuplot中使用它来更改输入文件.另外一个限制是(如从代码中可以看到的)如果在 @grepped 中写了一条完整的新行，则gnuplot仅应刷新图.否则，肯定会由于错误的数据分配而发生错误.

Second problem is to read the data into array @grepped and use it in Gnuplot in one script for a changing input file. One additional restriction is that (as you can see from the code) gnuplot only should refresh the plot if a complete new line in @grepped is written. Otherwise errors would occur for sure because of the wrong assignment of data.

当我尝试一个简单的脚本时，例如:

When I try a simple script like:

#!/usr/bin/gnuplot -persist
plot "data.dat" u 2:1
pause 1; replot; reread;

或

#!/usr/bin/gnuplot
plot "data.dat" u 2:1
pause 1; refresh; reread;

如果我手动更改data.dat并保存它，那么即时的部分会起作用.

the on-the-fly part works if I change data.dat by hand and save it.

推荐答案

以下是即时绘制数据的两种方法.

Here are two ways for plotting data on-the-fly.

您必须一遍又一遍地调用plot，数据将由外部脚本进行预处理.最小的gnuplot脚本filter.gp是:

You must call plot over and over again, the data is preprocessed by an external script. The minimal gnuplot script filter.gp is:

while (1) {
    plot '< ./myscript.pl' using 2:1
    pause 1
}

要停止此操作，请按Ctrl+C.

To stop this, hit Ctrl+C.

用于预处理的Perl脚本可能类似于以下myscript.pl:

The Perl script for the preprocessing may look like the following myscript.pl:

#!/usr/bin/perl
use strict;
use warnings;

my $path = "file.log";   
my @grepped;
my $t = 0;
open(INFILE,"< $path") or die "$! \n";

while (my $line = <INFILE>) {
    if ($line =~ m{^Time = (\d+)}){
        $t = $1;
    };

    if ($line =~ m{^Errors: local = (\d+), global = (\d+)}){
        print "$t\t$1\t$2\n";
    };
};
close(INFILE);

只需使用gnuplot filter.gp运行它即可.

要使其更具可配置性，可以将脚本更改为使用通过命令行传递给gnuplot的变量:

To make it more configurable, one can change the script to use a variable which is passed to gnuplot via the command line:

while (1) {
    plot '< ./myscript.pl -f '.path using 2:1
    pause 1
}

或为此使用reread

plot '< ./myscript.pl -f '.path using 2:1
pause 1
reread

通过gnuplot -e "path='file.log';" filtermod.gp调用此脚本.

可以，但是会一遍又一遍地过滤整个文件.

That works, but would filters the complete file over and over again.

这是一个Perl脚本，基本上对我有用，但是它是我的第一个真正的Perl脚本，因此可能有一些不理想的部分.随时对此发表评论.

Here is a Perl script, which basically works for me, but its my first real Perl script, so there may be some nonideal parts. Feel free to comment on that.

#!/usr/bin/perl
use strict;
use warnings;

my $path = "file.log";   
my @grepped;
my $switch = "off";

open(my $gp, "| gnuplot -persist") or die "$! \n";
$gp->autoflush(0);

use File::Tail;
my $file = File::Tail->new(name=>$path, maxinterval=>1, tail=>-1);

while (defined(my $line= $file->read)) {
    if ($line =~ m{^Time = (\d+)}){
        push(@grepped,"$1\t");
    };

    if ($line =~ m{^Errors: local = (\d+), global = (\d+)}){
        push(@grepped,"$1\t");
        push(@grepped,"$2\n");
        $switch = "refresh";
    };

    if ($switch eq "refresh") {
        print $gp <<"GNU_EOF";
plot '-' using 2:1
@grepped
e
GNU_EOF
        $gp->flush;
        $switch = "off"; 
    };
    Time::HiRes::sleep(0.1);
};

我发现这里很重要

1. 环绕正在更改的文件.
2. autoflush和flush的排列.
3. sleep，允许gnuplot正确处理数据.

1. Looping over a changing file.
2. The arrangement of the autoflush and flush.
3. The sleep to allow gnuplot to process the data properly.

这对于一个非常小的测试数据文件有用.不知道它是否还会适合更大的人，但应该会为您提供更多帮助.

That worked for a very small test data file. Don't know if it will for a larger one as well, but should help you a bit further.

这篇关于即时分析和绘制从日志文件接收到的数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！