Logstash csvparsefailure和dateparsefailure [英] Logstash csvparsefailure and dateparsefailure

问题描述

我正在使用此过滤器来解析从php文件生成的某些csv数据.我正在从名为msi afterburner的gpu监视工具获取输出，该工具输出.hml文件.有大量的空格和不相关的标头，我的php文件将其删除并输出以逗号分隔的值.

I am using this filter to parse some csv data that I am generating from a php file. I am taking the output from a gpu monitoring tool called msi afterburner which outputs a .hml file. There are a tonne of white spaces and an irrelevant header which my php file removes and outputs comma separated value.

 filter 
    {
        csv 
        {
            columns => ["somename","@timestamp","cpu.avg.temp","gpu.temp","fan.speed","gpu.usage","bus.usage","fan.tachometer","clock.core","framerate.hz","framerate.ms","cpu.temp.1","cpu.temp.2","cpu.temp.3","cpu.temp.4"]
            separator => ","
            skip_empty_columns => "true"
        }
        mutate 
        {
            convert => ["somename","integer"]
            convert => ["cpu.avg.temp","float"]
            convert => ["gpu.temp","float"]
            convert => ["fan.speed","float"]
            convert => ["gpu.usage","float"]
            convert => ["bus.usage","float"]
            convert => ["fan.tachometer","float"]
            convert => ["clock.core", "float"]
            convert => ["framerate.hz","float"]
            convert => ["framerate.ms","float"]
            convert => ["cpu.temp.1","float"]
            convert => ["cpu.temp.2","float"]
            convert => ["cpu.temp.3","float"]
            convert => ["cpu.temp.4","float"]
        }
        date 
        {
            match => ["@timestamp", "dd-MM-yyyyHH:mm:ss"]
        }
    }

这是输出logstash扔给我的信息.我想知道这是由于我的日期格式不正确，还是在消息末尾出现一个特殊字符'\ r'.我想知道logstash是否甚至可以读取dd-MM-yyyyHH:mm:ss格式，其中年和小时紧密结合在一起，否则我可能会遇到麻烦.

This is the output logstash is throwing at me. I am wondering if this is due to the fact that my date format is bad or if at the end of my message there appears to be a special character '\r'. I am wondering if logstash is even able to read dd-MM-yyyyHH:mm:ss format where year and hour are stuck together, if not I might be in a bit of trouble.

{
          "path" => "C:\\Users\\Public\\Documents\\gpumetrics.csv",
      "somename" => 80,
    "@timestamp" => 2017-02-20T02:33:10.764Z,
      "@version" => "1",
          "host" => "DESKTOP-Q8UEATO",
       "message" => "80,19-02-201721:33:10,32.000,41.000,0.000,0.000,0.000,0.000,215.000,0.000,0.000,31.000,32.000,30.000,31.000\r",
          "type" => "csv",
          "tags" => [
        [0] "_csvparsefailure",
        [1] "_dateparsefailure"
    ]
} 

这是我的日志文件中的一些示例行.您可能会注意到，时间戳记之前有一个字段.我想知道是否允许这样做.

Here are a few sample lines from my log file. As you may notice, there is a field before timestamp. I am wondering if this is allowed.

80,19-02-201713:20:32,44.000,43.000,0.000,0.000,0.000,0.000,215.000,,,37.000,42.000,41.000,38.000
80,19-02-201713:20:33,47.000,43.000,0.000,0.000,0.000,0.000,215.000,,,46.000,47.000,45.000,44.000
80,19-02-201713:20:34,53.000,43.000,0.000,0.000,0.000,0.000,215.000,,,35.000,50.000,36.000,37.000
80,19-02-201713:20:35,37.000,43.000,0.000,0.000,0.000,0.000,215.000,,,37.000,37.000,37.000,34.000
80,19-02-201713:20:36,34.000,44.000,0.000,0.000,0.000,0.000,1582.000,0.000,0.000,39.000,34.000,33.000,36.000
80,19-02-201713:20:37,46.000,44.000,0.000,0.000,0.000,0.000,1582.000,0.000,0.000,45.000,37.000,43.000,37.000

推荐答案

通过更改时间戳变量的名称可以非常简单地解决您的问题，因为 @timestamp 是在解析行之前在内部创建的

Your problem can be solved very simply by changing the name of your timestamp variable since @timestamp is created internally before your line is parsed.

filter 
    {
        csv 
        {
                               remove the @
                                    |
                                    v
            columns => ["somename","timestamp","cpu.avg.temp","gpu.temp","fan.speed","gpu.usage","bus.usage","fan.tachometer","clock.core","framerate.hz","framerate.ms","cpu.temp.1","cpu.temp.2","cpu.temp.3","cpu.temp.4"]
            separator => ","
            skip_empty_columns => "true"
        }
        ...
        date 
        {
            match => ["timestamp", "dd-MM-yyyyHH:mm:ss"]
                       ^
                       |
                  remove the @
        }
    }

这篇关于Logstash csvparsefailure和dateparsefailure的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！