使用logstash格式进行数据类型转换 [英] Data type conversion using logstash grok

问题描述

Basic 是一个浮动字段。上述指标不存在于弹性搜索中。当使用 logstash -f 运行配置文件时，我也不例外。然而，弹性搜索中反映并输入的数据显示了 Basic 作为 string 的映射。我该如何纠正？我如何为多个字段执行此操作？

Basic is a float field. The mentioned index is not present in elasticsearch. When running the config file with logstash -f, I am getting no exception. Yet, the data reflected and entered in elasticsearch shows the mapping of Basic as string. How do I rectify this? And how do I do this for multiple fields?

input {  
      file {
          path => "/home/sagnik/work/logstash-1.4.2/bin/promosms_dec15.csv"
          type => "promosms_dec15"
          start_position => "beginning"
          sincedb_path => "/dev/null"
      }
}
filter {
    grok{
        match => [
            "Basic", " %{NUMBER:Basic:float}"
        ]
    }

    csv {
        columns => ["Generation_Date","Basic"]
        separator => ","
    }  
    ruby {
          code => "event['Generation_Date'] = Date.parse(event['Generation_Date']);"
    }

}
output {  
    elasticsearch { 
        action => "index"
        host => "localhost"
        index => "promosms-%{+dd.MM.YYYY}"
        workers => 1
    }
}

推荐答案

你有两个问题。首先，您的Grok过滤器在csv过滤器之前列出，并且因为过滤器被应用，所以在应用grok过滤器时不会有Basic字段转换。

You have two problems. First, your grok filter is listed prior to the csv filter and because filters are applied in order there won't be a "Basic" field to convert when the grok filter is applied.

其次，除非您明确允许，否则grok不会覆盖现有的字段。换句话说，

Secondly, unless you explicitly allow it, grok won't overwrite existing fields. In other words,

grok{
    match => [
        "Basic", " %{NUMBER:Basic:float}"
    ]
}

将永远是一个无效的。请指定 overwrite => [基本] ，或者最好使用 mutate 的类型转换功能：

will always be a no-op. Either specify overwrite => ["Basic"] or, preferably, use mutate's type conversion feature:

mutate {
    convert => ["Basic", "float"]
}

这篇关于使用logstash格式进行数据类型转换的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！