Logstash-指定多个管道 [英] Logstash - specify more than one pipeline

问题描述

我希望以不同的方式处理不同的字段.

I want different fields to be processed in different way.

我有两个管道.一种是处理布尔值，另一种是将字符串转换为数组.

I have two pipelines. One is to process boolean values, another one is to convert a string to array.

 output {
    stdout {
        codec => rubydebug
    }
    elasticsearch {
        action => "index"
        hosts => ["127.0.0.1:9200"]
        index => "mini_system"
        document_id => "%{mini_system_key}"
        if [source] == "secure_flag" {
            pipeline => "bool-pipeline"
        } else if "partners" == %{FIELD} {
            pipeline => "partners-pipeline"
        }
    }
}

我正在尝试这样做.但是我无法做到这一点，也找不到参考.

I am trying to do this. But I am not able to achieve this and couldn't find a reference also.

示例文档:

key,partners,secure_flag,date_added
5369922730525,"1002300,1009747,12359,2285459",FALSE,2020-03-31T14:00:00Z    
2218100624,,FALSE,2020-03-31T14:00:00Z

在这里

"1002300,1009747,12359,2285459"是合作伙伴.FALSE是secure_flag.

"1002300,1009747,12359,2285459" is partners. FALSE is secure_flag.

合作伙伴管道:

{
  "description": "Converts \"a,b,c\" to [\"a\", \"b\",\"c\"]",
  "processors" : [
    {
      "split" : {
        "field" : "partners",
        "separator": ",",
        "ignore_missing": true
      }
    }
  ]
}

推荐答案

您不能在插件配置中应用逻辑，但是使用if/else逻辑肯定可以得到多个输出:

You cannot apply logic inside plugin configurations, but you can definitely have several output using if/else logic:

output {
    stdout {
        codec => rubydebug
    }
    if [source] == "secure_flag" {
        elasticsearch {
            action => "index"
            hosts => ["127.0.0.1:9200"]
            index => "mini_system"
            document_id => "%{mini_system_key}"
            pipeline => "bool-pipeline"
        }
    } else if [field_xyz] == "partners" {
        elasticsearch {
            action => "index"
            hosts => ["127.0.0.1:9200"]
            index => "mini_system"
            document_id => "%{mini_system_key}"
            pipeline => "partners-pipeline"
        }
    }
}

更新:

您实际上不需要任何逻辑，只需将两个处理器添加到同一管道中即可.

You don't actually need any logic, but simply add both of your processors in the same pipeline:

PUT _ingest/pipeline/mini-pipeline
{
  "processors" : [
    {
      "convert" : {
        "field" : "secure_flag",
        "type": "boolean",
        "ignore_missing": true
      }
    },
    {
      "split" : {
        "field" : "partners",
        "separator": ",",
        "ignore_missing": true
      }
    }
  ]
}

然后只需使用此配置

output {
    stdout {
        codec => rubydebug
    }
    elasticsearch {
        action => "index"
        hosts => ["127.0.0.1:9200"]
        index => "mini_system"
        document_id => "%{mini_system_key}"
        pipeline => "mini-pipeline"
    }
}

这篇关于Logstash-指定多个管道的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！