如何使用logstash将队列的内容发送到弹性搜索索引 [英] How to send contents of a queue to elasticsearch index with logstash

问题描述

我有一个logstash运行，消耗两个兔子队列并发送到弹性搜索。这是我的logstash.conf文件：

  input {
 rabbitmq {
 host => 'rabbit'
 durable => true 
 user => 'user'
 queue => 'dev-user_trace'
 password => 'pass'
} 
 rabbitmq {
 host => 'rabbit'
 durable => true 
 user => 'user'
 queue => 'min-price-queue'
 password => 'pass'
} 
 
} 
过滤器{
} 
输出{
 stdout {codec => json} 
 elasticsearch {
 hosts => [elasticsearch] 
 index => eventss  - ％{+ YYYY.MM.dd}
} 
 
} 
  

现在我有另一个队列，但是我想将其内容发送到一个不同的弹性搜索索引。我的问题是：如何将特定条目重定向到特定索引？还是我需要另一个logstash实例？

提前感谢

解决方案

p>很好的开始现在，您只需要输入每个输入，然后将事件转发到给定类型的相应输出，如下所示：

 输入{
 rabbitmq {
 host => 'rabbit'
 durable => true 
 user => 'user'
 queue => 'dev-user_trace'
 password => 'pass'
 type => 'trace'＃<  -  add this 
} 
 rabbitmq {
 host => 'rabbit'
 durable => true 
 user => 'user'
 queue => 'min-price-queue'
 password => 'pass'
 type => 'price'＃<  -  add this 
} 
 
} 
 filter {
} 
 output {
 stdout {codec => ; json} 
 
 if [type] =='traces'{＃<  -   -  check type 
 elasticsearch {
 hosts => [host1：9200] 
 index => index1  - ％{+ YYYY.MM.dd}
} 
} 
 
如果[type] =='price'{＃<  -  check type 
 elasticsearch {
 hosts => [host2：9200] 
 index => index2  - ％{+ YYYY.MM.dd}
} 
} 
} 
  

更新

以上是最通用的方法，因此您可以不同地配置两种输出。根据@pandaadb的建议，您还可以输入一个输出并定义一个类型，这将成为您的目标索引：

  input { 
 rabbitmq {
 host => 'rabbit'
 durable => true 
 user => 'user'
 queue => 'dev-user_trace'
 password => 'pass'
 type => 'index1'＃<  - 添加此
} 
 rabbitmq {
 host => 'rabbit'
 durable => true 
 user => 'user'
 queue => 'min-price-queue'
 password => 'pass'
 type => 'index2'＃<  - 添加此
} 
 
} 
过滤器{
} 
输出{
 stdout {codec => ; json} 
 
 elasticsearch {
 hosts => [localhost：9200] 
 index => ％{type}  - ％{+ YYYY.MM.dd}＃<  - 使用类型
} 
} 
  

I have a logstash up and running that consumes two rabbit queues and sends to an elasticsearch. This is my logstash.conf file:

input {
  rabbitmq {
    host => 'rabbit'
    durable => true
    user => 'user'
    queue => 'dev-user_trace'
    password => 'pass'
  }
  rabbitmq {
    host => 'rabbit'
    durable => true
    user => 'user'
    queue => 'min-price-queue'
    password => 'pass'
  }

}
filter{
}
output{
  stdout { codec => json}
    elasticsearch{
    hosts => ["elasticsearch"]
    index => "eventss-%{+YYYY.MM.dd}"
  }

}

Now I have another queue, but I want to send its content to a different elasticsearch index. My question is: how do I need to redirect specific entries to an specific index? Or do I need another logstash instance?

Thanks in advance.

解决方案

Very good start. Now you simply need to "type" each input and then forward the events to the appropriate output given its type, like this:

input {
  rabbitmq {
    host => 'rabbit'
    durable => true
    user => 'user'
    queue => 'dev-user_trace'
    password => 'pass'
    type => 'traces'               # <-- add this
  }
  rabbitmq {
    host => 'rabbit'
    durable => true
    user => 'user'
    queue => 'min-price-queue'
    password => 'pass'
    type => 'prices'               # <-- add this
  }

}
filter{
}
output{
  stdout { codec => json}

  if [type] == 'traces' {          # <-- check type
     elasticsearch{
       hosts => ["host1:9200"]
       index => "index1-%{+YYYY.MM.dd}"
     }
  }

  if [type] == 'prices' {          # <-- check type
     elasticsearch{
       hosts => ["host2:9200"]
       index => "index2-%{+YYYY.MM.dd}"
     }
  }
}

UPDATE

The above is the most general approach so that you can configure both outputs differently. As suggested by @pandaadb, you can also have a single output and define a type that would be your target index:

input {
  rabbitmq {
    host => 'rabbit'
    durable => true
    user => 'user'
    queue => 'dev-user_trace'
    password => 'pass'
    type => 'index1'                    # <-- add this
  }
  rabbitmq {
    host => 'rabbit'
    durable => true
    user => 'user'
    queue => 'min-price-queue'
    password => 'pass'
    type => 'index2'                    # <-- add this
  }

}
filter{
}
output{
  stdout { codec => json}

  elasticsearch{
    hosts => ["localhost:9200"]
    index => "%{type}-%{+YYYY.MM.dd}"   # <-- use type here
  }
}

这篇关于如何使用logstash将队列的内容发送到弹性搜索索引的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！