在Logstash中定义多个输出，同时处理Elasticsearch实例的潜在不可用性 [英] Defining multiple outputs in Logstash whilst handling potential unavailability of an Elasticsearch instance

问题描述

我为Logstash配置了两个输出，因为我需要将数据传递到位于不同位置的两个单独的Elasticsearch节点.

I have two outputs configured for Logstash as I need the data to be delivered to two separate Elasticsearch nodes in different locations.

下面是该配置的摘要(在需要的地方已编辑):

A snippet of the configuration is below (redacted where required):

output {
  elasticsearch {
    hosts => [ "https://host1.local:9200" ]
    cacert => '/etc/logstash/config/certs/ca.crt'
    user => XXXXX
    password => XXXXX
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
 }

output {
  elasticsearch {
    hosts => [ "https://host2.local:9200" ]
    cacert => '/etc/logstash/config/certs/ca.crt'
    user => XXXXX
    password => XXXXX
    index => "%{[@metadata][beat]}-%{[@metadata][version]}-%{+YYYY.MM.dd}"
  }
}

在测试过程中，我注意到，如果 host1.local 或 host2.local 这两个ES实例之一不可用，则Logstash无法处理数据或将数据传递到其他，即使它可用.

During testing I've noticed that if one of the ES instances, host1.local or host2.local is unavailable, Logstash fails to process/deliver data to the other, even though it's available.

我是否可以对配置进行修改，以使数据可以传输到可用的Elasticsearch实例，即使其他模块死了?

Is there a modification I can make to the configuration that will allow data to be delivered to the available Elasticsearch instance, even if the other dies?

推荐答案

logstash具有至少一次的交付模型.如果未启用永久队列，则数据可能会丢失在重新启动过程中，否则，logstash会将事件至少一次传递到所有 all 输出.结果，如果一个输出变得不可访问，则队列(内存中或持久性)将备份并阻止处理.您可以将持久性队列和流水线间通信与

logstash has an at-least-once delivery model. If persistent queues are not enabled data can be lost across a restart, but otherwise, logstash will delivery events to all of the outputs at least once. As a result, if one output becomes unreachable the queue (either in-memory or persistent) will back up and block processing. You can use persistent queues and pipeline-to-pipeline communication with an output isolator pattern to avoid stalling one output when another is unavailable.

这篇关于在Logstash中定义多个输出，同时处理Elasticsearch实例的潜在不可用性的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！