无法使用logstash从外部文件加载索引到弹性搜索 [英] Cannot load index to elasticsearch from external file, using logstash

问题描述

我在同一台计算机上并行运行一个弹性实例和一个logstash。

当尝试将文件加载到弹性文件中时，使用正在运行的logstash下面的配置文件，我得到以下输出msgs弹性和没有文件加载
（当输入被配置为stdin一切似乎工作正常）

任何想法？

[2014-06-17 22：42：24,748] [INFO] [集群。服务] [Masked Marvel]删除{[logstash-Eitan-PC-5928-2010] [Ql5fyvEGQyO96R9NIeP32g] [Eitan-PC] [inet [Eitan-PC / 10.0.0.5：9301]] {client = true，data = false} ，}，原因：zen-disco-node_failed（[logstash-Eitan-PC-5928-2010] [Ql5fyvEGQyO96R9NIeP32g] [Eitan-PC] [inet [Eitan-PC / 10.0.0.5：9301]] {client = true，data = false}），原因运输断开（wi 验证连接）

[2014-06-17 22： 43：00,686] [INFO] [cluster.service] [Masked Marvel]添加了{[logstash-Eitan-PC-5292-4014] [m0Tg-fcmTHW9aP6zHe UqTA] [Eitan-PC] [inet [/10.0.0.5:9301]] {client = true，data = false}，}，reason：zen-disco-receive（join from node [[logstash-Eitan-PC-5292在[/ code> et [/10.0.0.5:9301]] {client = true，data = false}]中，[m0Tg-fcmTHW9aP6zHeUqTA] [Eitan-PC]]
p>



配置文件：

 输入{
文件{
 path => c：\testLog.txt
} 
} 
 
 
输出{
 elasticsearch {host => localhost 
 index => amat1 
} 
 
} 
  




解决方案>

当您使用弹性搜索作为输出时 http://logstash.net/ docs / 1.4.1 / outputs / elasticsearch 而不是elasticsearch_http http： //logstash.net/docs/1.4.1/outputs/elasticsearch_http 您将要设置协议。

原因是它可以有3个不同的值，每个都有不同行为的节点，http或传输，默认选择没有很好的记录。

从外观您的日志文件似乎正在尝试使用节点协议，因为我看到端口9301上的连接尝试，其指示（以及其他日志条目）logstash尝试将该集群作为节点加入。这可能会导致任何数量的原因，包括群集名称上的不匹配。

我建议将协议设置为http - 以前更改已经修复了类似的问题。 / p>

另请参见：

http://logstash.net/docs/1.4.1/outputs/elasticsearch#cluster
http://logstash.net/docs/1.4.1/outputs/elasticsearch#protocol

编辑：

我在配置中看到的其他一些问题 -

• 您的主机和索引应该是字符串，在logstash config
  文件中应该用双引号localhost和amat1包装。
  没有报价可以工作，但他们建议您使用报价。

  
  
  

  http://logstash.net/docs/1.4.1/configuration#string




• 如果你不要使用http作为协议，也不要使用
  elasticsearch_http作为输出，您应该将集群设置为
  您的ES集群名称（因为它将试图成为
  集群）。




• 您应该将start_position设置为文件中的开始。
  否则它将默认从文件的末尾读取，而
  将看不到任何数据。这是Windows的一个特别的问题，现在
  作为跟踪文件中位置的另一种方式，sincedb是
  在Windows上断开：

  
  
  

  < a href =https://logstash.jira.com/browse/LOGSTASH-1587 =nofollow> https://logstash.jira.com/browse/LOGSTASH-1587

  
  
  

  http://logstash.net/docs /1.4.1/inputs/file#start_position




• 您应该将您的日志文件路径更改为：
  C：/testLog.txt。 Logstash更喜欢Windows下的斜杠和大写
  驱动器号。

  
  
  

  https：//logstash.jira。 com / browse / LOGSTASH-430

i am running one instance of elastic and one of logstash in parallel on the same computer.

when trying to load a file into elastic, using logstash that is running the config file below, i get the follwing output msgs on elastic and no file is loaded (when input is configured to be stdin everything seems to be working just fine)

any ideas?

" [2014-06-17 22:42:24,748][INFO ][cluster.service ] [Masked Marvel] removed {[logstash- Eitan-PC-5928-2010][Ql5fyvEGQyO96R9NIeP32g][Eitan-PC][inet[Eitan-PC/10.0.0.5:9301]]{client=true, data=false},}, reason: zen-disco-node_failed([logstash-Eitan-PC-5928-2010][Ql5fyvEGQyO96R9NIeP32g][Eitan-PC][inet[Eitan-PC/10.0.0.5:9301]]{client=true, data=false}), reason transport disconnected (with verified connect)

[2014-06-17 22:43:00,686][INFO ][cluster.service ] [Masked Marvel] added {[logstash-Eitan-PC-5292-4014][m0Tg-fcmTHW9aP6zHeUqTA][Eitan-PC][inet[/10.0.0.5:9301]]{client=true, data=false},}, reason: zen-disco-receive(join from node[[logstash-Eitan-PC-5292-4014][m0Tg-fcmTHW9aP6zHeUqTA][Eitan-PC][inet[/10.0.0.5:9301]]{client=true, data=false}]) "

config file:

    input { 
            file {
                path => "c:\testLog.txt"
            }
        } 


    output {
        elasticsearch { host => localhost  
                index=> amat1
                 }

}

解决方案

When you use "elasticsearch" as your output http://logstash.net/docs/1.4.1/outputs/elasticsearch as opposed to "elasticsearch_http" http://logstash.net/docs/1.4.1/outputs/elasticsearch_http you are going to want to set "protocol".

The reason is that it can have 3 different values, "node", "http" or "transport" with different behavior for each and the default selection is not well documented.

From the look of your log files it appears it's trying to use "node" protocol as I see connection attempts on port 9301 which indicates (along with other log entries) that logstash is trying to join the cluster as a node. This can fail for any number of reasons including mismatch on the cluster name.

I'd suggest setting protocol to "http" - that change has fixed similar issues before.

See also:

http://logstash.net/docs/1.4.1/outputs/elasticsearch#cluster http://logstash.net/docs/1.4.1/outputs/elasticsearch#protocol

EDIT:

A few other issues I see in your config -

• Your host and index should be strings, which in a logstash config file should be wrapped with double quotes, "localhost" and "amat1". No quotes may work but they recommend you use quotes.

  http://logstash.net/docs/1.4.1/configuration#string

• If you don't use "http" as the protocol or don't use "elasticsearch_http" as the output you should set cluster equal to your ES cluster name (as it will be trying to become a node of the cluster).

• You should set start_position under file in input to "beginning". Otherwise it will default to reading from the end of the file and you won't see any data. This a particular problem with Windows right now as the other way of tracking position within a file, sincedb, is broken on Windows:

  https://logstash.jira.com/browse/LOGSTASH-1587

  http://logstash.net/docs/1.4.1/inputs/file#start_position

• You should change your path to your log file to this: "C:/testLog.txt". Logstash prefers forward slashes and upper case drive letters under Windows.

  https://logstash.jira.com/browse/LOGSTASH-430

这篇关于无法使用logstash从外部文件加载索引到弹性搜索的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！