尝试使用ElasticSearch输出创建与请求路径值同名的索引时发生无效的FieldReference [英] Invalid FieldReference occurred when attempting to create index with the same name as request path value using ElasticSearch output

问题描述

这是我的 logstash.conf 文件：

input {
    http {
        host => "127.0.0.1"
        port => 31311 
    }
}

filter {
    mutate {
        split => ["%{headers.request_path}", "/"]
        add_field => { "index_id" => "%{headers.request_path[0]}" }
        add_field => { "document_id" => "%{headers.request_path[1]}" }
    }
}

output {
  elasticsearch {
    hosts => "http://localhost:9200"
    index => "%{index_id}"
    document_id => "%{document_id}"
  }
  stdout {
    codec => "rubydebug"
  }
}

当我发送 PUT 请求，例如

C：\Users\BolverkXR\Downloads\curl-7.64 .1-win64-mingw\bin> .\curl.exe
-XPUT' http：/ /127.0.0.1:31311/twitter '

我希望创建一个名称为<$ c的新索引$ c> twitter ，而不是使用ElasticSearch

I want a new index to be created with the name twitter, instead of using the ElasticSearch default.

但是，Logstash立即崩溃，并显示以下（已截断的）错误消息：

However, Logstash crashes immediately with the following (truncated) error message:

pipelineworker中的异常，管道停止处理新的
事件，请检查您的过滤器配置并重新启动Logstash。
org.logstash.FieldReference $ IllegalSyntaxException：无效的
FieldReference： headers.request_path [0]

我确定我在某个地方犯了语法错误，但是我看不到它在哪里。我该如何解决？

I am sure I have made a syntax error somewhere, but I can't see where it is. How can I fix this?

编辑：

我将 filter 段更改为以下内容：

The same error occurs when I change the filter segment to the following:

filter {
    mutate {
        split => ["%{[headers][request_path]}", "/"]
        add_field => { "index_id" => "%{[headers][request_path][0]}" }
        add_field => { "document_id" => "%{[headers][request_path][1]}" }
    }
}

推荐答案

要拆分字段，请使用％{foo} 语法。。另外，您应该从数组的位置[1]开始，因为在位置[0]处将有一个空字符串（ ），原因是没有第一个分隔符（ / ）左侧的字符。相反，您的过滤器部分应该是这样的：

To split the field the %{foo} syntax is not used. Also you should start at position [1] of the array, because in position [0] there will be an empty string("") due to the reason that there are no characters at the left of the first separator(/). Instead, your filter section should be something like this:

 filter {
    mutate {
        split => ["[headers][request_path]", "/"]
        add_field => { "index_id" => "%{[headers][request_path][1]}" }
        add_field => { "document_id" => "%{[headers][request_path][2]}" }
    }
}

您现在可以使用％{index_id} 和％{document_id} 中的值。我使用 logstash 6.5.3 版本对此进行了测试，并使用邮递员发送了<< a href = http://127.0.0.1:31311/twitter/1 rel = nofollow noreferrer> http://127.0.0.1:31311/twitter/1 'HTTP请求，控制台中的输出如下：

You can now use the value in %{index_id} and %{document_id}. I tested this using logstash 6.5.3 version and used Postman to send the 'http://127.0.0.1:31311/twitter/1' HTTP request and the output in console was as follows:

{
        "message" => "",
       "index_id" => "twitter",
    "document_id" => "1",
       "@version" => "1",
           "host" => "127.0.0.1",
     "@timestamp" => 2019-04-09T12:15:47.098Z,
        "headers" => {
             "connection" => "keep-alive",
           "http_version" => "HTTP/1.1",
            "http_accept" => "*/*",
          "cache_control" => "no-cache",
         "content_length" => "0",
          "postman_token" => "cb81754f-6d1c-4e31-ac94-fde50c0fdbf8",
        "accept_encoding" => "gzip, deflate",
           "request_path" => [
            [0] "",
            [1] "twitter",
            [2] "1"
        ],
              "http_host" => "127.0.0.1:31311",
        "http_user_agent" => "PostmanRuntime/7.6.1",
         "request_method" => "PUT"
    }
}

配置的输出部分不更改。因此，最终的logstash.conf文件将如下所示：

The output section of your configuration does not change. So, your final logstash.conf file will be something like this:

input {
    http {
        host => "127.0.0.1"
        port => 31311 
    }
}

filter {
    mutate {
        split => ["[headers][request_path]", "/"]
        add_field => { "index_id" => "%{[headers][request_path][1]}" }
        add_field => { "document_id" => "%{[headers][request_path][2]}" }
    }
}

output {
  elasticsearch {
    hosts => "http://localhost:9200"
    index => "%{index_id}"
    document_id => "%{document_id}"
  }
  stdout {
    codec => "rubydebug"
  }
}

这篇关于尝试使用ElasticSearch输出创建与请求路径值同名的索引时发生无效的FieldReference的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！