如何阻止logstash创建ElasticSearch中的默认映射 [英] How to stop logstash from creating a default mapping in ElasticSearch
问题描述
我正在使用logstash将日志提供给ElasticSearch。
我正在配置logstash输出：
input {
file {
path => ; /tmp/foo.log
codec =>
plain {
format => ％{message}
}
}
}
输出{
elasticsearch {
#host => localhost
codec => json {}
manage_template => false
index => 4glogs
}
}
我注意到，一旦我开始logstash它在ES中创建一个映射（日志），如下所示。
{
4glogs：{
mappings：{
logs：{
properties：{
@timestamp：{
type：date，
format：dateOptionalTime
}，
@version：{
type：string
}，
message：{
type：string
}
}
}
}
}
}
如何防止logstash创建此映射？
更新：新评新新p新新新旗新新旗新新旗旗新新旗新新旗新新旗新新旗新新旗旗新新旗新新旗新新旗旗新新旗新新旗新新旗旗规规200 Chanol旗新而新新旗新新旗新新旗新新旗新新旗新新旗新200新新旗新新旗新新旗新新200新新旗旗规规200 Chan
作为对象的对象映射尝试解析为对象，但获得了EOF，具有具体的值？ 200新200新200新200新200新200新200新200新200新200新新200新200新新200新新200新新200新新200新新200新新200新新200新新200新新新200新新200新新200新新200新新200新Version旗新新旗新旗在我的情况下，我已经定义了一个type：nested的映射，但是logstash的输出是一个字符串。
所以我从我的logstash配置中删除了所有的编解码器（不管是json还是普通的），并允许json文件通过而不会改变。
这是我的新的logstash配置（包含多行日志的一些其他过滤器）。
input {
kafka {
zk_connect = > localhost：2181
group_id => logstash_group
topic_id => platform-logger
reset_beginning => false
consumer_threads => 1
queue_size => 2000
consumer_id =>新新旗200 200 200 200 200 200 200 200 200 200 200:45新新旗旗哨旗新新新新新旗新新200新新旗新新旗新新200新新旗新新旗新新200新新旗新新旗新新款： 1048576
}
文件{
path => /tmp/foo.log
}
}
过滤器{
multiline {
pattern => ^ \s
what => previous
}
multiline {
pattern => [0-9] + $
what => previous
}
multiline {
pattern => ^ $
what =>新新p新200新新新新旗新新旗新新旗旗新1992新新新旗新新旗200新新新新旗新新旗200新新新新旗新新名： [kafka]
remove_field => [@version]
remove_field => [@timestamp]
remove_tag => [多行]
}
}
输出{
elasticsearch {
manage_template => false
index => 4glogs
}
}
p>您将需要一个映射来存储Elasticsearch中的数据并在其上进行搜索 - 这就是ES知道如何索引和搜索这些内容类型。您可以让logstash动态创建它，也可以阻止它自动创建。
请记住，您不能更改现有的映射（尽管可以添加给他们）。所以首先你需要删除现有的索引。然后，您将修改设置以防止动态映射创建。同时，您将需要创建自己的映射。
例如，这将创建logstash数据的映射，但也可以通过strict来限制任何动态映射创建：
$ curl -XPUT'http：// localhost：9200 / 4glogs / logs / _mapping'-d'
{
logs：{
dynamic：strict，
properties：{
@timestamp：{
type ：date，
format：dateOptionalTime
}，
@version：{
type：string
}，
message：{
type：string
}
}
}
}
'
请记住，索引名称4glogs和类型logs需要匹配来自logstash的内容。 p>
对于我的生产系统，我通常更喜欢关闭动态mapp因为它避免了意外的映射创建。
如果要对动态映射进行调整，以下链接应该很有用：
https：//www.elastic。 co / guide / en / elasticsearch / guide / current / dynamic-mapping.html
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/custom-dynamic-mapping .html
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/dynamic-mapping.html
I am using logstash to feed logs into ElasticSearch. I am configuring logstash output as:
input {
file {
path => "/tmp/foo.log"
codec =>
plain {
format => "%{message}"
}
}
}
output {
elasticsearch {
#host => localhost
codec => json {}
manage_template => false
index => "4glogs"
}
}
I notice that as soon as I start logstash it creates a mapping ( logs ) in ES as below.
{
"4glogs": {
"mappings": {
"logs": {
"properties": {
"@timestamp": {
"type": "date",
"format": "dateOptionalTime"
},
"@version": {
"type": "string"
},
"message": {
"type": "string"
}
}
}
}
}
}
How can I prevent logstash from creating this mapping ?
UPDATE:
I have now resolved this error too. "object mapping for [logs] tried to parse as object, but got EOF, has a concrete value been provided to it?"
As John Petrone has stated below, once you define a mapping, you have to ensure that your documents conform to the mapping. In my case, I had defined a mapping of "type: nested" but the output from logstash was a string. So I removed all codecs ( whether json or plain ) from my logstash config and that allowed the json document to pass through without changes.
Here is my new logstash config ( with some additional filters for multiline logs ).
input {
kafka {
zk_connect => "localhost:2181"
group_id => "logstash_group"
topic_id => "platform-logger"
reset_beginning => false
consumer_threads => 1
queue_size => 2000
consumer_id => "logstash-1"
fetch_message_max_bytes => 1048576
}
file {
path => "/tmp/foo.log"
}
}
filter {
multiline {
pattern => "^\s"
what => "previous"
}
multiline {
pattern => "[0-9]+$"
what => "previous"
}
multiline {
pattern => "^$"
what => "previous"
}
mutate{
remove_field => ["kafka"]
remove_field => ["@version"]
remove_field => ["@timestamp"]
remove_tag => ["multiline"]
}
}
output {
elasticsearch {
manage_template => false
index => "4glogs"
}
}
You will need a mapping to store data in Elasticsearch and to search on it - that's how ES knows how to index and search those content types. You can either let logstash create it dynamically or you can prevent it from doing so and instead create it manually.
Keep in mind you cannot change existing mappings (although you can add to them). So first off you will need to delete the existing index. You would then modify your settings to prevent dynamic mapping creation. At the same time you will want to create your own mapping.
For example, this will create the mappings for the logstash data but also restrict any dynamic mapping creation via "strict":
$ curl -XPUT 'http://localhost:9200/4glogs/logs/_mapping' -d '
{
"logs" : {
"dynamic": "strict",
"properties" : {
"@timestamp": {
"type": "date",
"format": "dateOptionalTime"
},
"@version": {
"type": "string"
},
"message": {
"type": "string"
}
}
}
}
'
Keep in mind that the index name "4glogs" and the type "logs" need to match what is coming from logstash.
For my production systems I generally prefer to turn off dynamic mapping as it avoids accidental mapping creation.
The following links should be useful if you want to make adjustments to your dynamic mappings:
https://www.elastic.co/guide/en/elasticsearch/guide/current/dynamic-mapping.html
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/custom-dynamic-mapping.html
http://www.elasticsearch.org/guide/en/elasticsearch/guide/current/dynamic-mapping.html
这篇关于如何阻止logstash创建ElasticSearch中的默认映射的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！
