Filebeat日期字段映射为type关键字 [英] Filebeat date field mapped as type keyword

问题描述

Filebeat正在从文件中读取日志，其中日志的格式如下:

Filebeat is reading logs from a file, where logs are in the following format:

{"logTimestamp":"2019-11-29T16:39:43.027Z","@version":"1","message":"Hello world","logger_name":"se.lolotron.App","thread_name":"thread-1","level":"INFO","level_value":40000,"application":"my-app"}

因此，存在一个以 ISO 8601 时间格式记录的字段 logTimestamp . 问题在于该字段在Elasticsearch filebeat索引中被映射为关键字

So there is a field logTimestamp logged in ISO 8601 time format. The problem is that this field is mapped as a keyword In Elasticsearch filebeat index

"logTimestamp": {
    "type": "keyword",
    "ignore_above": 1024
},

另一方面，如果我在相同的Elasticsearch实例中为相似的文档建立了索引，但索引不同，例如

On the other hand if I index a similar document in the same Elasticsearch instance but different index, e.g.

POST /new_index/_doc/
{
    "message": "hello world",
    "logTimestamp":"2019-11-29T16:39:43.027Z"
}

映射是

"logTimestamp": {
     "type": "date"
},

根据文档 和

According to docs here and here by default Elastic should detect a date if formatted with strict_date_optional_time. And strict_date_optional_time is described as

通用ISO日期时间解析器，其中日期为必填项，时间为必填项 是可选的.

A generic ISO datetime parser where the date is mandatory and the time is optional.

我假定是ISO 8601，并认为我在上面的示例中将新文档索引到new_index证明了这一点.

Which I presume is ISO 8601 and think I proved that with indexing a new doc to new_index in the example above.

在Filebeat的情况下，为什么logTimestamp保存为关键字?有什么想法吗?

Why is logTimestamp saved as keyword in the case of Filebeat? Any ideas?

我正在使用Filbeat 7.2.1，Elasticsearch 7.2.1. 还使用默认的 fields.yml .

I'm using Filbeat 7.2.1, Elasticsearch 7.2.1. Also the default fields.yml is used.

推荐答案

我刚刚发现默认情况下，filebeat索引已禁用 date_detection (Filebeat版本7.2.1). 可以在此处

I just found out that date_detection is disabled for filebeat indices by default (Filebeat version 7.2.1). This can be seen here

var (
    // Defaults used in the template
    defaultDateDetection         = false
    ...

看起来好像不能被覆盖.

Does not look like it can be overridden.

此问题的解决方法是使用实​​验性功能 append_fields (至少在撰写本文时为实验性功能.请参见

The workaround for this is to use experimental feature append_fields (experimental at least at the time of writing this post. See here for more.) and add the following to the filebeat.yml config

setup.template.overwrite: true
setup.template.append_fields:
- name: logTimestamp
  type: date

这将确保logTimestamp的映射是日期.

This will make sure that the mapping for logTimestamp is date.

这篇关于Filebeat日期字段映射为type关键字的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！