引导索引设置为写入索引，但日志将写入旧索引 [英] Bootstrapped index is set as the write index but logs are getting written to old index

问题描述

容器日志在Kubernetes上运行Elastic+Fluentbit+Kibana堆栈，基于日期(NEW-YYYY-MM-DD)的每日翻转正常，但在大容量时会导致分片过大问题，所以创建了下文提到的ILM策略，以便快速翻转。引导索引是可写的，但仍在写入旧索引(NEW-YYYY-MM-DD)，而不是新索引NEW-YYYY-MM-DD-000001。我已经提过尝试过的事情，但还没有成功。

Created new policy with following condition:

PUT /_ilm/policy/new_policy
{
  "policy": {
    "phases": {
      "hot": {
        "actions": {
          "rollover": {
            "max_age" : "10m" #just to test faster, actually want to set it to an hour.
          }
        }
      },
      "delete": {
        "min_age": "20d",
        "actions": {
          "delete": {}
        }
      }
    }
  }
}

Created template:

PUT _template/new_template
{
  "index_patterns": ["new*"], 
  "settings": {
    "number_of_shards": 1,
    "number_of_replicas": 1,
    "index.lifecycle.name": "new_policy",
    "index.lifecycle.rollover_alias": "new-alias" 
  }
}

Bootstrapped new index which is creating and rolling over correctly.

PUT /%3Cnew-%7Bnow%2Fd%7D-000001%3E
{
  "aliases": {
    "new-alias":{
      "is_write_index": true 
    }
  }
}

Output plugin section inside fluentbit-configmap

[OUTPUT]
    Name            es
    Match           *
    Host            ${FLUENT_ELASTICSEARCH_HOST}
    Port            ${FLUENT_ELASTICSEARCH_PORT}
    HTTP_User       ${FLUENT_ELASTICSEARCH_USER}
    HTTP_Passwd     ${FLUENT_ELASTICSEARCH_PASSWORD}
    Logstash_Format On
    Logstash_Prefix new
    Trace_Error     On
    Replace_Dots    On
    Retry_Limit     False
    tls             On
    tls.verify      Off

Tried indexing but no luck.

POST _reindex
{
  "source": {
    "index": "new-2021.09.30" 
  },
  "dest": {
    "index": "new-2021.09.30-000001", 
    "op_type": "create" 
  }
}

另外，我还尝试创建了整个新的索引模式，但它似乎来自任何系统默认值，并且没有将我的ILM索引设为新日志的默认值。

alias           index                       filter routing.index routing.search is_write_index
new-alias new-2021.09.30-000001 -      -             -              false
new-alias new-2021.09.30-000002 -      -             -              false
new-alias new-2021.09.30-000003 -      -             -              false
new-alias new-2021.09.30-000004 -      -             -              false
new-alias new-2021.09.30-000005 -      -             -              false
new-alias new-2021.09.30-000006 -      -             -              false
new-alias new-2021.09.30-000007 -      -             -              false
new-alias new-2021.09.30-000008 -      -             -              false
new-alias new-2021.09.30-000009 -      -             -              false
new-alias new-2021.09.30-000010 -      -             -              false
new-alias new-2021.09.30-000011 -      -             -              false
new-alias new-2021.09.30-000012 -      -             -              false
new-alias new-2021.09.30-000013 -      -             -              true

推荐答案

在Fluentbit配置中，您需要更改以下设置：

Logstash_Format Off
Index new-alias

并删除Logstash_Prefix new

就是这样。由于ILM将负责命名索引并将其滚动，因此您只需写入new-alias，而ILM将负责睡觉。不再需要Fluentbit组成索引名。

这篇关于引导索引设置为写入索引，但日志将写入旧索引的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！