更新已使用logstash和filebeat上传的弹性搜索数据 [英] update elasticsearch data that is already uploaded with logstash and filebeat

问题描述

我们有一个问题：我们正在使用带有转发器文件捕获的logstash。我们已经上传了2100万条日志，现在我们在logstash中更改了配置文件。我们不想删除所有的数据并重新加载它，所以我们想知道是否有更新一个字段的方法。但是，我们不想更新每个字段，只有当其中包含特定内容时。
任何人都可以帮助我们，也许给出代码示例？
感谢您的帮助！

we have a question: we are using logstash with the forwarder filebeat. We have already uploaded 21 Million logs and now we have a change in our config file in logstash. We don't want to delete all the data and reload it so we want to know if there's a way to update a field. But we don't want to update each field, only if there is a specific content in it. Can anyone help us and maybe give an code example? Thanks for your help!

    {
  "_index": "logstash-2016.06.06",
  "_type": "log",
  "_id": "4f63b12b098bd5ff02de89e7057347c8ea39ae96",
  "_score": null,
  "_source": {
    "message": "[06/Jun/2016:23:59:58 -0700] \"GET CFNetwork/758.4.3 Darwin/15.5.0\"",
    "@version": "1",
    "@timestamp": "2016-06-06T21:59:58.000Z",
    "type": "log",
    "fields": null,
    "beat": {
      "hostname": "xxx",
      "name": "xxx"
    },
    "source": "xxx",
    "offset": xxx,
    "input_type": "log",
    "count": 1,
    "host": "xxx",
    "iOSVersion": "Unknown",
    "tags": [
      "beats_input_codec_plain_applied"
    ],
    "@uuid": "79e6a34e-13e4-9b5b-467b3a1f04fa",
    "fingerprint": "xxx",
    "logDate": "06/Jun/2016:23:59:58",
    "timezone": "0700",
    "httpRequest": "GET",
    "network": "CFNetwork",
    "CFNetworkNumber": "758.4.3",
    "DarwinVersion": "Darwin",
    "darwinVersionNumber": "15.5.0"
  },
  "fields": {
    "@timestamp": [
      1465250398000
    ]
  },
  "sort": [
    1465250398000
  ]
}

所以我们通过在logstash配置的匹配表中使用CFNetworkNumber和DarwinVersion来获得iOSVersion。在这个例子中，iOSVersion是未知的，因为这个组合不在配置中。所以我想添加一个这种情况，并更新这些文件，我们是未知的，但这个更改后，cas是已知的。那么我在logstash配置中添加的是这样的：

So we get the iOSVersion by using the CFNetworkNumber and the DarwinVersion in a matching table in the logstash config. In this exampe the iOSVersion is unknown because this combination is not in the config yet. So I want to add a this case and update those documents wehre it is unknown but the cas is known after this change. So what I add in the logstash config is this:

else if [darwinVersionNumber] == "15.5.0" { 
    if[CFNetworkNumber] == "758.4.3" {
        mutate{
            gsub => ["iOSVersion", "Unknown", "9.3.2"]
        }
    }
}

推荐答案

如果您在ES 2.x上，您可以使用更新查询端点，以便使用以下查询更新这些文档

If you are on ES 2.x, you can simply use the update by query endpoint in order to update those documents with the following query

POST /logstash-*/_update_by_query
{
  "script": {
    "inline": "ctx._source.iOSVersion = '9.3.2'"
  },
  "query": {
    "bool": {
      "filter": [
        {"term": {"iOSVersion.raw": "Unknown"}},
        {"term": {"darwinVersionNumber.raw": "15.5.0"}},
        {"term": {"CFNetworkNumber.raw": "758.4.3"}}
      ]
    }
  }
}

如果你还在ES 1.x，那么你可以使用按查询更新p lugin 做同样的事情

If you're still on ES 1.x, then you can use the update by query plugin which does the same thing

这篇关于更新已使用logstash和filebeat上传的弹性搜索数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！