将twitter数据索引到elasticsearch中：已超出索引中的总字段数[1000]个限制 [英] indexing twitter data into elasticsearch: Limit of total fields [1000] in index has been exceeded

问题描述

我有一个将Twitter Stream索引到Elasticsearch中的系统。它已经运行了几个星期。

I have a system that indexes the Twitter Stream into Elasticsearch. It has been running for a few weeks now.

最近出现了一个错误，指出：索引[dev_tweets]中总字段[1000]的限制已超出。

Lately an error has been showed up that says: Limit of total fields [1000] in index [dev_tweets] has been exceeded.

我想知道是否有人遇到过同样的问题？

I was wondering, if anyone has encountered the same problem?

此外，如果我运行此curl：

In addition if I run this curl:

$ curl -s -XGET http://localhost:9200/dev_tweets/_mapping?pretty | grep type | wc -l
     890

应该给我更多或更少的字段数映射。有很多字段，但不超过1000

it should give me more or less the number of fields in the mapping. It is a lot of fields, but it isn't more than 1000

推荐答案

此限制已在以下GitHub 问题。

This limit has been introduced in following GitHub issue.

命令计数 grep类型| wc -l <​​/ code>计算带有文本 type 的行数。因此，我认为计数有可能不准确。我写了一小段文字，得到的值比实际字段数高。这样一来，您得到的字段数也可能少于实际数量，但我还无法想到一种情况。

The command counts grep type | wc -l counts the number of lines with text "type". Therefore I guess there is a chance for the count to be inaccurate. I did a small text and I got a higher value than the actual number of fields. So you could get less than the actual number of fields as well, but I can't think of a scenario yet.

这是我所做的测试。

curl -s -XGET http://localhost:9200/stackoverflow/_mapping?pretty

{
  "stackoverflow" : {
    "mappings" : {
      "os" : {
        "properties" : {
          "NAME" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            }
          },
          "TITLE" : {
            "type" : "text",
            "fields" : {
              "keyword" : {
                "type" : "keyword",
                "ignore_above" : 256
              }
            },
            "fielddata" : true
          },
          "title" : {
            "type" : "text",
            "fielddata" : true
          }
        }
      }
    }
  }
}

由于 type 出现在5行中，即使我只有3个字段，我的输出也仍然是5。

Since the "type" is there in 5 lines I get the output as 5 even though I only have 3 fields.

您可以尝试增加该限制并查看它是否有效吗？

Can you try increasing the limit and see if it works?

PUT my_index/_settings
{
  "index.mapping.total_fields.limit": 2000
}

您也可以在创建索引期间增加此限制。

You can also increase this limit during index creation.

PUT my_index
{
  "settings": {
    "index.mapping.total_fields.limit": 2000,
    "number_of_shards": 1,
    "number_of_replicas": 0
  },
  "mappings": {
    ...
  }
}

信用： https：//discuss.elastic .co / t / total-fields-limit-setting / 53004/2

这篇关于将twitter数据索引到elasticsearch中：已超出索引中的总字段数[1000]个限制的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！