将twitter数据索引到elasticsearch中:已超出索引中的总字段数[1000]个限制 [英] indexing twitter data into elasticsearch: Limit of total fields [1000] in index has been exceeded
问题描述
我有一个将Twitter Stream索引到Elasticsearch中的系统。它已经运行了几个星期。
I have a system that indexes the Twitter Stream into Elasticsearch. It has been running for a few weeks now.
最近出现了一个错误,指出:索引[dev_tweets]中总字段[1000]的限制已超出
。
Lately an error has been showed up that says: Limit of total fields [1000] in index [dev_tweets] has been exceeded
.
我想知道是否有人遇到过同样的问题?
I was wondering, if anyone has encountered the same problem?
此外,如果我运行此curl:
In addition if I run this curl:
$ curl -s -XGET http://localhost:9200/dev_tweets/_mapping?pretty | grep type | wc -l
890
应该给我更多或更少的字段数映射。有很多字段,但不超过1000
it should give me more or less the number of fields in the mapping. It is a lot of fields, but it isn't more than 1000
推荐答案
此限制已在以下GitHub 问题。
This limit has been introduced in following GitHub issue.
命令计数 grep类型| wc -l </ code>计算带有文本 type 的行数。因此,我认为计数有可能不准确。我写了一小段文字,得到的值比实际字段数高。这样一来,您得到的字段数也可能少于实际数量,但我还无法想到一种情况。
The command counts grep type | wc -l
counts the number of lines with text "type". Therefore I guess there is a chance for the count to be inaccurate. I did a small text and I got a higher value than the actual number of fields. So you could get less than the actual number of fields as well, but I can't think of a scenario yet.
这是我所做的测试。
curl -s -XGET http://localhost:9200/stackoverflow/_mapping?pretty
{
"stackoverflow" : {
"mappings" : {
"os" : {
"properties" : {
"NAME" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
}
},
"TITLE" : {
"type" : "text",
"fields" : {
"keyword" : {
"type" : "keyword",
"ignore_above" : 256
}
},
"fielddata" : true
},
"title" : {
"type" : "text",
"fielddata" : true
}
}
}
}
}
}
由于 type 出现在5行中,即使我只有3个字段,我的输出也仍然是5。
Since the "type" is there in 5 lines I get the output as 5 even though I only have 3 fields.
您可以尝试增加该限制并查看它是否有效吗?
Can you try increasing the limit and see if it works?
PUT my_index/_settings
{
"index.mapping.total_fields.limit": 2000
}
您也可以在创建索引期间增加此限制。
You can also increase this limit during index creation.
PUT my_index
{
"settings": {
"index.mapping.total_fields.limit": 2000,
"number_of_shards": 1,
"number_of_replicas": 0
},
"mappings": {
...
}
}
信用: https://discuss.elastic .co / t / total-fields-limit-setting / 53004/2
这篇关于将twitter数据索引到elasticsearch中:已超出索引中的总字段数[1000]个限制的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!