我可以在ids筛选器或一般的查询子句中指定的值的最大限制？ [英] Max limit on the number of values I can specify in the ids filter or generally query clause?

问题描述

在弹性搜索中，最大限制是指定可执行匹配的值的值的值？我读到某处是1024，但也是可配置的。真的吗？它如何影响性能？

  curl -XPOST'localhost：9200 / my_index / _search？pretty'-d'{
query {
filtered：{
filter：{
not：{
ids：{
type：my_type，
values：[1，2，3] 
}}}}}}'
  

我可以在这个数组中指定多少值？有什么限制？如果可配置性能对增加限制有什么影响？

解决方案

我不认为Elaticsearch或Lucene明确规定了任何限制。您可能会遇到的限制是JDK中的一个限制。

为了证明我上面的说明，我查看了Elasticsearch的源代码：当请求进入

• /src/main/java/org/elasticsearch/index/query/IdsFilterParser.java#L55rel =nofollow noreferrer>有一个解析器来解析ids的数组。所有它使用的是一个 ArrayList 。然后将其传递给Lucene，而Lucene则依次使用列表。




• 这是Lucene TermsFilter类（行＃84），它从List中的Elasticsearch获取IDS列表。




• 源代码 ArrayList class from Oracle JDK 1.7.0_67：

  / ** 
 *要分配的数组的最大大小。 
 *某些虚拟机在阵列中保留一些头文字。 
 *尝试分配更大的数组可能会导致
 * OutOfMemoryError：请求的数组大小超过VM限制
 * / 
 private static final int MAX_ARRAY_SIZE = Integer.MAX_VALUE  -  8; 
 
 / ** 
 *增加容量，以确保它至少可以容纳最小容量参数指定的
 *元素数。 
 * 
 * @param minCapacity所需的最小容量
 * / 
 private void grow（int minCapacity）{
 ... 
 if（newCapacity  - MAX_ARRAY_SIZE> 0）
 newCapacity = hugeCapacity（minCapacity）; 
 ... 
} 
 
 private static int hugeCapacity（int minCapacity）{
 if（minCapacity< 0）// overflow 
 throw new OutOfMemoryError （）; 
 return（minCapacity> MAX_ARRAY_SIZE）？ 
 Integer.MAX_VALUE：
 MAX_ARRAY_SIZE; 
} 
  

那个数字（ Integer.MAX_VALUE - 8 ）是 2147483639 。所以，这将是该数组的理论最大大小。

我在我的ES实例中测试了一个150000个元素的数组。这里有性能影响：当然，你会得到一个降级的性能，数组越大。在我用150k ids的简单测试中，我有一个800 ms的执行时间。但是，这一切都取决于CPU，内存，加载，数据化，数据映射等。最好的方法是让您实际测试。

UPDATED Dec. 2016 ：此回答适用于2014年底存在的Elasticsearch版本，即1.x分支。当时最新的是1.4.x。

In elasticsearch what is the max limit to specify the value in the number of values a match can be performed on? I read somewhere that it is 1024 but is also configurable. Is that true? And how does it affect the performance?

curl -XPOST 'localhost:9200/my_index/_search?pretty' -d '{
  "query": {
    "filtered": {
      "filter": {
        "not": {
          "ids": {
            "type": "my_type",
            "values": ["1", "2", "3"]
}}}}}}'

How many values can I specify in this array ? What is the limit? If it is configurable what is the performance impact on increasing the limit?

解决方案

I don't think there is any limit set by Elaticsearch or Lucene explicitly. The limit you might hit, though, is the one set in place by the JDK.

To prove my statement above, I looked at the source code of Elasticsearch:

• when the request comes in there is a parser that parses the array of ids. All it's using is an ArrayList. This is then passed along to Lucene, which in turn it's using a List.

• this is the Lucene TermsFilter class (line #84) that gets the list of IDS from Elasticsearch within a List.

• source code of ArrayList class from Oracle JDK 1.7.0_67:

/**
 * The maximum size of array to allocate.
 * Some VMs reserve some header words in an array.
 * Attempts to allocate larger arrays may result in
 * OutOfMemoryError: Requested array size exceeds VM limit
 */
private static final int MAX_ARRAY_SIZE = Integer.MAX_VALUE - 8;   

/**
 * Increases the capacity to ensure that it can hold at least the
 * number of elements specified by the minimum capacity argument.
 *
 * @param minCapacity the desired minimum capacity
 */
private void grow(int minCapacity) {
    ...
    if (newCapacity - MAX_ARRAY_SIZE > 0)
        newCapacity = hugeCapacity(minCapacity);
    ...
}

private static int hugeCapacity(int minCapacity) {
    if (minCapacity < 0) // overflow
        throw new OutOfMemoryError();
    return (minCapacity > MAX_ARRAY_SIZE) ?
        Integer.MAX_VALUE :
        MAX_ARRAY_SIZE;
}

And that number (Integer.MAX_VALUE - 8) is 2147483639. So, this would be the theoretical max size of that array.

I've tested locally in my ES instance an array of 150000 elements. And here comes the performance implications: of course, you would get a degrading performance the larger the array gets. In my simple test with 150k ids I got a 800 ms execution time. But, all depends on CPU, memory, load, datasize, data mapping etc etc. The best would be for you to actually test this.

UPDATED Dec. 2016: this answer applies for the Elasticsearch version in existence at the end of 2014, ie in the 1.x branch. The latest available at that time was 1.4.x.

这篇关于我可以在ids筛选器或一般的查询子句中指定的值的最大限制？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！