弹性搜索未提供大量的页面大小数据 [英] Elastic search not giving data with big number for page size

问题描述

要获取的数据大小：大约20,000

Size of data to get: 20,000 approx

问题：在python中使用以下命令搜索Elastic Search索引数据

Issue: searching Elastic Search indexed data using below command in python

但没有返回任何结果。

from pyelasticsearch import ElasticSearch
es_repo = ElasticSearch(settings.ES_INDEX_URL)
search_results = es_repo.search(
            query, index=advertiser_name, es_from=_from, size=_size)

如果我给的尺寸小于或等于10,000，可以正常工作，但不能与20,000相匹配
请帮助我找到最佳的解决方案。

If I give size less than or equal to 10,000 it works fine but not with 20,000 Please help me find an optimal solution to this.

PS：在深入研究ES时发现以下消息错误：

PS: On digging deeper into ES found this message error:

结果窗口太大， +号必须小于或等于：[10000]，但等于[19999]。请参见滚动API，以获取请求大型数据集的更有效方法。

Result window is too large, from + size must be less than or equal to: [10000] but was [19999]. See the scrolling API for a more efficient way to request large data sets.

推荐答案

实时使用最好的解决方案是使用查询后搜索 。您只需要一个日期字段和另一个唯一标识文档的字段- _id 字段或 _uid 领域。
尝试类似的操作，在我的示例中，我想提取属于一个用户的所有文档-在我的示例中，用户字段具有关键字数据类型 ：

for real time use the best solution is to use the search after query . You need only a date field, and another field that uniquely identify a doc - it's enough a _id field or an _uid field. Try something like this, in my example I would like to extract all the documents that belongs to a single user - in my example the user field has a keyword datatype:

from elasticsearch import Elasticsearch


es = Elasticsearch()
es_index = "your_index_name"
documento = "your_doc_type"

user = "Francesco Totti"

body2 = {
        "query": {
        "term" : { "user" : user } 
            }
        }

res = es.count(index=es_index, doc_type=documento, body= body2)
size = res['count']


body = { "size": 10,
            "query": {
                "term" : {
                    "user" : user
                }
            },
            "sort": [
                {"date": "asc"},
                {"_uid": "desc"}
            ]
        }

result = es.search(index=es_index, doc_type=documento, body= body)
bookmark = [result['hits']['hits'][-1]['sort'][0], str(result['hits']['hits'][-1]['sort'][1]) ]

body1 = {"size": 10,
            "query": {
                "term" : {
                    "user" : user
                }
            },
            "search_after": bookmark,
            "sort": [
                {"date": "asc"},
                {"_uid": "desc"}
            ]
        }




while len(result['hits']['hits']) < size:
    res =es.search(index=es_index, doc_type=documento, body= body1)
    for el in res['hits']['hits']:
        result['hits']['hits'].append( el )
    bookmark = [res['hits']['hits'][-1]['sort'][0], str(result['hits']['hits'][-1]['sort'][1]) ]
    body1 = {"size": 10,
            "query": {
                "term" : {
                    "user" : user
                }
            },
            "search_after": bookmark,
            "sort": [
                {"date": "asc"},
                {"_uid": "desc"}
            ]
        }

然后，您将找到附加到结果 var

Then you will find all the doc appended to the result var

的所有文档。喜欢使用滚动查询-doc 此处：

If you would like to use scroll query - doc here:

from elasticsearch import Elasticsearch, helpers

es = Elasticsearch()
es_index = "your_index_name"
documento = "your_doc_type"

user = "Francesco Totti"

body = {
        "query": {
        "term" : { "user" : user } 
             }
        }

res = helpers.scan(
                client = es,
                scroll = '2m',
                query = body, 
                index = es_index)

for i in res:
    print(i)

这篇关于弹性搜索未提供大量的页面大小数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！