ElasticSearch查询到 pandas 数据框 [英] ElasticSearch query to pandas dataframe

问题描述

我有一个查询:

s = Search(using=client, index='myindex', doc_type='mytype')
s.query = Q('bool', must=[Q('match', BusinessUnit=bunit),
                          Q('range', **dicdate)])

res = s.execute()

返回627033行，我想将此字典转换为627033行的数据帧

return me 627033 lines, I want to convert this dictionary in a dataframe with 627033 lines

推荐答案

如果您的请求可能会从Elasticsearch返回10,000多个文档，则需要使用Elasticsearch的滚动功能.很难找到此功能的文档和示例，因此，我将为您提供完整的示例:

If your request is likely to return more than 10,000 documents from Elasticsearch, you will need to use the scrolling function of Elasticsearch. Documentation and examples for this function are rather difficult to find, so I will provide you with a full, working example:

import pandas as pd
from elasticsearch import Elasticsearch
import elasticsearch.helpers


es = Elasticsearch('127.0.0.1',
        http_auth=('my_username', 'my_password'),
        port=9200)

body={"query": {"match_all": {}}}
results = elasticsearch.helpers.scan(es, query=body, index="my_index")
df = pd.DataFrame.from_dict([document['_source'] for document in results])

只需编辑以"my_"开头的字段即可对应您自己的值

Simply edit the fields that start with "my_" to correspond to your own values

这篇关于ElasticSearch查询到 pandas 数据框的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！