从 DynamoDb 查询的 Python 脚本不提供所有项目 [英] Python Script to query from DynamoDb not giving all items

问题描述

我已经编写了以下 python 代码来从表中获取数据，但它没有获取我想要的所有项目.当我查看 DynamoDb 的 AWS 控制台页面时，与从脚本中获得的相比，我可以看到更多的条目.

I have written following python code to fetch data from a table but its not fetching all the items as I want. When I check on AWS console page of DynamoDb, I can see much more entries as compared to what I get from script.

from __future__ import print_function # Python 2/3 compatibility
import boto3
import json
import decimal
from datetime import datetime
from boto3.dynamodb.conditions import Key, Attr
import sys

# Helper class to convert a DynamoDB item to JSON.
class DecimalEncoder(json.JSONEncoder):
    def default(self, o):
        if isinstance(o, decimal.Decimal):
            if o % 1 > 0:
                return float(o)
            else:
                return int(o)
        return super(DecimalEncoder, self).default(o)

dynamodb = boto3.resource('dynamodb', aws_access_key_id = '',
        aws_secret_access_key = '',
        region_name='eu-west-1', endpoint_url="http://dynamodb.eu-west-1.amazonaws.com")

mplaceId = int(sys.argv[1])
table = dynamodb.Table('XYZ')

response = table.query(
    KeyConditionExpression=Key('mplaceId').eq(mplaceId)
)

print('Number of entries found ', len(response['Items']))

我也从 aws 控制台做了同样的事情.按 mplaceId 查询.

I did the same thing from aws console also. Query by mplaceId.

它发生的任何原因?

推荐答案

dynamodb.Table.query() 返回最大 1MB 的数据.来自 boto3文档:

dynamodb.Table.query() returns at max 1MB of data. From the boto3 documentation:

单个 Query 操作将读取最多设置的最大项目数(如果使用 Limit 参数)或最多 1 MB 的数据，然后应用任何使用 FilterExpression 过滤结果.如果响应中存在 LastEvaluatedKey，则需要对结果集进行分页.有关详细信息，请参阅对结果进行分页在 Amazon DynamoDB 开发人员指南中.

A single Query operation will read up to the maximum number of items set (if using the Limit parameter) or a maximum of 1 MB of data and then apply any filtering to the results using FilterExpression. If LastEvaluatedKey is present in the response, you will need to paginate the result set. For more information, see Paginating the Results in the Amazon DynamoDB Developer Guide .

这实际上不是 boto3-限制，而是底层 query-API 的限制.

That's actually no boto3-limitation, but a limitation of the underlying query-API.

您可以使用 boto3 的 内置分页.这是一个示例，展示了 分页器的使用用于查询 boto3 提供的 DynamoDB 表:

Instead of implementing pagination yourself, you can use boto3's built-in pagination . Here is an example showing the use of the paginator for querying DynamoDB tables provided by boto3:

import boto3
from boto3.dynamodb.conditions import Key

dynamodb_client = boto3.client('dynamodb')
paginator = dynamodb_client.get_paginator('query')
page_iterator = paginator.paginate(
    TableName='XYZ',
    KeyConditionExpression='mplaceId = :mplaceId',
    ExpressionAttributeValues={':mplaceId': {'S' : mplaceid}}
)

for page in page_iterator:
    print(page['Items'])

这篇关于从 DynamoDb 查询的 Python 脚本不提供所有项目的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！