如何在dynamodb中超出1mb的扫描数据限制 [英] How to exceed the limit of scan data for 1mb in dynamodb

问题描述

我将dynamodb与nodejs一起使用，有3000条记录，并且在代码中写入60多个段，每个段扫描1mb的数据并将结果显示到60+的1mb限制的段中.因此，请提供解决方案如何一步一步地获取3000条记录扫描，这意味着要分段.请尽快提供解决方案，因为我对项目感到震惊.请帮我.下面是我的代码.

I am using dynamodb with nodejs, I am having 3000 records, and I am writing 60+ segments in the code, each segment scanning the data of 1mb and displaying the results to 60+ segments of 1 mb limit. So please provide the solution how to get the 3000 records scan in single step that means in one segment. Please provide the solution quickly because i am strucked in the middle of my project. Please help me. Below is my code.

var AWS = require("aws-sdk");
var async = require("async");

    AWS.config.update({
      region: "---",
      endpoint: "-----------",
      accessKeyId: "-----------------",
      secretAccessKey:"----------"
    });


var db = new AWS.DynamoDB.DocumentClient()

var table = "rets_property_all";
var pstart =new Date () .getTime ();

async.parallel({
            0 : function(callback){
        db.scan ({TableName: table,
          ProjectionExpression: "#cityname,ListingKey ",
          FilterExpression: "#cityname = :v_id",
          ExpressionAttributeNames: {
              "#cityname": "CityName",
          },
          ExpressionAttributeValues: {":v_id" : 'BALTIMORE'},
            TotalSegments: 63,
            Segment:  0//by the worker who has been called
        },function (err , res) {
            callback (null , res.Items);
        });
            },
            1 : function(callback){
        db.scan ({TableName: table,
          ProjectionExpression: "#cityname,ListingKey ",
          FilterExpression: "#cityname = :v_id",
          ExpressionAttributeNames: {
              "#cityname": "CityName",
          },
          ExpressionAttributeValues: {":v_id" : 'BALTIMORE'},
            TotalSegments: 63,
            Segment: 1//by the worker who has been called
        }, function (err , res) {
            callback (null , res.Items);
        });
            },
      2 : function(callback){
        db.scan ({TableName: table,
          ProjectionExpression: "#cityname,ListingKey ",
          FilterExpression: "#cityname = :v_id",
          ExpressionAttributeNames: {
              "#cityname": "CityName",
          },
          ExpressionAttributeValues: {":v_id" : 'BALTIMORE'},
            TotalSegments: 63,
            Segment: 2//by the worker who has been called
        }, function (err , res) {
            callback (null , res.Items);
        });
            },
--------
---------
------

      62 : function(callback){
        db.scan ({TableName: table,
          ProjectionExpression: "#cityname,ListingKey ",
          FilterExpression: "#cityname = :v_id",
          ExpressionAttributeNames: {
              "#cityname": "CityName",
          },
          ExpressionAttributeValues: {":v_id" : 'BALTIMORE'},
            TotalSegments: 63,
            Segment: 62//by the worker who has been called
        }, function (err , res) {
            callback (null , res.Items);
        });
      },

        },function(err,results){
      if (err) {throw err; }
      var pend = new Date () .getTime ();

      console.log (results);
        })

推荐答案

实际上，无法覆盖扫描的 1 MB限制.这是DynamoDB设计限制，不能被任何API覆盖.您可能需要了解架构或AWS服务设计的局限性.

Actually, there is no way to override the 1 MB limit of scan. This is the DynamoDB design restriction and can't be overridden by any API. You may need to understand the limitation of the architecture or AWS service design.

您可以在后续扫描中使用 LastEvaluatedKey 从先前扫描结束的地方开始.

You can use LastEvaluatedKey on the subsequent scans to start from where the previous scan ended.

扫描"的结果集限制为每个呼叫1 MB.您可以使用扫描响应中的LastEvaluatedKey来检索更多结果.

The result set from a Scan is limited to 1 MB per call. You can use the LastEvaluatedKey from the scan response to retrieve more results.

用例尚不清楚，为什么要一次扫描所有3000条记录.即使您有特定的用例，也无法在DynamoDB扫描中实现.

The use case is unclear why you wanted to get all 3000 records in one scan. Even, if you have a specific use case, simply it can't be achieved on DynamoDB scan.

即使在关系数据库中，您也可以获取光标并进行迭代以依次获取所有记录.同样，在DynamoDB中，您必须递归使用Scan，直到 LastEvaluatedKey 为空.

Even, in relation database, you get the cursor and go through the iterations to get all the records sequentially. Similarly, in DynamoDB you have to use Scan recursively until LastEvaluatedKey is null.

递归地进行单次扫描的示例代码，直到LastEvaluatedKey为null:-

var docClient = new AWS.DynamoDB.DocumentClient();
var params = {
    TableName: table,
    ProjectionExpression: "#cityname,ListingKey ",
    FilterExpression: "#cityname = :v_id",
    ExpressionAttributeNames: {
        "#cityname": "CityName",
    },
    ExpressionAttributeValues: { ":v_id": 'BALTIMORE' }
};

docClient.scan(params, onScan);

function onScan(err, data) {
    if (err) {
        console.error("Unable to scan the table. Error JSON:", JSON.stringify(err, null, 2));
    } else {
        // print all the movies
        console.log("Scan succeeded.");
        data.Items.forEach(function (movie) {
            console.log("Item :", JSON.stringify(movie));
        });

        // continue scanning if we have more movies
        if (typeof data.LastEvaluatedKey != "undefined") {
            console.log("Scanning for more...");
            params.ExclusiveStartKey = data.LastEvaluatedKey;
            docClient.scan(params, onScan);
        }
    }
}

这篇关于如何在dynamodb中超出1mb的扫描数据限制的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！