除非增加写容量，否则DynamoDB javascript SDK batchWriteItem不会完成 [英] DynamoDB javascript SDK batchWriteItem doesn't complete unless I increase write capacity

问题描述

我正在运行一系列单元测试(node.js 4.x，aws-sdk，mocha)，这些单元测试在每次测试之前将数据加载到表中，然后在测试之后清除表.

I'm running a series of unit tests (node.js 4.x, aws-sdk, mocha) which load data into a table before each test then clears the table after the test.

由于ConditionExpression会触发ConditionCheckFailedException，所以我有两个测试失败.但是，如果我增加了读/写能力，它们的测试就会通过.

I have two tests that are failing because of a ConditionExpression which triggers a ConditionCheckFailedException. But if I increase the read/write capacity they tests pass.

据我了解，SDK会处理节流异常并为您重试它们，那么为什么我的测试不会更慢地运行并通过?相反，似乎测试无法完成scan-> batchWriteItem过程，因此当新的测试开始时，表中仍然有记录.

It's my understanding that the SDK handles throttling exceptions and retries them for you so why wouldn't my tests just run slower and pass? Instead it seems as though the tests fail to complete the scan -> batchWriteItem process and so there are records still left in the table when a new tests starts.

团队成员告诉我，他们也遇到过类似的问题，他们只是增加了解决问题的吞吐量.这不适合我.是我做错了事，还是在测试中出现了竞争条件，或者应该制定一种模式来确保节流时我的操作完成?我应该能够使用限制指标来通知何时需要增加吞吐量，但是我仍然可以继续重试，直到内存用完为止.

I'm told by team members that they've seen similar problems and they just increased the throughput to fix the problem. This doesn't sit right with me. Either I'm doing something wrong and there's a race condition with my tests or there should be a pattern I can implement to make sure that my operations complete when being throttled? I should be able to use throttling metrics to inform when I need to increase throughput but I should still be able to keep retrying until I run out of memory.

还有其他人遇到这个问题吗?您做了什么处理该问题?

Has anyone else run into this and what have you done to handle the problem?

推荐答案

在进行一些调试之后，我注意到了UnprocessedItems响应元素.在文档中查找UnprocessedItems 后， 我意识到我应该更仔细地阅读.下面的代码将运行重试循环，并带有延迟(指数补偿):

After some debugging I noticed the UnprocessedItems response element. After looking up UnprocessedItems in the docs I realize I should have read more closely. The code below will run a retry loop with a delay (exponential back-off):

var clearEventTable = function (tableName, client, cleared) {
  var exclusiveStartKey = null;
  var retryCount = 0;

  var read = function(query, callback) {
    client.scan(query, function (err, page) {
      if(err) {
        console.log(err);
        return callback(err);
      }

      retryCount = 0;
      exclusiveStartKey = page.LastEvaluatedKey || null;
      if(page.Count == 0) {
        return callback(null, {});
      }

      if(page.Count < 25 && exclusiveStartKey) {
        console.log("read capacity limit reached: " + JSON.stringify(page, null, 2));
      }

      var keys = _.map(page.Items, function(n) {
        return { DeleteRequest: { Key: n } };
      });

      var batch = {
        RequestItems: {},
        ReturnConsumedCapacity: "INDEXES",
        ReturnItemCollectionMetrics: "SIZE"
      };

      batch.RequestItems[tableName] = keys;

      callback(null, batch);
    });
  };

  var write = function(batch, callback) {
    if(batch && batch.RequestItems){
      client.batchWriteItem(batch, function(err, result) {
        if(err) {
          console.log(err);
          return callback(err);
        }

        if(Object.keys(result.UnprocessedItems).length !== 0) {
          console.log("Retry batchWriteItem: " + JSON.stringify(result, null, 2));
          retryCount++;
          var retry = {
            RequestItems: result.UnprocessedItems,
            ReturnConsumedCapacity: "INDEXES",
            ReturnItemCollectionMetrics: "SIZE"
          };
          // retry with exponential backoff
          var delay = retryCount > 0 ? (50 * Math.pow(2, retryCount - 1)) : 0;
          setTimeout(write(retry, callback), delay);
          return;
        }

        callback(null, result);
      });
    } else {
      callback(null);
    }
  };

  var params = {
    TableName: tableName,
    ProjectionExpression: "aggregateId,id",
    Limit: 25, // max 25 per batchWriteItem 
    ConsistentRead: false,
    ReturnConsumedCapacity: "TOTAL"
  };

  async.doWhilst(function (next) {
    // retrieve entities
    if (exclusiveStartKey)
      params.ExclusiveStartKey = exclusiveStartKey;

    async.compose(write, read)(params, function (err, result) {
      if (err) next(err);
      else next(null, result);
    });
  }, function () {
    // test if we need to load more
    return exclusiveStartKey !== null;
  }, function (err, r) {
    // return results
    if (err) {
      console.log(err);
      return cleared(err);
    }
    return cleared(null);;
  });
};

这篇关于除非增加写容量，否则DynamoDB javascript SDK batchWriteItem不会完成的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！