在DynamoDB中批量加载 [英] Batch loading in DynamoDB

问题描述

您好，目前我正在Dynamodb中加载每一行，这非常慢.
我有大量的数据，希望通过JAVA API加载到DynamoDb.
但这需要花费大量时间，例如要加载100万个数据，我花了2天时间才将其加载到Dynamo.

Hi Currently I am loading each row row in Dynamodb which is veryslow.
I have a huge data which i want to load to DynamoDb by JAVA API.
But this takes huge time .For example to load 1 million data it took me 2 days to load to Dynamo.

在DynamoDb中是否可以批量加载.我找不到有关批量加载或批量加载的任何信息.

Is Batch load possible in DynamoDb.I am not finding any information about bulkload or batch load.

在这里感谢任何帮助.

推荐答案

我知道这是一篇老文章，但是我们很快探索了如何优化这一点，因此着手进行科学发现:)

i know it's an old post, but we came up short exploring how to optimize this so embarked on a scientific discovery :)

http://tech.equinox.com/driving-miss-dynamodb/

它的缺点和缺点Eive上的Hive是一个很好的选择(我知道，它是老套子"). 使用和调整这些参数可以解决问题(有关详细信息，请参见博客):

long and short of it it Hive on EMR is an excellent option (i know, it's "old skool").. using and tuning these parameters do the trick (see blog for details):

SET dynamodb.throughput.write.percent = x;
SET mapred.reduce.tasks = x;
SET hive.exec.reducers.bytes.per.reducer = x;
SET tez.grouping.split-count = x;

这篇关于在DynamoDB中批量加载的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！