如何将批量数据从CSV文件导入DynamoDB？ [英] How can I import bulk data from a CSV file into DynamoDB?

问题描述

我正在尝试将CS​​V文件数据导入AWS DynamoDB。

I am trying to import a CSV file data into AWS DynamoDB.

这是我的CSV文件的样子：

Here's what my CSV file looks like:

first_name  last_name
sri ram
Rahul   Dravid
JetPay  Underwriter
Anil Kumar  Gurram

推荐答案

您要使用哪种语言导入数据？我刚刚在Node.js中编写了一个函数，该函数可以将CSV文件导入DynamoDB表。它首先将整个CSV解析为一个数组，将数组拆分为25个块，然后将 batchWriteItem 分解为表。

In which language do you want to import the data? I just wrote a function in Node.js that can import a CSV file into a DynamoDB table. It first parses the whole CSV into an array, splits array into (25) chunks and then batchWriteItem into table.

注意：DynamoDB一次最多只能在 batchinsert 中写入25条记录。因此，我们必须将数组拆分为大块。

Note: DynamoDB only allows writing up to 25 records at a time in batchinsert. So we have to split our array into chunks.

    var fs = require('fs');
    var parse = require('csv-parse');
    var async = require('async');

    var csv_filename = "YOUR_CSV_FILENAME_WITH_ABSOLUTE_PATH";

    rs = fs.createReadStream(csv_filename);
    parser = parse({
        columns : true,
        delimiter : ','
    }, function(err, data) {

        var split_arrays = [], size = 25;

        while (data.length > 0) {
            split_arrays.push(data.splice(0, size));
        }
        data_imported = false;
        chunk_no = 1;

        async.each(split_arrays, function(item_data, callback) {
            ddb.batchWriteItem({
                "TABLE_NAME" : item_data
            }, {}, function(err, res, cap) {
                console.log('done going next');
                if (err == null) {
                    console.log('Success chunk #' + chunk_no);
                    data_imported = true;
                } else {
                    console.log(err);
                    console.log('Fail chunk #' + chunk_no);
                    data_imported = false;
                }
                chunk_no++;
                callback();
            });

        }, function() {
            // run after loops
            console.log('all data imported....');

        });

    });
    rs.pipe(parser);

这篇关于如何将批量数据从CSV文件导入DynamoDB？的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！