如何将 CSV 文件中的批量数据导入 DynamoDB? [英] How can I import bulk data from a CSV file into DynamoDB?

问题描述

我正在尝试将 CSV 文件数据导入 AWS DynamoDB.

I am trying to import a CSV file data into AWS DynamoDB.

这是我的 CSV 文件的样子:

Here's what my CSV file looks like:

first_name  last_name
sri ram
Rahul   Dravid
JetPay  Underwriter
Anil Kumar  Gurram

推荐答案

您要导入哪种语言的数据?我刚刚在 Node.js 中编写了一个可以将 CSV 文件导入 DynamoDB 表的函数.它首先将整个 CSV 解析为一个数组，将数组拆分为 (25) 个块，然后将 batchWriteItem 放入表中.

In which language do you want to import the data? I just wrote a function in Node.js that can import a CSV file into a DynamoDB table. It first parses the whole CSV into an array, splits array into (25) chunks and then batchWriteItem into table.

注意:DynamoDB 只允许在 batchinsert 中一次最多写入 25 条记录.所以我们必须把我们的数组分成块.

Note: DynamoDB only allows writing up to 25 records at a time in batchinsert. So we have to split our array into chunks.

    var fs = require('fs');
    var parse = require('csv-parse');
    var async = require('async');

    var csv_filename = "YOUR_CSV_FILENAME_WITH_ABSOLUTE_PATH";

    rs = fs.createReadStream(csv_filename);
    parser = parse({
        columns : true,
        delimiter : ','
    }, function(err, data) {

        var split_arrays = [], size = 25;

        while (data.length > 0) {
            split_arrays.push(data.splice(0, size));
        }
        data_imported = false;
        chunk_no = 1;

        async.each(split_arrays, function(item_data, callback) {
            ddb.batchWriteItem({
                "TABLE_NAME" : item_data
            }, {}, function(err, res, cap) {
                console.log('done going next');
                if (err == null) {
                    console.log('Success chunk #' + chunk_no);
                    data_imported = true;
                } else {
                    console.log(err);
                    console.log('Fail chunk #' + chunk_no);
                    data_imported = false;
                }
                chunk_no++;
                callback();
            });

        }, function() {
            // run after loops
            console.log('all data imported....');

        });

    });
    rs.pipe(parser);

这篇关于如何将 CSV 文件中的批量数据导入 DynamoDB?的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！