AWS Lambda&节点:在流式传输时写入数据-过早结束并且缺少数据 [英] AWS Lambda & Node: Write data while streaming - ends prematurely and data is missing

问题描述

我有一个Lambda函数，该函数由对S3存储桶的写入操作触发.它会读取写入存储桶的JSON文件，解析出各个记录，然后将其写入数据库.

I've got a Lambda function that is triggered by a write to an S3 bucket. It reads the JSON file that is written to the bucket, parses out the individual records, and writes them to a database.

问题是；我不确定自己在做什么错，因为在所有数据写入之前，流结束并且Lambda退出.

Problem is; I'm not sure what I'm doing wrong, because the stream ends and the Lambda exits before all the data is written.

我处于可读流上的流模式"，并且在数据库写入期间暂停/恢复.根据文档，这应该可以解决问题，但是不能按预期工作.

I'm in "flowing mode" on my readable stream, and I'm pausing/resuming during the db write. According to the docs, this should do the trick, but it's not working as expected.

Lambda处理程序:

Lambda handler:

exports.handler = async (event, context) => {
    let result = false;
    try {
        result = await parseData(event);
    } catch (e) {
        console.error(e);
    }
    return result;
};

承诺:

const StreamArray = require("stream-json/streamers/StreamArray");

async parseData(event) {
    try {
        let objectStream = s3.getObject(params).createReadStream();
        const streamParser = StreamArray.withParser();
        return new Promise((resolve, reject) => {
            objectStream.pipe(streamParser).on("data", async streamData => {
                objectStream.pause();
                let result = await writeData(streamData);
                objectStream.resume();
            }).on("finish", () => {
                console.log("STREAM FINISH!");
                resolve(true);
            }).on("error", e => {
                console.error("Stream error:", e);
                reject(e);
            });
        });
    } catch (e) {
        console.error(e);
    }
}

推荐答案

通过简单地将stream-json与JSONStream换出即可使其正常工作，JSONStream是一个使用范围更广的软件包.现在就像魅力一样！

Got it working by simply swapping-out stream-json with JSONStream, which is a more widely-used package anyhow. Works like a charm now!

const JSONStream = require("JSONStream");

async parseData(event) {
    try {
        let objectStream = s3.getObject(params).createReadStream();
        const streamParser = JSONStream.parse("*");
        return new Promise((resolve, reject) => {
            objectStream.pipe(streamParser).on("data", async streamData => {
                streamParser.pause();
                let result = await writeData(streamData);
                streamParser.resume();
            }).on("finish", () => {
                console.log("STREAM FINISH!");
                resolve(true);
            }).on("error", e => {
                console.error("Stream error:", e);
                reject(e);
            });
        });
    } catch (e) {
        console.error(e);
    }
}

这篇关于AWS Lambda&节点:在流式传输时写入数据-过早结束并且缺少数据的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！