AWS Firehose换行符 [英] AWS Firehose newline Character

问题描述

关于在Firehose中添加换行符，我已经阅读了很多类似的问题，但它们都是围绕在源代码中添加换行符的.问题是我无权访问源，并且第三方正在将数据管道传输到我们的Kinesis实例，并且无法将'\ n'添加到源中.

I've read a lot of similar questions around adding newline characters to firehose, but they're all around adding the newline character to the source. The problem is that I don't have access to the source, and a third party is piping data to our Kinesis instance and I cannot add the '\n' to the source.

我尝试使用以下代码进行firehose数据转换:

I've tried doing a firehose data transformation using the following code:

'use strict';
console.log('Loading function');

exports.handler = (event, context, callback) => {
    /* Process the list of records and transform them */
    const output = [];
    event.records.forEach((record) => {
        const results = {
        /* This transformation is the "identity" transformation, the data is left intact */
            recordId: record.recordId,
            result: record.data.event_type === 'alert' ? 'Dropped' : 'Ok',
            data: record.data + '\n'
        };
        output.push(results);
    });
    console.log(`Processing completed.  Successful records ${output.length}.`);
    callback(null, { records: output });
};

，但是换行符仍然丢失.我也尝试过JSON.stringify(record.data) + '\n'，但是随后出现Invalid output structure错误.

but the newline is still lost. I've also tried JSON.stringify(record.data) + '\n' but then I get an Invalid output structure error.

推荐答案

尝试解码record.data 添加新行 然后再次将其编码为base 64.

Try decoding the record.data add a new line then encode it again as base 64.

这是python，但想法是相同的

This is python but the idea is the same

for record in event['records']:
    payload = base64.b64decode(record['data'])
    # Do custom processing on the payload here
    payload = payload + '\n'
    output_record = {
        'recordId': record['recordId'],
        'result': 'Ok',
        'data': base64.b64encode(json.dumps(payload))
    }
    output.append(output_record)
return {'records': output}

来自@Matt Westlake的评论:

From the comment of @Matt Westlake:

对于那些寻找节点答案的人，是

For those looking for the node answer, it's

const数据= JSON.parse(newBuffer.from(record.data，'base64').toString('utf8'));

const data = JSON.parse(newBuffer.from(record.data,'base64').toString('utf8'));

和

新Buffer.from(JSON.stringify(data)+'\ n').toString('base64')

new Buffer.from(JSON.stringify(data) + '\n').toString('base64')

这篇关于AWS Firehose换行符的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！