AWS Firehose换行符 [英] AWS Firehose newline Character
问题描述
关于在Firehose中添加换行符,我已经阅读了很多类似的问题,但它们都是围绕在源代码中添加换行符的.问题是我无权访问源,并且第三方正在将数据管道传输到我们的Kinesis实例,并且无法将'\ n'添加到源中.
I've read a lot of similar questions around adding newline characters to firehose, but they're all around adding the newline character to the source. The problem is that I don't have access to the source, and a third party is piping data to our Kinesis instance and I cannot add the '\n' to the source.
我尝试使用以下代码进行firehose数据转换:
I've tried doing a firehose data transformation using the following code:
'use strict';
console.log('Loading function');
exports.handler = (event, context, callback) => {
/* Process the list of records and transform them */
const output = [];
event.records.forEach((record) => {
const results = {
/* This transformation is the "identity" transformation, the data is left intact */
recordId: record.recordId,
result: record.data.event_type === 'alert' ? 'Dropped' : 'Ok',
data: record.data + '\n'
};
output.push(results);
});
console.log(`Processing completed. Successful records ${output.length}.`);
callback(null, { records: output });
};
,但是换行符仍然丢失.我也尝试过JSON.stringify(record.data) + '\n'
,但是随后出现Invalid output structure
错误.
but the newline is still lost. I've also tried JSON.stringify(record.data) + '\n'
but then I get an Invalid output structure
error.
推荐答案
尝试解码record.data 添加新行 然后再次将其编码为base 64.
Try decoding the record.data add a new line then encode it again as base 64.
这是python,但想法是相同的
This is python but the idea is the same
for record in event['records']:
payload = base64.b64decode(record['data'])
# Do custom processing on the payload here
payload = payload + '\n'
output_record = {
'recordId': record['recordId'],
'result': 'Ok',
'data': base64.b64encode(json.dumps(payload))
}
output.append(output_record)
return {'records': output}
来自@Matt Westlake的评论:
From the comment of @Matt Westlake:
对于那些寻找节点答案的人,是
For those looking for the node answer, it's
const数据= JSON.parse(newBuffer.from(record.data,'base64').toString('utf8'));
const data = JSON.parse(newBuffer.from(record.data,'base64').toString('utf8'));
和
新Buffer.from(JSON.stringify(data)+'\ n').toString('base64')
new Buffer.from(JSON.stringify(data) + '\n').toString('base64')
这篇关于AWS Firehose换行符的文章就介绍到这了,希望我们推荐的答案对大家有所帮助,也希望大家多多支持IT屋!