Kinesis Firehose在不使用分隔符逗号的情况下将JSON对象放入S3 [英] Kinesis Firehose putting JSON objects in S3 without seperator comma

问题描述

在发送数据之前，我正在使用JSON.stringify到数据，看起来像这样

Before sending the data I am using JSON.stringify to the data and it looks like this

{"data": [{"key1": value1, "key2": value2}, {"key1": value1, "key2": value2}]}

但是一旦它通过AWS API网关，并且Kinesis Firehose将其放到S3中，就看起来像这样

But once it passes through AWS API Gateway and Kinesis Firehose puts it to S3 it looks like this

    {
     "key1": value1, 
     "key2": value2
    }{
     "key1": value1, 
     "key2": value2
    }

JSON对象之间的分隔符逗号消失了，但是我需要它来正确处理数据.

The seperator comma between the JSON objects are gone but I need it to process data properly.

API网关中的模板:

Template in the API Gateway:

#set($root = $input.path('$'))
{
    "DeliveryStreamName": "some-delivery-stream",
    "Records": [
#foreach($r in $root.data)
#set($data = "{
    ""key1"": ""$r.value1"",
    ""key2"": ""$r.value2""
}")
    {
        "Data": "$util.base64Encode($data)"
    }#if($foreach.hasNext),#end
#end
    ]
}

推荐答案

我最近也遇到了同样的问题，而我唯一能找到的答案就是在行尾添加换行符("\ n").每当您将每条JSON消息发布到Kinesis流中，或使用某种原始的JSON解码器方法(可以处理无分隔符的串联JSON对象)时，每条JSON消息.

I had this same problem recently, and the only answers I was able to find were basically just to add line breaks ("\n") to the end of every JSON message whenever you posted them to the Kinesis stream, or to use a raw JSON decoder method of some sort that can process concatenated JSON objects without delimiters.

我发布了一个python代码解决方案，可以在此处的相关Stack Overflow帖子中找到: https://stackoverflow.com/a/49417680/1546785

I posted a python code solution which can be found over here on a related Stack Overflow post: https://stackoverflow.com/a/49417680/1546785

这篇关于Kinesis Firehose在不使用分隔符逗号的情况下将JSON对象放入S3的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！