解码JSON，因为它仍通过net/http流进 [英] Decode JSON as it is still streaming in via net/http

问题描述

过去，我曾经按照下面所示的方式去从API端点解码JSON.

In the past I've used go to decode JSON from an API endpoint in the manner shown below.

client := &http.Client{}

req, err := http.NewRequest("GET", "https://some/api/endpoint", nil)
res, err := client.Do(req)
defer res.Body.Close()

buf, _ := ioutil.ReadAll(res.Body)

// ... Do some error checking etc ...

err = json.Unmarshal(buf, &response)

不久之后，我将在一个端点上工作，该端点可以通过以下格式向我发送几兆字节的JSON数据.

I am shortly going to be working on an endpoint that could send me several megabytes of JSON data in the following format.

{
    "somefield": "value",
    "items": [
        { LARGE OBJECT },
        { LARGE OBJECT },
        { LARGE OBJECT },
        { LARGE OBJECT },
        ...
    ]
}

JSON有时会包含一个任意长度的大对象数组.我想将这些对象中的每个对象分别放置到消息队列中.我不需要解码对象本身.

The JSON will at some point contain an array of large, arbitrary length, objects. I want to take each one of these objects and place them, separately, into a message queue. I do not need to decode the objects themselves.

如果我使用常规方法，则会在解码之前将整个响应加载到内存中.

If I used my normal method, this would load the entire response into memory before decoding it.

由于响应仍在流中并将其分派到队列中，是否有一种很好的方法可以将每个大对象项分开?我这样做是为了避免在内存中保存尽可能多的数据.

Is there a good way to split out each of the LARGE OBJECT items as the response is still streaming in and dispatch it off to the queue? I'm doing this to avoid holding as much data in memory.

推荐答案

使用 json.Decoder .

使用 Decoder.Decode() ，我们可以阅读(解组)单个价值，而无需消耗和整理完整的流.这很酷，但是您的输入是一个单个" JSON对象，而不是一系列JSON对象，这意味着对Decoder.Decode()的调用将尝试解组包含所有项目(大对象)的完整JSON对象.

With Decoder.Decode(), we may read (unmarshal) a single value without consuming and unmarshaling the complete stream. This is cool, but your input is a "single" JSON object, not a series of JSON objects, which means a call to Decoder.Decode() would attempt to unmarshal the complete JSON object with all items (large objects).

我们想要的是对单个JSON对象进行部分实时处理.为此，我们可以使用 Decoder.Token() 仅解析(高级) JSON输入流中的下一个后续令牌并返回它.这称为事件驱动解析.

What we want is partially, on-the-fly processing of a single JSON object. For this, we may use Decoder.Token() which parses (advances) only the next subsequent token in the JSON input stream and returns it. This is called event-driven parsing.

当然，我们必须处理"(解释并作用于)令牌，并构建一个状态机"，以跟踪我们在处理的JSON结构中的位置.

Of course we have to "process" (interpret and act upon) the tokens and build a "state machine" that keeps track of where we're in the JSON structure we're processing.

这是解决您问题的实现.

Here's an implementation that solves your problem.

我们将使用以下JSON输入:

We will use the following JSON input:

{
    "somefield": "value",
    "otherfield": "othervalue",
    "items": [
        { "id": "1", "data": "data1" },
        { "id": "2", "data": "data2" },
        { "id": "3", "data": "data3" },
        { "id": "4", "data": "data4" }
    ]
}

并阅读items，即以此类型为模型的大对象":

And read the items, the "large objects" modeled by this type:

type LargeObject struct {
    Id   string `json:"id"`
    Data string `json:"data"`
}

我们还将解析和解释JSON对象中的其他字段，但我们只会记录/打印它们.

We will also parse and interpret other fields in the JSON object, but we will only log / print them.

为了简洁，方便地处理错误，我们将使用以下帮助程序错误处理程序功能:

For brevity and easy error handling, We'll use this helper error handler function:

he := func(err error) {
    if err != nil {
        log.Fatal(err)
    }
}

现在让我们看一下操作.为了简洁起见，并在Go Playground上进行了有效的演示，在下面的示例中，我们将读取string值.要读取实际的HTTP响应正文，我们只需要更改一行，这就是我们创建json.Decoder:

And now let's see some action. In the example below for brevity and to have a working demonstration on the Go Playground, we'll read from a string value. To read from an actual HTTP response body, we only have to change a single line, which is how we create the json.Decoder:

dec := json.NewDecoder(res.Body)

因此，演示:

dec := json.NewDecoder(strings.NewReader(jsonStream))
// We expect an object
t, err := dec.Token()
he(err)
if delim, ok := t.(json.Delim); !ok || delim != '{' {
    log.Fatal("Expected object")
}

// Read props
for dec.More() {
    t, err = dec.Token()
    he(err)
    prop := t.(string)
    if t != "items" {
        var v interface{}
        he(dec.Decode(&v))
        log.Printf("Property '%s' = %v", prop, v)
        continue
    }

    // It's the "items". We expect it to be an array
    t, err := dec.Token()
    he(err)
    if delim, ok := t.(json.Delim); !ok || delim != '[' {
        log.Fatal("Expected array")
    }
    // Read items (large objects)
    for dec.More() {
        // Read next item (large object)
        lo := LargeObject{}
        he(dec.Decode(&lo))
        fmt.Printf("Item: %+v\n", lo)
    }
    // Array closing delim
    t, err = dec.Token()
    he(err)
    if delim, ok := t.(json.Delim); !ok || delim != ']' {
        log.Fatal("Expected array closing")
    }
}

// Object closing delim
t, err = dec.Token()
he(err)
if delim, ok := t.(json.Delim); !ok || delim != '}' {
    log.Fatal("Expected object closing")
}

这将产生以下输出:

2009/11/10 23:00:00 Property 'somefield' = value
2009/11/10 23:00:00 Property 'otherfield' = othervalue
Item: {Id:1 Data:data1}
Item: {Id:2 Data:data2}
Item: {Id:3 Data:data3}
Item: {Id:4 Data:data4}

在转到游乐场上尝试完整的有效示例.

Try the full, working example on the Go Playground.

这篇关于解码JSON，因为它仍通过net/http流进的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！