无需一次将所有对象都存储在内存中的MarshalJSON [英] MarshalJSON without having all objects in memory at once

问题描述

我想使用json.Encoder编码大量数据，而又不将所有数据立即加载到内存中.

I want to use json.Encoder to encode a large stream of data without loading all of it into memory at once.

// I want to marshal this
t := struct {
    Foo string

    // Bar is a stream of objects 
    // I don't want it all to be in memory at the same time.
    Bar chan string 
}{
    Foo: "Hello World",
    Bar: make(chan string),
}

// long stream of data
go func() {
    for _, x := range []string{"one", "two", "three"} {
        t.Bar <- x
    }
    close(t.Bar)
}()

我认为也许json包中内置了此功能，但事实并非如此.

I thought maybe the json package had this functionality build in, but that's not the case.

游乐场

// error: json: unsupported type: chan string
if err := json.NewEncoder(os.Stdout).Encode(&t); err != nil {
    log.Fatal(err)
}

我目前只是在自己构建json字符串.

I'm currently just building the json string myself.

游乐场

w := os.Stdout
w.WriteString(`{ "Foo": "` + t.Foo + `", "Bar": [`)

for x := range t.Bar {
    _ = json.NewEncoder(w).Encode(x)
    w.WriteString(`,`)
}

w.WriteString(`]}`)

有更好的方法吗?

如果 json.Marshaler 像这样，那是微不足道的.

If the json.Marshaler was like this is would be trivial.

type Marshaler interface {
    MarshalJSON(io.Writer) error
}

推荐答案

不幸的是，encoding/json软件包还没有办法做到这一点.您现在(手动)执行的操作是执行此操作的最佳方法，而无需修改内置程序包.

Unfortunately the encoding/json package doesn't have a way to do this yet. What you're doing now (manually) is the best way to do it, without modifying the built-in package.

如果要修补encoding/json，则可以在

If you were to patch encoding/json, you could modify the reflectValueQuoted function in encoding/json/encode.go

您需要关注Array案例(Slice的fallthrough):

You would want to focus on the Array case (Slice has a fallthrough):

// Inside switch:
case reflect.Array:
    e.WriteByte('[')
    n := v.Len()
    for i := 0; i < n; i++ {
        if i > 0 {
            e.WriteByte(',')
        }
        e.reflectValue(v.Index(i))
    }
    e.WriteByte(']')

我假设您想以相同的方式对待频道.看起来像这样:

I'm assuming you'd want to treat channel the same way. It would look something like this:

// Inside switch:
case reflect.Chan:
    e.WriteByte('[')
    i := 0
    for {
        x, ok := v.Recv()
        if !ok {
            break
        }
        if i > 0 {
            e.WriteByte(',')
        }
        e.reflectValue(x)
        i++
    }
    e.WriteByte(']')

我对reflect中的频道没有做太多事情，因此上述内容可能需要其他检查.

I haven't done much with channels in reflect, so the above may need other checks.

如果您确实选择了此方法，则可以随时提交补丁.

If you do end up going this route, you could always submit a patch.

这篇关于无需一次将所有对象都存储在内存中的MarshalJSON的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！