从compessed HTTP中解组JSON：无效字符查找值的开头 [英] Go unmarshalling JSON from compessed HTTP: invalid character looking for beginning of value

问题描述

我刚刚编写了我的第一个Go应用程序，它通过http下载和解组简单的JSON对象。 Http内容被压缩：
'content-encoding'：'deflate'

I've just written my first Go application which downloads and unmarshales simple JSON object over http. Http content is compressed: 'content-encoding': 'deflate'

我用了好几个已知示例（如此）。不幸的是，应用程序无法解析所需的JSON，并且出现了非常罕见和奇怪的错误。我无法找出问题所在。任何帮助将不胜感激。

I used several well-known examples (like this). Unfortunately the application fails to parse the desired JSON with quite rare and strange error. I wasn't able to find out what's the problem. Any help will be appreciated.

JSON输入
（Python用于调试）

JSON input (Python was used for debugging)

In [8]: r = requests.get("http://172.17.0.31:20000/top")

In [9]: r.text
Out[9]: u'{"timestamp":{"tv_sec":1428447555,"tv_usec":600186},"string_timestamp":"2015-04-07 22:59:15.600186","monitor_status":"enabled"}'
In [18]: r.headers
Out[18]: {'content-length': '111', 'content-type': 'application/json', 'connection': 'close', 'content-encoding': 'deflate'}

源代码（根据答案更新）

package main

import (
    "encoding/json"
    "fmt"
    "io/ioutil"
    "net/http"
)

type Top struct {
    Timestamp        Timestamp `json:"timestamp"`
    String_timestamp string    `json:"string_timestamp"`
    Monitor_status   string    `json:"monitor_status"`
}

type Timestamp struct {
    Tv_sec  int `json:"tv_sec"`
    Tv_usec int `json:"tv_usec"`
}

func get_content() {

    url := "http://172.17.0.31:20000/top"

    res, err := http.Get(url)
    if err != nil {
        panic(err.Error())
    }
    fmt.Println(res)

    body, err := ioutil.ReadAll(res.Body)
    if err != nil {
        panic(err.Error())
    }
    fmt.Println(body)

    var jsondata Top
    err = json.Unmarshal(body, &jsondata)
    if err != nil {
        panic(err.Error())
    }

    fmt.Println(jsondata)
}

func main() {
    get_content()
}

错误

[vitaly@thermaltake elliptics-manager]$ go run main.go 
&{200 OK 200 HTTP/1.1 1 1 map[Content-Type:[application/json] Content-Length:[111] Content-Encoding:[deflate]] 0xc20803e340 111 [] true map[] 0xc208028820 <nil>}
[120 156 77 203 65 14 130 48 16 70 225 171 152 127 93 76 59 51 162 244 50 13 96 99 154 216 98 232 192 134 112 119 81 55 110 95 190 183 65 83 142 85 251 252 130 223 160 107 168 113 132 119 66 55 145 182 117 108 62 109 249 70 98 234 108 183 27 84 157 83 121 132 191 19 100 221 165 177 210 216 235 137 200 11 123 230 243 207 195 32 79 37 233 52 135 3 235 82 15 29 75 63 60 227 29 251 27 195 90 38 189]
panic: invalid character 'x' looking for beginning of value

UPD：谢谢大家。现在很明显，这个问题的原因是 deflate 压缩HTTP响应。但是，目前还不清楚如何在Golang中执行解压缩（参见这里）。

UPD: Thanks everyone. Now it's obvious that the reason of this issue was a deflate compression of HTTP response. However, it's still not clear how to perform a decompression in Golang (see here).

推荐答案

Go JSON编组只能编组unicode字符串。看来你的JSON不是用unicode编码的，而是用其他编码编码（deflate？）。

The Go JSON marshaller can only marshal unicode strings. It seems that your JSON is not encoded in unicode, but with some other encoding (deflate?).

如果你拿走你的字节流：

If you take your bytes stream:

[120 156 77 203 65 14 130 48 16 70 225 171 152 127 93 76 59 51 162 244 50 13 96 99 154 216 98 232 192 134 112 119 81 55 110 95 190 183 65 83 142 85 251 252 130 223 160 107 168 113 132 119 66 55 145 182 117 108 62 109 249 70 98 234 108 183 27 84 157 83 121 132 191 19 100 221 165 177 210 216 235 137 200 11 123 230 243 207 195 32 79 37 233 52 135 3 235 82 15 29 75 63 60 227 29 251 27 195 90 38 189]

并尝试从中获取一个unicode字符串：

And try to get a unicode string out of it:

body := []byte{120, 156, 77, 203, 65, 14, 130, 48, 16, 70, 225, 171, 152, 127, 93, 76, 59, 51, 162, 244, 50, 13, 96, 99, 154, 216, 98, 232, 192, 134, 112, 119, 81, 55, 110, 95, 190, 183, 65, 83, 142, 85, 251, 252, 130, 223, 160, 107, 168, 113, 132, 119, 66, 55, 145, 182, 117, 108, 62, 109, 249, 70, 98, 234, 108, 183, 27, 84, 157, 83, 121, 132, 191, 19, 100, 221, 165, 177, 210, 216, 235, 137, 200, 11, 123, 230, 243, 207, 195, 32, 79, 37, 233, 52, 135, 3, 235, 82, 15, 29, 75, 63, 60, 227, 29, 251, 27, 195, 90, 38, 189}
fmt.Println(string(body))

你会在控制台中看到一个奇怪的（压缩的？）字符串，而不是JSON。

You would see a weird (compressed?) string in the console, not JSON.

我猜python http客户端会自动解压缩缩放的字节，而Go http客户端则不会解压缩（我知道它对gzip有效，但不确定是否为deflate）。您必须读出缩小的字节并将它们转换为unicode字符串，然后才能使用JSON编组器来解析它们。

I guess that the python http client automatically decompresses the deflated bytes while the Go http client does not (I know it does so for gzip but not sure if for deflate). You would have to read out the deflated bytes and convert them into a unicode string before you will be able to use the JSON marshaller to parse them.

这篇关于从compessed HTTP中解组JSON：无效字符查找值的开头的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！