从压缩的HTTP中解析JSON时出现错误:寻找值的开头时出现无效字符。

huangapple go评论84阅读模式
英文:

Go unmarshalling JSON from compessed HTTP: invalid character looking for beginning of value

问题

我刚刚写了我的第一个Go应用程序,它通过http下载和解析简单的JSON对象。HTTP内容是压缩的,'content-encoding': 'deflate'

我使用了几个众所周知的示例(比如这个)。不幸的是,该应用程序无法解析所需的JSON,并出现了非常罕见和奇怪的错误。我无法找出问题所在。任何帮助将不胜感激。

JSON输入
(使用Python进行调试)

In [8]: r = requests.get("http://172.17.0.31:20000/top")

In [9]: r.text
Out[9]: u'{"timestamp":{"tv_sec":1428447555,"tv_usec":600186},"string_timestamp":"2015-04-07 22:59:15.600186","monitor_status":"enabled"}'

In [18]: r.headers
Out[18]: {'content-length': '111', 'content-type': 'application/json', 'connection': 'close', 'content-encoding': 'deflate'}

源代码(根据答案进行了更新)

package main

import (
	"encoding/json"
	"fmt"
	"io/ioutil"
	"net/http"
)

type Top struct {
	Timestamp        Timestamp `json:"timestamp"`
	String_timestamp string    `json:"string_timestamp"`
	Monitor_status   string    `json:"monitor_status"`
}

type Timestamp struct {
	Tv_sec  int `json:"tv_sec"`
	Tv_usec int `json:"tv_usec"`
}

func get_content() {

	url := "http://172.17.0.31:20000/top"

	res, err := http.Get(url)
	if err != nil {
		panic(err.Error())
	}
	fmt.Println(res)

	body, err := ioutil.ReadAll(res.Body)
	if err != nil {
		panic(err.Error())
	}
	fmt.Println(body)

	var jsondata Top
	err = json.Unmarshal(body, &jsondata)
	if err != nil {
		panic(err.Error())
	}

	fmt.Println(jsondata)
}

func main() {
	get_content()
}

错误

[vitaly@thermaltake elliptics-manager]$ go run main.go 
&{200 OK 200 HTTP/1.1 1 1 map[Content-Type:[application/json] Content-Length:[111] Content-Encoding:[deflate]] 0xc20803e340 111 [] true map[] 0xc208028820 <nil>}
[120 156 77 203 65 14 130 48 16 70 225 171 152 127 93 76 59 51 162 244 50 13 96 99 154 216 98 232 192 134 112 119 81 55 110 95 190 183 65 83 142 85 251 252 130 223 160 107 168 113 132 119 66 55 145 182 117 108 62 109 249 70 98 234 108 183 27 84 157 83 121 132 191 19 100 221 165 177 210 216 235 137 200 11 123 230 243 207 195 32 79 37 233 52 135 3 235 82 15 29 75 63 60 227 29 251 27 195 90 38 189]
panic: invalid character 'x' looking for beginning of value

**更新:**谢谢大家。现在很明显,这个问题的原因是HTTP响应的deflate压缩。然而,如何在Golang中执行解压缩仍然不清楚(参见这里)。

英文:

I've just written my first Go application which downloads and unmarshales simple JSON object over http. Http content is compressed:
&#39;content-encoding&#39;: &#39;deflate&#39;

I used several well-known examples (like this). Unfortunately the application fails to parse the desired JSON with quite rare and strange error. I wasn't able to find out what's the problem. Any help will be appreciated.

JSON input
(Python was used for debugging)

In [8]: r = requests.get(&quot;http://172.17.0.31:20000/top&quot;)

In [9]: r.text
Out[9]: u&#39;{&quot;timestamp&quot;:{&quot;tv_sec&quot;:1428447555,&quot;tv_usec&quot;:600186},&quot;string_timestamp&quot;:&quot;2015-04-07 22:59:15.600186&quot;,&quot;monitor_status&quot;:&quot;enabled&quot;}&#39;
In [18]: r.headers
Out[18]: {&#39;content-length&#39;: &#39;111&#39;, &#39;content-type&#39;: &#39;application/json&#39;, &#39;connection&#39;: &#39;close&#39;, &#39;content-encoding&#39;: &#39;deflate&#39;}

Source code (UPDATED according to the answers)

package main

import (
	&quot;encoding/json&quot;
	&quot;fmt&quot;
	&quot;io/ioutil&quot;
	&quot;net/http&quot;
)

type Top struct {
	Timestamp        Timestamp `json:&quot;timestamp&quot;`
	String_timestamp string    `json:&quot;string_timestamp&quot;`
	Monitor_status   string    `json:&quot;monitor_status&quot;`
}

type Timestamp struct {
	Tv_sec  int `json:&quot;tv_sec&quot;`
	Tv_usec int `json:&quot;tv_usec&quot;`
}

func get_content() {

	url := &quot;http://172.17.0.31:20000/top&quot;

	res, err := http.Get(url)
	if err != nil {
		panic(err.Error())
	}
	fmt.Println(res)

	body, err := ioutil.ReadAll(res.Body)
	if err != nil {
		panic(err.Error())
	}
	fmt.Println(body)

	var jsondata Top
	err = json.Unmarshal(body, &amp;jsondata)
	if err != nil {
		panic(err.Error())
	}

	fmt.Println(jsondata)
}

func main() {
	get_content()
}

Error

[vitaly@thermaltake elliptics-manager]$ go run main.go 
&amp;{200 OK 200 HTTP/1.1 1 1 map[Content-Type:[application/json] Content-Length:[111] Content-Encoding:[deflate]] 0xc20803e340 111 [] true map[] 0xc208028820 &lt;nil&gt;}
[120 156 77 203 65 14 130 48 16 70 225 171 152 127 93 76 59 51 162 244 50 13 96 99 154 216 98 232 192 134 112 119 81 55 110 95 190 183 65 83 142 85 251 252 130 223 160 107 168 113 132 119 66 55 145 182 117 108 62 109 249 70 98 234 108 183 27 84 157 83 121 132 191 19 100 221 165 177 210 216 235 137 200 11 123 230 243 207 195 32 79 37 233 52 135 3 235 82 15 29 75 63 60 227 29 251 27 195 90 38 189]
panic: invalid character &#39;x&#39; looking for beginning of value

UPD: Thanks everyone. Now it's obvious that the reason of this issue was a deflate compression of HTTP response. However, it's still not clear how to perform a decompression in Golang (see here).

答案1

得分: 4

Go的JSON编组器只能编组Unicode字符串。看起来你的JSON并不是以Unicode编码,而是使用其他编码(例如deflate)。

如果你拿到了字节流:

[120 156 77 203 65 14 130 48 16 70 225 171 152 127 93 76 59 51 162 244 50 13 96 99 154 216 98 232 192 134 112 119 81 55 110 95 190 183 65 83 142 85 251 252 130 223 160 107 168 113 132 119 66 55 145 182 117 108 62 109 249 70 98 234 108 183 27 84 157 83 121 132 191 19 100 221 165 177 210 216 235 137 200 11 123 230 243 207 195 32 79 37 233 52 135 3 235 82 15 29 75 63 60 227 29 251 27 195 90 38 189]

并尝试从中获取一个Unicode字符串:

body := []byte{120, 156, 77, 203, 65, 14, 130, 48, 16, 70, 225, 171, 152, 127, 93, 76, 59, 51, 162, 244, 50, 13, 96, 99, 154, 216, 98, 232, 192, 134, 112, 119, 81, 55, 110, 95, 190, 183, 65, 83, 142, 85, 251, 252, 130, 223, 160, 107, 168, 113, 132, 119, 66, 55, 145, 182, 117, 108, 62, 109, 249, 70, 98, 234, 108, 183, 27, 84, 157, 83, 121, 132, 191, 19, 100, 221, 165, 177, 210, 216, 235, 137, 200, 11, 123, 230, 243, 207, 195, 32, 79, 37, 233, 52, 135, 3, 235, 82, 15, 29, 75, 63, 60, 227, 29, 251, 27, 195, 90, 38, 189}
fmt.Println(string(body))

你会在控制台上看到一个奇怪的(压缩的?)字符串,而不是JSON。

我猜测Python的HTTP客户端会自动解压缩deflate字节,而Go的HTTP客户端不会(我知道它对gzip是这样做的,但不确定对deflate是否也是如此)。在使用JSON编组器解析之前,你需要读取解压缩的字节并将它们转换为Unicode字符串。

英文:

The Go JSON marshaller can only marshal unicode strings. It seems that your JSON is not encoded in unicode, but with some other encoding (deflate?).

If you take your bytes stream:

[120 156 77 203 65 14 130 48 16 70 225 171 152 127 93 76 59 51 162 244 50 13 96 99 154 216 98 232 192 134 112 119 81 55 110 95 190 183 65 83 142 85 251 252 130 223 160 107 168 113 132 119 66 55 145 182 117 108 62 109 249 70 98 234 108 183 27 84 157 83 121 132 191 19 100 221 165 177 210 216 235 137 200 11 123 230 243 207 195 32 79 37 233 52 135 3 235 82 15 29 75 63 60 227 29 251 27 195 90 38 189]

And try to get a unicode string out of it:

body := []byte{120, 156, 77, 203, 65, 14, 130, 48, 16, 70, 225, 171, 152, 127, 93, 76, 59, 51, 162, 244, 50, 13, 96, 99, 154, 216, 98, 232, 192, 134, 112, 119, 81, 55, 110, 95, 190, 183, 65, 83, 142, 85, 251, 252, 130, 223, 160, 107, 168, 113, 132, 119, 66, 55, 145, 182, 117, 108, 62, 109, 249, 70, 98, 234, 108, 183, 27, 84, 157, 83, 121, 132, 191, 19, 100, 221, 165, 177, 210, 216, 235, 137, 200, 11, 123, 230, 243, 207, 195, 32, 79, 37, 233, 52, 135, 3, 235, 82, 15, 29, 75, 63, 60, 227, 29, 251, 27, 195, 90, 38, 189}
fmt.Println(string(body))

You would see a weird (compressed?) string in the console, not JSON.

I guess that the python http client automatically decompresses the deflated bytes while the Go http client does not (I know it does so for gzip but not sure if for deflate). You would have to read out the deflated bytes and convert them into a unicode string before you will be able to use the JSON marshaller to parse them.

答案2

得分: 2

我不了解关于 'x' 的情况,但是结构体字段必须是公开的(以大写字母开头)才能被 JSON 反序列化器考虑。当然,这样字段的名称就不会与 JSON 键匹配,你需要添加类似以下的 JSON 注解:

type Top struct {
    Timestamp        Timestamp `json:"timestamp"`
    String_timestamp string    `json:"string_timestamp"`
    Monitor_status   string    `json:"monitor_status"`
}
英文:

I don't know about 'x', but struct fields must be public (start with a capital letter) to be considered by the json Unmarshaller. Of course then the names don't match the json keys, and you have to add json annotations like so:

type Top struct {
    Timestamp        Timestamp `json:&quot;timestamp&quot;`
    String_timestamp string `json:&quot;string_timestamp&quot;`
    Monitor_status   string `json:&quot;monitor_status&quot;`
}

答案3

得分: 1

我相信这是由于你进行了双重编码。ioutil.ReadAll(res.Body)返回一个[]byte,所以当你执行[]byte(body)时,你正在将一个已经是字节数组的内容进行类型转换,我猜测第一个字节的UTF值是x。只需将这一行代码更新为json.Unmarshal(body, &jsondata),我敢打赌它会成功解析。

此外,这与你的错误无关,但正如其他答案中指出的那样,如果你不导出结构体中的字段(在Go语言中,这意味着字段名以大写字母开头),那么解析器将无法使用它们。为了使其工作,你需要更新你的类型定义为:

type Top struct {
    Timestamp        Timestamp `json:"timestamp"`
    String_timestamp string    `json:"string_timestamp"`
    Monitor_status   string    `json:"monitor_status"`
}

JSON注释是必需的,因为解析器非常严格,要求字段名完全匹配(区分大小写)。

英文:

I believe this is due to you double encoding. ioutil.ReadAll(res.Body) returns a []byte so when you do []byte(body) you're casting what is already a byte array, my guess is that first bytes UTF value is x. Just update this; json.Unmarshal([]byte(body), &amp;jsondata) to json.Unmarshal(body, &amp;jsondata) and I bet it will unmarshal just fine.

Also, unrelated to your error but as pointed out in the other answer if you do not export the fields in your struct (in go this means start the field name with a capital letter) then the unmarshaler will not be able to make user of them. To make that work you need to update you type to;

type Top struct {
    Timestamp        Timestamp `json:&quot;timestamp&quot;`
    String_timestamp string `json:&quot;string_timestamp&quot;`
    Monitor_status   string `json:&quot;monitor_status&quot;`
}

The json annotations are required because the unmarshaler is very strict and requires exact matches (case sensitive) for field names.

答案4

得分: 1

请尝试这个:

func get_content() {
    url := "http://172.17.0.31:20000/top"
    res, err := http.Get(url)
    if err != nil {
        panic(err.Error())
    }
    defer res.Body.Close()

    fmt.Println("res body:", res.Body)

    body, err := ioutil.ReadAll(res.Body)
    fmt.Println("body:", body)

    re, err := zlib.NewReader(bytes.NewReader(body))
    fmt.Println("zlib:", re)

    enflated, err := ioutil.ReadAll(re)
    fmt.Println("enflated:", string(enflated))

    var jsondata Top
    err = json.Unmarshal(body, &jsondata)
    if err != nil {
        panic(err.Error())
    }

    fmt.Println(jsondata)
}

并确保 http://172.17.0.31:20000/top 返回的是 JSON 类型。

英文:

Try this please

func get_content() {

    url := &quot;http://172.17.0.31:20000/top&quot;

    res, err := http.Get(url)
    if err != nil {
        panic(err.Error())
    }
    defer res.Body.Close()

    fmt.Println(&quot;res body:&quot;, res.Body)

    body, err := ioutil.ReadAll(resp=.Body)

    fmt.Println(&quot;body:&quot;, body)
    re, err := zlib.NewReader(bytes.NewReader(body))
    fmt.Println(&quot;zlib:&quot;, re)
    enflated, err := ioutil.ReadAll(re)
    fmt.Println(&quot;enflated:&quot;, string(enflated))

    var jsondata Top
    err = json.Unmarshal(body, &amp;jsondata)
    if err != nil {
        panic(err.Error())
    }

    fmt.Println(jsondata)
}

and ensure http://172.17.0.31:20000/top return json type.

huangapple
  • 本文由 发表于 2015年4月8日 07:03:56
  • 转载请务必保留本文链接:https://go.coder-hub.com/29502917.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定