Golang: “compress/flate”模块无法解压有效的deflate压缩的HTTP正文。

huangapple go评论87阅读模式
英文:

Golang: "compress/flate" module can't decompress valid deflate compressed HTTP body

问题

这个问题是继续讨论这里开始的。我发现由于后者的deflate压缩,HTTP响应体无法解组为JSON对象。现在我想知道如何在Golang中执行解压缩。我将感激任何能够指出我代码中错误的人。

输入数据

我已将HTTP响应体转储到'test'文件中。这是它的内容:

$ cat test
x��PAN�0�
;��NtJ�FӮdU�|"oVR�C%�f�����Z.�^Hs�dW뮑�'��DH�S�SFVC����r)G,�����<���z}�x_g�+�2��sl�r/�Oy>��J3\�G�9���N���#[5M�^v/�2Ҕ��|�h��[�~7�_崛<D*���/��i

让我们确保可以解压缩这个文件,甚至包含有效的JSON:

$ zlib-flate -uncompress < test
{"timestamp":{"tv_sec":1428488670,"tv_usec":197041},"string_timestamp":"2015-04-08 10:24:30.197041","monitor_status":"enabled","commands":{"REVERSE_LOOKUP":{"cache":{"outside":{"successes":0,"failures":0,"size":0,"time":0},"internal":{"successes":0,"failures":0,"size":0,"time":0}},"disk":{"outside":{"successes":0,"failures":0,"size":0,"time":0},"internal":{"successes":13366,"failures":0,"size":0,"time":501808}},"total":{"storage":{"successes":0,"failures":0},"proxy":{"successes":13366,"failures":0}}},"clients":{}}}
$ zlib-flate -uncompress < test | python -m json.tool
{
    "commands": {
        "REVERSE_LOOKUP": {
            "cache": {
               ....

源代码

package main

import (
    "bytes"
    "compress/flate"
    "fmt"
    "io/ioutil"
)

func main() {
    fname := "./test"
    content, err := ioutil.ReadFile(fname)
    if err != nil {
        panic(err)
    }
    fmt.Println("文件内容:\n", content)

    enflated, err := ioutil.ReadAll(flate.NewReader(bytes.NewReader(content)))
    if err != nil {
        panic(err)
    }
    fmt.Println("解压后的内容:\n", enflated)
}

错误

$ go run uncompress.go 
文件内容:
 [120 156 181 80 65 78 195 48 16 252 10 242 57 69 118 226 166 38 247 156 64 42 42 130 107 100 156 165 88 196 118 149 93 35 160 234 223 89 183 61 112 42 226 192 109 118 118 102 103 180 123 65 62 0 146 13 59 209 237 5 189 15 8 78 116 74 215 70 27 211 174 100 85 184 124 34 111 86 82 171 67 37 144 102 31 183 195 15 167 168 165 90 46 164 94 72 115 165 100 87 235 174 145 215 39 189 168 68 72 209 83 154 7 22 83 70 86 67 180 207 19 140 188 114 41 4 27 71 44 225 155 254 169 223 60 244 195 221 122 125 251 120 95 24 103 221 43 20 144 50 161 31 143 16 179 115 128 8 108 225 114 47 214 79 121 62 15 232 191 224 8 74 51 6 92 213 71 130 57 218 233 175 78 182 142 30 223 254 35 91 53 77 219 94 118 47 165 50 210 148 18 148 232 124 128 31 104 183 151 91 176 126 55 167 143 207 95 3 15 229 180 155 60 68 42 159 231 241 27 47 165 167 25]
panic: flate: corrupt input before offset 5

goroutine 1 [running]:
runtime.panic(0x4a7180, 0x5)
    /usr/lib/go/src/pkg/runtime/panic.c:266 +0xb6
main.main()
    /home/isaev/side-projects/elliptics-manager/uncompress.go:20 +0x2a3
exit status 2

PS Ubuntu 14.10, Go 1.2.1

英文:

This question continues the discussion started here. I found out that the HTTP response body can't be unmarshaled into JSON object because of deflate compression of the latter. Now I wonder how can I perform decompression with Golang. I will appreciate anyone who can show the errors in my code.

Input data

I've dumped the HTTP response body into the 'test' file. Here is it:

$ cat test
x��PAN�0�
;��NtJ�FӮdU�|"oVR�C%�f�����Z.�^Hs�dW뮑�'��DH�S�SFVC����r)G,�����<���z}�x_g�+�2��sl�r/�Oy>��J3\�G�9���N���#[5M�^v/�2Ҕ��|�h��[�~7�_崛<D*���/��i

Let's make sure that this file can be decompressed and even contains valid JSON:

$ zlib-flate -uncompress < test
{"timestamp":{"tv_sec":1428488670,"tv_usec":197041},"string_timestamp":"2015-04-08 10:24:30.197041","monitor_status":"enabled","commands":{"REVERSE_LOOKUP":{"cache":{"outside":{"successes":0,"failures":0,"size":0,"time":0},"internal":{"successes":0,"failures":0,"size":0,"time":0}},"disk":{"outside":{"successes":0,"failures":0,"size":0,"time":0},"internal":{"successes":13366,"failures":0,"size":0,"time":501808}},"total":{"storage":{"successes":0,"failures":0},"proxy":{"successes":13366,"failures":0}}},"clients":{}}}
$ zlib-flate -uncompress < test | python -m json.tool
{
    "commands": {
        "REVERSE_LOOKUP": {
            "cache": {
               ....

Source code

package main

import (
    "bytes"
    "compress/flate"
    "fmt"
    "io/ioutil"
)

func main() {
    fname := "./test"
    content, err := ioutil.ReadFile(fname)
    if err != nil {
        panic(err)
    }
    fmt.Println("File content:\n", content)

    enflated, err := ioutil.ReadAll(flate.NewReader(bytes.NewReader(content)))
    if err != nil {
        panic(err)
    }
    fmt.Println("Enflated:\n", enflated)
}

Error

$ go run uncompress.go 
File content:
 [120 156 181 80 65 78 195 48 16 252 10 242 57 69 118 226 166 38 247 156 64 42 42 130 107 100 156 165 88 196 118 149 93 35 160 234 223 89 183 61 112 42 226 192 109 118 118 102 103 180 123 65 62 0 146 13 59 209 237 5 189 15 8 78 116 74 215 70 27 211 174 100 85 184 124 34 111 86 82 171 67 37 144 102 31 183 195 15 167 168 165 90 46 164 94 72 115 165 100 87 235 174 145 215 39 189 168 68 72 209 83 154 7 22 83 70 86 67 180 207 19 140 188 114 41 4 27 71 44 225 155 254 169 223 60 244 195 221 122 125 251 120 95 24 103 221 43 20 144 50 161 31 143 16 179 115 128 8 108 225 114 47 214 79 121 62 15 232 191 224 8 74 51 6 92 213 71 130 57 218 233 175 78 182 142 30 223 254 35 91 53 77 219 94 118 47 165 50 210 148 18 148 232 124 128 31 104 183 151 91 176 126 55 167 143 207 95 3 15 229 180 155 60 68 42 159 231 241 27 47 165 167 25]
panic: flate: corrupt input before offset 5

goroutine 1 [running]:
runtime.panic(0x4a7180, 0x5)
    /usr/lib/go/src/pkg/runtime/panic.c:266 +0xb6
main.main()
    /home/isaev/side-projects/elliptics-manager/uncompress.go:20 +0x2a3
exit status 2

PS Ubuntu 14.10, Go 1.2.1

答案1

得分: 12

您的输入不是一个简单的压缩块,而是一个zlib流

根据ZLIB压缩数据格式规范3.3,前两个字节是:

-------------
| CMF | FLG |
-------------

压缩方法和标志位。您的输入以[120, 156]开头,即十六进制表示为78 9C。这是默认的压缩方法。此外,没有跟随的字典,因此后续的数据就是压缩数据。

位0到3是CM压缩方法,位4到7是CINFO压缩信息。在这种情况下,CINFO=7表示32K的窗口大小,CM=8表示"deflate"压缩方法。FLG的第5位表示是否有预设的字典,这在这种情况下是有的。FLG的详细信息也在链接的RFC 1950中。

因此,您的输入基本上告诉我们剩余的数据是使用默认压缩构建的,但是flate包无法解码这个数据。

将解压缩的代码更改为省略前两个字节,像这样就可以正常工作:

enflated, err := ioutil.ReadAll(flate.NewReader(bytes.NewReader(content[2:])))

Go Playground上尝试一下。但是...

请使用正确的ZLib解压缩!

这次我们运气好,因为压缩级别是默认的,并且字典是预设的。如果不是这样,您将无法使用flate包解码它。由于输入是一个zlib流,您应该使用compress/zlib包来正确解码,而不是依赖运气:

r, err := zlib.NewReader(bytes.NewReader(content))
if err != nil {
    panic(err)
}
enflated, err := ioutil.ReadAll(r)
if err != nil {
    panic(err)
}
fmt.Println(string(enflated))

Go Playground上尝试zlib版本。

英文:

Your input is not a simple deflated block, it's a zlib stream.

According to the ZLIB Compressed Data Format Specification 3.3 the first 2 bytes are:

-------------
| CMF | FLG |
-------------

The Compression Method and flags. Your input starts with [120, 156] which is 78 9C in hexa. This is the Default Compression. Also no dictionary follows, so the subsequent data is the compressed data.

<sup>Bits 0 to 3 are CM Compression Method and bits 4 to 7 are CINFO Compression Info. In this case CINFO=7 indicates a 32K window size, CM=8 denotes the "deflate" compression method. FLG bit 5 tells if a dictionary is preset, which is in this case. Details of the FLG are also in the linked RFC 1950.</sup>

So your input basically tells the rest of the data was constructed using default compression, but the go flate package does not decode this.

Change your decompression to omit the first 2 bytes like this and it will work:

enflated, err := ioutil.ReadAll(flate.NewReader(bytes.NewReader(content[2:])))

Try it on the Go Playground. But...

Use Proper ZLib decompression!

We got lucky this time because the compression level is the default and dictionary was preset. If not, you won't be able to decode it using the flate package. Since the input is a zlib stream, you should use the compress/zlib package to properly decode it and not rely on luck:

r, err := zlib.NewReader(bytes.NewReader(content))
if err != nil {
    panic(err)
}
enflated, err := ioutil.ReadAll(r)
if err != nil {
    panic(err)
}
fmt.Println(string(enflated))

Try the zlib variant on the Go Playground.

huangapple
  • 本文由 发表于 2015年4月8日 19:35:06
  • 转载请务必保留本文链接:https://go.coder-hub.com/29513472.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定