在Go中读取gzip压缩的HTTP响应

huangapple go评论79阅读模式
英文:

Reading gzipped HTTP response in Go

问题

我正在尝试使用Go语言读取一个经过gzip压缩的HTTP响应,但我总是得到以下错误信息:

panic: gzip: invalid header
[...] stack trace [...]

如果我运行“curl -H "Accept-Encoding: gzip" http://foo.com/ | gunzip -”,我可以正确解压缩响应。我还使用ngrep进行了双重检查,Accept-Encoding/Content-Encoding对是正确发送/返回的。

如果我创建一个带有一些虚拟内容的文件并对其进行gzip压缩,我可以从我的Go程序中读取它。

我用于测试的程序:

package main

import (
    "io"
    //"os"
    "fmt"
    "compress/gzip"
    "net/http"
)

func main() {
    /* 这个可以正常工作
    f, _ := os.Open("/tmp/test.gz")
    defer f.Close()
    reader, err := gzip.NewReader(f)
    */

    // 这个不行 :/
    resp, _ := http.Get("http://foo.com/")
    defer resp.Body.Close()
    reader, err := gzip.NewReader(resp.Body)

    if err != nil { panic(err) }

    buff := make([]byte, 1024)
    for {
        n, err := reader.Read(buff)

        if err != nil && err != io.EOF {
            panic(err)
        }

        if n == 0 {
            break
        }
    }

    s := fmt.Sprintf("%s", buff)
    fmt.Println(s)
}

我有什么遗漏吗?

英文:

I am trying to read a gzipped HTTP response with Go! but I always get the following error message :

panic: gzip: invalid header
[...] stack trace [...]

If I run "curl -H "Accept-Encoding: gzip" http://foo.com/ | gunzip -" I get the response correctly gunzipped. I also double checked with ngrep and the pair Accept-Encoding/Content-Encoding is correctly sent/returned.

If I create a file with some dummy content and gzip it, I can read it from my Go! program.

The program I used for testing:

package main

import (
	"io"
	//"os"
	"fmt"
	"compress/gzip"
	"net/http"
)

func main() {
	/* This works fine
	f, _ := os.Open("/tmp/test.gz")
	defer f.Close()
	reader, err := gzip.NewReader(f)
	*/

    // This does not :/
	resp, _ := http.Get("http://foo.com/")
	defer resp.Body.Close()
	reader, err := gzip.NewReader(resp.Body)

	if err != nil { panic(err) }

	buff := make([]byte, 1024)
	for {
		n, err := reader.Read(buff)

		if err != nil && err != io.EOF {
			panic(err)
		}

		if n == 0 {
			break
		}
	}

	s := fmt.Sprintf("%s", buff)
	fmt.Println(s)
}

Have I overlooked something ?

答案1

得分: 71

EDIT: 下面是手动处理压缩的示例。如果您不设置头部, 默认的传输将会为您设置并在读取响应体时解压缩。

client := new(http.Client)

request, err := http.NewRequest("GET", "http://stackoverflow.com", nil)
request.Header.Add("Accept-Encoding", "gzip")

response, err := client.Do(request)
defer response.Body.Close()

// 检查服务器是否实际发送了压缩数据
var reader io.ReadCloser
switch response.Header.Get("Content-Encoding") {
case "gzip":
    reader, err = gzip.NewReader(response.Body)
    defer reader.Close()
default:
    reader = response.Body
}

io.Copy(os.Stdout, reader) // 将HTML打印到标准输出

为了简洁起见,省略了错误处理。保留了defer语句。

英文:

EDIT: The following is an example of manually handling compression. If you don't set the header, the default Transport will do it for you and then decompress while you read the response.Body.

client := new(http.Client)

request, err := http.NewRequest("GET", "http://stackoverflow.com", nil)
request.Header.Add("Accept-Encoding", "gzip")

response, err := client.Do(request)
defer response.Body.Close()

// Check that the server actually sent compressed data
var reader io.ReadCloser
switch response.Header.Get("Content-Encoding") {
case "gzip":
    reader, err = gzip.NewReader(response.Body)
    defer reader.Close()
default:
    reader = response.Body
}

io.Copy(os.Stdout, reader) // print html to standard out

Error handling removed for brevity. I kept the defers.

答案2

得分: 37

net/http#Transport 处理 gzip 压缩的响应。你不需要做任何特殊处理。

可以查看 DisableCompression 选项,这里

英文:

net/http#Transport handles gzip compressed responses. You don't have to do anything special.

Take a look at the DisableCompression option, here.

答案3

得分: 22

根据net/http文档(第110行),如果您手动设置了Accept-Encoding请求头,则http.Transport不会自动解压缩gzip响应。否则,行为由Transport的DisableCompression布尔值控制。

英文:

According to net/http docs (line 110) if you manually set the Accept-Encoding request header, than gzipped response will not automatically decompressed by the http.Transport. Otherwise, behavior is controlled by the Transport's DisableCompression boolean

答案4

得分: 0

以下是手动处理压缩的示例。


import "compress/gzip"

func ReadAll(r io.Reader) ([]byte, error) {
	reader, err := gzip.NewReader(r)
	if err != nil {
		return nil, err
	}
	defer reader.Close()
	buff, err := ioutil.ReadAll(reader)
	return buff, err
}
英文:

The following is an example of manually handling compression.


import "compress/gzip"

func ReadAll(r io.Reader) ([]byte, error) {
	reader, err := gzip.NewReader(r)
	if err != nil {
		return nil, err
	}
	defer reader.Close()
	buff, err := ioutil.ReadAll(reader)
	return buff, err
}

huangapple
  • 本文由 发表于 2012年10月30日 06:06:00
  • 转载请务必保留本文链接:https://go.coder-hub.com/13130341.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定