英文:
Reading gzipped HTTP response in Go
问题
我正在尝试使用Go语言读取一个经过gzip压缩的HTTP响应,但我总是得到以下错误信息:
panic: gzip: invalid header
[...] stack trace [...]
如果我运行“curl -H "Accept-Encoding: gzip" http://foo.com/ | gunzip -”,我可以正确解压缩响应。我还使用ngrep进行了双重检查,Accept-Encoding/Content-Encoding对是正确发送/返回的。
如果我创建一个带有一些虚拟内容的文件并对其进行gzip压缩,我可以从我的Go程序中读取它。
我用于测试的程序:
package main
import (
"io"
//"os"
"fmt"
"compress/gzip"
"net/http"
)
func main() {
/* 这个可以正常工作
f, _ := os.Open("/tmp/test.gz")
defer f.Close()
reader, err := gzip.NewReader(f)
*/
// 这个不行 :/
resp, _ := http.Get("http://foo.com/")
defer resp.Body.Close()
reader, err := gzip.NewReader(resp.Body)
if err != nil { panic(err) }
buff := make([]byte, 1024)
for {
n, err := reader.Read(buff)
if err != nil && err != io.EOF {
panic(err)
}
if n == 0 {
break
}
}
s := fmt.Sprintf("%s", buff)
fmt.Println(s)
}
我有什么遗漏吗?
英文:
I am trying to read a gzipped HTTP response with Go! but I always get the following error message :
panic: gzip: invalid header
[...] stack trace [...]
If I run "curl -H "Accept-Encoding: gzip" http://foo.com/ | gunzip -" I get the response correctly gunzipped. I also double checked with ngrep and the pair Accept-Encoding/Content-Encoding is correctly sent/returned.
If I create a file with some dummy content and gzip it, I can read it from my Go! program.
The program I used for testing:
package main
import (
"io"
//"os"
"fmt"
"compress/gzip"
"net/http"
)
func main() {
/* This works fine
f, _ := os.Open("/tmp/test.gz")
defer f.Close()
reader, err := gzip.NewReader(f)
*/
// This does not :/
resp, _ := http.Get("http://foo.com/")
defer resp.Body.Close()
reader, err := gzip.NewReader(resp.Body)
if err != nil { panic(err) }
buff := make([]byte, 1024)
for {
n, err := reader.Read(buff)
if err != nil && err != io.EOF {
panic(err)
}
if n == 0 {
break
}
}
s := fmt.Sprintf("%s", buff)
fmt.Println(s)
}
Have I overlooked something ?
答案1
得分: 71
EDIT: 下面是手动处理压缩的示例。如果您不设置头部, 默认的传输将会为您设置并在读取响应体时解压缩。
client := new(http.Client)
request, err := http.NewRequest("GET", "http://stackoverflow.com", nil)
request.Header.Add("Accept-Encoding", "gzip")
response, err := client.Do(request)
defer response.Body.Close()
// 检查服务器是否实际发送了压缩数据
var reader io.ReadCloser
switch response.Header.Get("Content-Encoding") {
case "gzip":
reader, err = gzip.NewReader(response.Body)
defer reader.Close()
default:
reader = response.Body
}
io.Copy(os.Stdout, reader) // 将HTML打印到标准输出
为了简洁起见,省略了错误处理。保留了defer语句。
英文:
EDIT: The following is an example of manually handling compression. If you don't set the header, the default Transport will do it for you and then decompress while you read the response.Body.
client := new(http.Client)
request, err := http.NewRequest("GET", "http://stackoverflow.com", nil)
request.Header.Add("Accept-Encoding", "gzip")
response, err := client.Do(request)
defer response.Body.Close()
// Check that the server actually sent compressed data
var reader io.ReadCloser
switch response.Header.Get("Content-Encoding") {
case "gzip":
reader, err = gzip.NewReader(response.Body)
defer reader.Close()
default:
reader = response.Body
}
io.Copy(os.Stdout, reader) // print html to standard out
Error handling removed for brevity. I kept the defers.
答案2
得分: 37
net/http#Transport
处理 gzip
压缩的响应。你不需要做任何特殊处理。
可以查看 DisableCompression
选项,这里。
英文:
net/http#Transport
handles gzip
compressed responses. You don't have to do anything special.
Take a look at the DisableCompression
option, here.
答案3
得分: 22
根据net/http文档(第110行),如果您手动设置了Accept-Encoding请求头,则http.Transport不会自动解压缩gzip响应。否则,行为由Transport的DisableCompression布尔值控制。
英文:
According to net/http docs (line 110) if you manually set the Accept-Encoding request header, than gzipped response will not automatically decompressed by the http.Transport. Otherwise, behavior is controlled by the Transport's DisableCompression boolean
答案4
得分: 0
以下是手动处理压缩的示例。
import "compress/gzip"
func ReadAll(r io.Reader) ([]byte, error) {
reader, err := gzip.NewReader(r)
if err != nil {
return nil, err
}
defer reader.Close()
buff, err := ioutil.ReadAll(reader)
return buff, err
}
英文:
The following is an example of manually handling compression.
import "compress/gzip"
func ReadAll(r io.Reader) ([]byte, error) {
reader, err := gzip.NewReader(r)
if err != nil {
return nil, err
}
defer reader.Close()
buff, err := ioutil.ReadAll(reader)
return buff, err
}
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论