英文:
Gzip header forces file download
问题
我正在尝试对所有响应进行gzip压缩。
在main.go中:
mux := mux.NewRouter()
mux.Use(middlewareHeaders)
mux.Use(gzipHandler)
然后我有以下中间件:
func gzipHandler(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gz := gzip.NewWriter(w)
defer gz.Close()
gzr := gzipResponseWriter{Writer: gz, ResponseWriter: w}
next.ServeHTTP(gzr, r)
})
}
func middlewareHeaders(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Cache-Control", "max-age=2592000") // 30 days
w.Header().Set("Content-Encoding", "gzip")
w.Header().Set("Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload")
w.Header().Set("Access-Control-Allow-Headers", "Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token")
w.Header().Set("Access-Control-Allow-Methods", "POST")
w.Header().Set("Access-Control-Allow-Origin", "origin")
w.Header().Set("Access-Control-Allow-Credentials", "true")
w.Header().Set("Access-Control-Expose-Headers", "AMP-Access-Control-Allow-Source-Origin")
w.Header().Set("AMP-Access-Control-Allow-Source-Origin", os.Getenv("DOMAIN"))
next.ServeHTTP(w, r)
})
}
当我使用curl访问该网站时,我得到以下结果:
curl -v https://example.com
* Trying 44.234.222.27:443...
* TCP_NODELAY set
* Connected to example.com (XX.XXX.XXX.XX) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=example.com
* start date: Mar 16 00:00:00 2021 GMT
* expire date: Apr 16 23:59:59 2022 GMT
* subjectAltName: host "example.com" matched cert's "example.com"
* issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55cadcebfe10)
> GET / HTTP/2
> Host: example.com
> user-agent: curl/7.68.0
> accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< date: Mon, 07 Jun 2021 20:13:19 GMT
< access-control-allow-credentials: true
< access-control-allow-headers: Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token
< access-control-allow-methods: POST
< access-control-allow-origin: origin
< access-control-expose-headers: AMP-Access-Control-Allow-Source-Origin
< amp-access-control-allow-source-origin: example.com
< cache-control: max-age=2592000
< content-encoding: gzip
< strict-transport-security: max-age=63072000; includeSubDomains; preload
< vary: Accept-Encoding
<
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
* Failed writing body (0 != 3506)
* stopped the pause stream!
* Connection #0 to host example.com left intact
当启用gzip处理程序和gzip头部时,浏览器会下载一个文件。
有人能发现我的错误吗?
英文:
I am trying to gzip all responses.
In main.go
mux := mux.NewRouter()
mux.Use(middlewareHeaders)
mux.Use(gzipHandler)
Then I have the middlewares:
func gzipHandler(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
gz := gzip.NewWriter(w)
defer gz.Close()
gzr := gzipResponseWriter{Writer: gz, ResponseWriter: w}
next.ServeHTTP(gzr, r)
})
}
func middlewareHeaders(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Cache-Control", "max-age=2592000") // 30 days
w.Header().Set("Content-Encoding", "gzip")
w.Header().Set("Strict-Transport-Security", "max-age=63072000; includeSubDomains; preload")
w.Header().Set("Access-Control-Allow-Headers", "Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token")
w.Header().Set("Access-Control-Allow-Methods", "POST")
w.Header().Set("Access-Control-Allow-Origin", "origin")
w.Header().Set("Access-Control-Allow-Credentials", "true")
w.Header().Set("Access-Control-Expose-Headers", "AMP-Access-Control-Allow-Source-Origin")
w.Header().Set("AMP-Access-Control-Allow-Source-Origin", os.Getenv("DOMAIN"))
next.ServeHTTP(w, r)
})
}
When I curl the site I get
curl -v https://example.com
* Trying 44.234.222.27:443...
* TCP_NODELAY set
* Connected to example.com (XX.XXX.XXX.XX) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/certs/ca-certificates.crt
CApath: /etc/ssl/certs
* TLSv1.3 (OUT), TLS handshake, Client hello (1):
* TLSv1.3 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=example.com
* start date: Mar 16 00:00:00 2021 GMT
* expire date: Apr 16 23:59:59 2022 GMT
* subjectAltName: host "example.com" matched cert's "example.com"
* issuer: C=GB; ST=Greater Manchester; L=Salford; O=Sectigo Limited; CN=Sectigo RSA Domain Validation Secure Server CA
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x55cadcebfe10)
> GET / HTTP/2
> Host: example.com
> user-agent: curl/7.68.0
> accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 128)!
< HTTP/2 200
< date: Mon, 07 Jun 2021 20:13:19 GMT
< access-control-allow-credentials: true
< access-control-allow-headers: Content-Type,X-Amz-Date,Authorization,X-Api-Key,X-Amz-Security-Token
< access-control-allow-methods: POST
< access-control-allow-origin: origin
< access-control-expose-headers: AMP-Access-Control-Allow-Source-Origin
< amp-access-control-allow-source-origin: example.com
< cache-control: max-age=2592000
< content-encoding: gzip
< strict-transport-security: max-age=63072000; includeSubDomains; preload
< vary: Accept-Encoding
<
Warning: Binary output can mess up your terminal. Use "--output -" to tell
Warning: curl to output it to your terminal anyway, or consider "--output
Warning: <FILE>" to save to a file.
* Failed writing body (0 != 3506)
* stopped the pause stream!
* Connection #0 to host example.com left intact
When enabling the gzip handler and gzip header the browser wants to download a file.
Can anyone spot my error?
答案1
得分: 3
- 只有在客户端请求时才应该使用
gzip
。
即使没有请求Accept-Encoding: gzip
,但你仍然对响应进行了gzip
压缩。
所以curl
将原样返回给你。
-
根据你的浏览器行为,听起来像是双重压缩。也许你已经使用了一些HTTP反向代理,它已经处理了向浏览器的压缩,但没有对后端流量进行压缩。所以你可能根本不需要在后端进行任何压缩 - 尝试使用
curl --compressed
来确认这一点。 -
你应该从响应中过滤掉
Content-Length
。Content-Length是压缩后的HTTP响应的最终大小,因此在压缩过程中该值会发生变化。 -
你不应该盲目地对所有URI应用压缩。一些处理程序已经执行了压缩(例如prometheus的
/metrics
),而有些URI是无意义的压缩对象(例如.png
,.zip
,.gz
)。至少在将请求传递给处理程序链之前,应该从请求中删除Accept-Encoding: gzip
,以避免双重压缩。 -
在Go中实现了透明的gzip压缩。快速搜索发现了这个代码片段(根据上述第4点进行了调整):
package main
import (
"compress/gzip"
"io"
"io/ioutil"
"net/http"
"strings"
"sync"
)
var gzPool = sync.Pool{
New: func() interface{} {
w := gzip.NewWriter(ioutil.Discard)
return w
},
}
type gzipResponseWriter struct {
io.Writer
http.ResponseWriter
}
func (w *gzipResponseWriter) WriteHeader(status int) {
w.Header().Del("Content-Length")
w.ResponseWriter.WriteHeader(status)
}
func (w *gzipResponseWriter) Write(b []byte) (int, error) {
return w.Writer.Write(b)
}
func Gzip(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if !strings.Contains(r.Header.Get("Accept-Encoding"), "gzip") {
next.ServeHTTP(w, r)
return
}
w.Header().Set("Content-Encoding", "gzip")
gz := gzPool.Get().(*gzip.Writer)
defer gzPool.Put(gz)
gz.Reset(w)
defer gz.Close()
r.Header.Del("Accept-Encoding")
next.ServeHTTP(&gzipResponseWriter{ResponseWriter: w, Writer: gz}, r)
})
}
注意 - 上述代码不支持分块编码和尾部。因此仍有改进的机会。
英文:
1. You should only gzip
when it's requested by the client.
Accept-Encoding: gzip
is never requested, but you gzip
the response anyway.
So curl
gives it back to you as-is.
2. Given the behavior of your browser, it sounds like double-compression. Maybe you have some HTTP reverse proxy in place which already handles compression to the browser, but doesn't compress backend traffic. So you may not need any gzipping at the backend at all - try curl --compressed
to confirm this.
3. You should filter out Content-Length
from the response. Content-Length is the final size of the compressed HTTP response, so the value changes during compression.
4. You should not blindly apply compression to all URI's. Some handlers perform gzipping already (e.g. prometheus /metrics
), and some are pointless to compress (e.g. .png
, .zip
, .gz
). At the very least strip Accept-Encoding: gzip
from the request before passing it down the handler chain, to avoid double-gzipping.
5. Transparent gzipping in Go has been implemented before. A quick search reveals this gist (adjusted for point #4 above):
package main
import (
"compress/gzip"
"io"
"io/ioutil"
"net/http"
"strings"
"sync"
)
var gzPool = sync.Pool{
New: func() interface{} {
w := gzip.NewWriter(ioutil.Discard)
return w
},
}
type gzipResponseWriter struct {
io.Writer
http.ResponseWriter
}
func (w *gzipResponseWriter) WriteHeader(status int) {
w.Header().Del("Content-Length")
w.ResponseWriter.WriteHeader(status)
}
func (w *gzipResponseWriter) Write(b []byte) (int, error) {
return w.Writer.Write(b)
}
func Gzip(next http.Handler) http.Handler {
return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
if !strings.Contains(r.Header.Get("Accept-Encoding"), "gzip") {
next.ServeHTTP(w, r)
return
}
w.Header().Set("Content-Encoding", "gzip")
gz := gzPool.Get().(*gzip.Writer)
defer gzPool.Put(gz)
gz.Reset(w)
defer gz.Close()
r.Header.Del("Accept-Encoding")
next.ServeHTTP(&gzipResponseWriter{ResponseWriter: w, Writer: gz}, r)
})
}
Note - the above doesn't support chunked encoding and trailers. So there's still opportunity for improvement.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论