英文:
Mocking HTTPS responses in Go
问题
我正在尝试为一个向 Web 服务发起请求的包编写测试。我遇到了问题,可能是由于我对 TLS 的理解不够。
目前我的测试代码大致如下:
func TestSimple() {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(200)
fmt.Fprintf(w, `{ "fake" : "json data here" }`)
}))
transport := &http.Transport{
Proxy: func(req *http.Request) (*url.URL, error) {
return url.Parse(server.URL)
},
}
// Client 是我的包中发起请求的类型
client := Client{
c: http.Client{Transport: transport},
}
client.DoRequest() // ...
}
我的包中有一个包变量(我希望它是一个常量),用于查询 Web 服务的基本地址。它是一个 HTTPS URL。我上面创建的测试服务器是普通的 HTTP,没有 TLS。
默认情况下,我的测试失败,错误信息是 "tls: first record does not look like a TLS handshake."。
为了使其工作,我的测试在发起请求之前将包变量更改为普通的 HTTP URL。
有没有办法解决这个问题?我可以将包变量设置为常量(HTTPS),然后设置一个可以 "降级" 为非加密 HTTP 的 http.Transport
,或者使用 httptest.NewTLSServer()
替代?
(当我尝试使用 NewTLSServer()
时,我得到的错误是 "http: TLS handshake error from 127.0.0.1:45678: tls: oversized record received with length 20037")
英文:
I'm trying to write tests for a package that makes requests to a web service. I'm running into issues probably due to my lack of understanding of TLS.
Currently my test looks something like this:
func TestSimple() {
server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
w.Header().Set("Content-Type", "application/json")
w.WriteHeader(200)
fmt.Fprintf(w, `{ "fake" : "json data here" }`)
}))
transport := &http.Transport{
Proxy: func(req *http.Request) (*url.URL, error) {
return url.Parse(server.URL)
},
}
// Client is the type in my package that makes requests
client := Client{
c: http.Client{Transport: transport},
}
client.DoRequest() // ...
}
My package has a package variable (I'd like for it to be a constant..) for the base address of the web service to query. It is an https URL. The test server I created above is plain HTTP, no TLS.
By default, my test fails with the error "tls: first record does not look like a TLS handshake."
To get this to work, my tests change the package variable to a plain http URL instead of https before making the query.
Is there any way around this? Can I make the package variable a constant (https), and either set up a http.Transport
that "downgrades" to unencrypted HTTP, or use httptest.NewTLSServer()
instead?
(When I try to use NewTLSServer()
I get "http: TLS handshake error from 127.0.0.1:45678: tls: oversized record received with length 20037")
答案1
得分: 16
net/http
中的大部分行为都可以进行模拟、扩展或修改。虽然http.Client
是一个实现HTTP客户端语义的具体类型,但它的所有字段都是可导出的,可以进行自定义。
特别是Client.Transport
字段可以被替换,使得Client可以执行各种操作,从使用自定义协议(如ftp://或file://)到直接连接到本地处理程序(而不生成HTTP协议字节或通过网络发送任何内容)。
客户端函数,例如http.Get
,都使用导出的http.DefaultClient
包变量(您可以修改它),因此使用这些便利函数的代码不需要更改为在自定义的Client变量上调用方法。请注意,虽然在公开可用的库中修改全局行为是不合理的,但在应用程序和测试中(包括库测试)这样做非常有用。
http://play.golang.org/p/afljO086iB 包含一个自定义的http.RoundTripper
,它重写请求URL,以便将其路由到本地托管的httptest.Server
,以及另一个示例,它直接将请求传递给http.Handler
,并使用自定义的http.ResponseWriter
实现来创建http.Response
。第二种方法不像第一种方法那样仔细(它没有填充Response值中的许多字段),但更高效,并且应该与大多数处理程序和客户端调用者兼容。
上面链接的代码也包含在下面:
package main
import (
"fmt"
"io"
"log"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path"
"strings"
)
func Handler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "hello %s\n", path.Base(r.URL.Path))
}
func main() {
s := httptest.NewServer(http.HandlerFunc(Handler))
u, err := url.Parse(s.URL)
if err != nil {
log.Fatalln("failed to parse httptest.Server URL:", err)
}
http.DefaultClient.Transport = RewriteTransport{URL: u}
resp, err := http.Get("https://google.com/path-one")
if err != nil {
log.Fatalln("failed to send first request:", err)
}
fmt.Println("[First Response]")
resp.Write(os.Stdout)
fmt.Print("\n", strings.Repeat("-", 80), "\n\n")
http.DefaultClient.Transport = HandlerTransport{http.HandlerFunc(Handler)}
resp, err = http.Get("https://google.com/path-two")
if err != nil {
log.Fatalln("failed to send second request:", err)
}
fmt.Println("[Second Response]")
resp.Write(os.Stdout)
}
// RewriteTransport is an http.RoundTripper that rewrites requests
// using the provided URL's Scheme and Host, and its Path as a prefix.
// The Opaque field is untouched.
// If Transport is nil, http.DefaultTransport is used
type RewriteTransport struct {
Transport http.RoundTripper
URL *url.URL
}
func (t RewriteTransport) RoundTrip(req *http.Request) (*http.Response, error) {
// note that url.URL.ResolveReference doesn't work here
// since t.u is an absolute url
req.URL.Scheme = t.URL.Scheme
req.URL.Host = t.URL.Host
req.URL.Path = path.Join(t.URL.Path, req.URL.Path)
rt := t.Transport
if rt == nil {
rt = http.DefaultTransport
}
return rt.RoundTrip(req)
}
type HandlerTransport struct{ h http.Handler }
func (t HandlerTransport) RoundTrip(req *http.Request) (*http.Response, error) {
r, w := io.Pipe()
resp := &http.Response{
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: make(http.Header),
Body: r,
Request: req,
}
ready := make(chan struct{})
prw := &pipeResponseWriter{r, w, resp, ready}
go func() {
defer w.Close()
t.h.ServeHTTP(prw, req)
}()
<-ready
return resp, nil
}
type pipeResponseWriter struct {
r *io.PipeReader
w *io.PipeWriter
resp *http.Response
ready chan<- struct{}
}
func (w *pipeResponseWriter) Header() http.Header {
return w.resp.Header
}
func (w *pipeResponseWriter) Write(p []byte) (int, error) {
if w.ready != nil {
w.WriteHeader(http.StatusOK)
}
return w.w.Write(p)
}
func (w *pipeResponseWriter) WriteHeader(status int) {
if w.ready == nil {
// already called
return
}
w.resp.StatusCode = status
w.resp.Status = fmt.Sprintf("%d %s", status, http.StatusText(status))
close(w.ready)
w.ready = nil
}
英文:
Most of the behavior in net/http
can be mocked, extended, or altered. Although http.Client
is a concrete type that implements HTTP client semantics, all of its fields are exported and may be customized.
The Client.Transport
field, in particular, may be replaced to make the Client do anything from using custom protocols (such as ftp:// or file://) to connecting directly to local handlers (without generating HTTP protocol bytes or sending anything over the network).
The client functions, such as http.Get
, all utilize the exported http.DefaultClient
package variable (which you may modify), so code that utilizes these convenience functions does not, for example, have to be changed to call methods on a custom Client variable. Note that while it would be unreasonable to modify global behavior in a publicly-available library, it's very useful to do so in applications and tests (including library tests).
http://play.golang.org/p/afljO086iB contains a custom http.RoundTripper
that rewrites the request URL so that it'll be routed to a locally hosted httptest.Server
, and another example that directly passes the request to an http.Handler
, along with a custom http.ResponseWriter
implementation, in order to create an http.Response
. The second approach isn't as diligent as the first (it doesn't fill out as many fields in the Response value) but is more efficient, and should be compatible enough to work with most handlers and client callers.
The above-linked code is included below as well:
package main
import (
"fmt"
"io"
"log"
"net/http"
"net/http/httptest"
"net/url"
"os"
"path"
"strings"
)
func Handler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, "hello %s\n", path.Base(r.URL.Path))
}
func main() {
s := httptest.NewServer(http.HandlerFunc(Handler))
u, err := url.Parse(s.URL)
if err != nil {
log.Fatalln("failed to parse httptest.Server URL:", err)
}
http.DefaultClient.Transport = RewriteTransport{URL: u}
resp, err := http.Get("https://google.com/path-one")
if err != nil {
log.Fatalln("failed to send first request:", err)
}
fmt.Println("[First Response]")
resp.Write(os.Stdout)
fmt.Print("\n", strings.Repeat("-", 80), "\n\n")
http.DefaultClient.Transport = HandlerTransport{http.HandlerFunc(Handler)}
resp, err = http.Get("https://google.com/path-two")
if err != nil {
log.Fatalln("failed to send second request:", err)
}
fmt.Println("[Second Response]")
resp.Write(os.Stdout)
}
// RewriteTransport is an http.RoundTripper that rewrites requests
// using the provided URL's Scheme and Host, and its Path as a prefix.
// The Opaque field is untouched.
// If Transport is nil, http.DefaultTransport is used
type RewriteTransport struct {
Transport http.RoundTripper
URL *url.URL
}
func (t RewriteTransport) RoundTrip(req *http.Request) (*http.Response, error) {
// note that url.URL.ResolveReference doesn't work here
// since t.u is an absolute url
req.URL.Scheme = t.URL.Scheme
req.URL.Host = t.URL.Host
req.URL.Path = path.Join(t.URL.Path, req.URL.Path)
rt := t.Transport
if rt == nil {
rt = http.DefaultTransport
}
return rt.RoundTrip(req)
}
type HandlerTransport struct{ h http.Handler }
func (t HandlerTransport) RoundTrip(req *http.Request) (*http.Response, error) {
r, w := io.Pipe()
resp := &http.Response{
Proto: "HTTP/1.1",
ProtoMajor: 1,
ProtoMinor: 1,
Header: make(http.Header),
Body: r,
Request: req,
}
ready := make(chan struct{})
prw := &pipeResponseWriter{r, w, resp, ready}
go func() {
defer w.Close()
t.h.ServeHTTP(prw, req)
}()
<-ready
return resp, nil
}
type pipeResponseWriter struct {
r *io.PipeReader
w *io.PipeWriter
resp *http.Response
ready chan<- struct{}
}
func (w *pipeResponseWriter) Header() http.Header {
return w.resp.Header
}
func (w *pipeResponseWriter) Write(p []byte) (int, error) {
if w.ready != nil {
w.WriteHeader(http.StatusOK)
}
return w.w.Write(p)
}
func (w *pipeResponseWriter) WriteHeader(status int) {
if w.ready == nil {
// already called
return
}
w.resp.StatusCode = status
w.resp.Status = fmt.Sprintf("%d %s", status, http.StatusText(status))
close(w.ready)
w.ready = nil
}
答案2
得分: 2
你之所以会收到错误信息 http: TLS handshake error from 127.0.0.1:45678: tls: oversized record received with length 20037
是因为 HTTPS 需要一个域名(而不是 IP 地址)。SSL 证书是分配给域名的。
你可以使用自己的证书在 TLS 模式下启动 httptest 服务器。
cert, err := tls.LoadX509KeyPair("cert.pem", "key.pem")
if err != nil {
log.Panic("bad server certs: ", err)
}
certs := []tls.Certificate{cert}
server = httptest.NewUnstartedServer(router)
server.TLS = &tls.Config{Certificates: certs}
server.StartTLS()
serverPort = ":" + strings.Split(server.URL, ":")[2] // it's always https://127.0.0.1:<port>
server.URL = "https://sub.domain.com" + serverPort
为了提供有效的 SSL 证书进行连接,有以下选项:
- 不提供证书和密钥
- 提供自签名的证书和密钥
- 提供真实有效的证书和密钥
没有证书
如果你不提供自己的证书,那么将默认加载一个 example.com
的证书。
自签名证书
要创建一个测试证书,可以使用附带的自签名证书生成器 $GOROOT/src/crypto/tls/generate_cert.go --host "*.domain.name"
由于是自签名证书,你会收到 x509: certificate signed by unknown authority
的警告,所以你需要在 http.Transport 字段中添加以下内容,以跳过这些警告:
TLSClientConfig: &tls.Config{InsecureSkipVerify: true}
有效的真实证书
最后,如果你要使用真实的证书,将有效的证书和密钥保存在可以加载的位置。
关键在于使用 server.URL = "https://sub.domain.com"
来提供你自己的域名。
英文:
The reason you're getting the error http: TLS handshake error from 127.0.0.1:45678: tls: oversized record received with length 20037
is because https requires a domain name (not an IP Address). Domain names are SSL certificates are assigned to.
Start the httptest server in TLS mode with your own certs
cert, err := tls.LoadX509KeyPair("cert.pem", "key.pem")
if err != nil {
log.Panic("bad server certs: ", err)
}
certs := []tls.Certificate{cert}
server = httptest.NewUnstartedServer(router)
server.TLS = &tls.Config{Certificates: certs}
server.StartTLS()
serverPort = ":" + strings.Split(server.URL, ":")[2] // it's always https://127.0.0.1:<port>
server.URL = "https://sub.domain.com" + serverPort
To provide a valid SSL certificate for a connection are the options of:
- Not supplying a cert and key
- Supplying a self-signed cert and key
- Supplying a real valid cert and key
No Cert
If you don't supply your own cert, then an example.com
cert is loaded as default.
Self-Signed Cert
To create a testing cert can use the included self-signed cert generator at $GOROOT/src/crypto/tls/generate_cert.go --host "*.domain.name"
You'll get x509: certificate signed by unknown authority
warnings because it's self-signed so you'll need to have your client skip those warnings, by adding the following to your http.Transport field:
TLSClientConfig: &tls.Config{InsecureSkipVerify: true}
Valid Real Cert
Finally, if you're going to use a real cert, then save the valid cert and key where they can be loaded.
The key here is to use server.URL = https://sub.domain.com
to supply your own domain.
答案3
得分: 1
从Go 1.9+开始,你可以在httptest
包中使用func (s *Server) Client() *http.Client
:
> Client
返回一个配置好的HTTP客户端,用于向服务器发送请求。它配置为信任服务器的TLS测试证书,并在Server.Close
时关闭空闲连接。
以下是该包的示例代码:
package main
import (
"fmt"
"io"
"log"
"net/http"
"net/http/httptest"
)
func main() {
ts := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "Hello, client")
}))
defer ts.Close()
client := ts.Client()
res, err := client.Get(ts.URL)
if err != nil {
log.Fatal(err)
}
greeting, err := io.ReadAll(res.Body)
res.Body.Close()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s", greeting)
}
英文:
From Go 1.9+ you can use func (s *Server) Client() *http.Client
in the httptest
package:
> Client returns an HTTP client configured for making requests to the server. It is configured to trust the server's TLS test certificate and will close its idle connections on Server.Close.
Example from the package:
package main
import (
"fmt"
"io"
"log"
"net/http"
"net/http/httptest"
)
func main() {
ts := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
fmt.Fprintln(w, "Hello, client")
}))
defer ts.Close()
client := ts.Client()
res, err := client.Get(ts.URL)
if err != nil {
log.Fatal(err)
}
greeting, err := io.ReadAll(res.Body)
res.Body.Close()
if err != nil {
log.Fatal(err)
}
fmt.Printf("%s", greeting)
}
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论