在Go中模拟HTTPS响应

huangapple go评论88阅读模式
英文:

Mocking HTTPS responses in Go

问题

我正在尝试为一个向 Web 服务发起请求的包编写测试。我遇到了问题,可能是由于我对 TLS 的理解不够。

目前我的测试代码大致如下:

func TestSimple() {
    server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json")
        w.WriteHeader(200)
        fmt.Fprintf(w, `{ "fake" : "json data here" }`)
    }))
    transport := &http.Transport{
        Proxy: func(req *http.Request) (*url.URL, error) {
            return url.Parse(server.URL)
        },
    }
    // Client 是我的包中发起请求的类型
    client := Client{
        c: http.Client{Transport: transport},
    }

    client.DoRequest() // ...
}

我的包中有一个包变量(我希望它是一个常量),用于查询 Web 服务的基本地址。它是一个 HTTPS URL。我上面创建的测试服务器是普通的 HTTP,没有 TLS。

默认情况下,我的测试失败,错误信息是 "tls: first record does not look like a TLS handshake."。

为了使其工作,我的测试在发起请求之前将包变量更改为普通的 HTTP URL。

有没有办法解决这个问题?我可以将包变量设置为常量(HTTPS),然后设置一个可以 "降级" 为非加密 HTTP 的 http.Transport,或者使用 httptest.NewTLSServer() 替代?

(当我尝试使用 NewTLSServer() 时,我得到的错误是 "http: TLS handshake error from 127.0.0.1:45678: tls: oversized record received with length 20037")

英文:

I'm trying to write tests for a package that makes requests to a web service. I'm running into issues probably due to my lack of understanding of TLS.

Currently my test looks something like this:

func TestSimple() {
    server := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        w.Header().Set("Content-Type", "application/json")
        w.WriteHeader(200)
        fmt.Fprintf(w, `{ "fake" : "json data here" }`)
    }))
    transport := &http.Transport{
        Proxy: func(req *http.Request) (*url.URL, error) {
            return url.Parse(server.URL)
        },
    }
    // Client is the type in my package that makes requests
    client := Client{
        c: http.Client{Transport: transport},
    }

    client.DoRequest() // ...
}

My package has a package variable (I'd like for it to be a constant..) for the base address of the web service to query. It is an https URL. The test server I created above is plain HTTP, no TLS.

By default, my test fails with the error "tls: first record does not look like a TLS handshake."

To get this to work, my tests change the package variable to a plain http URL instead of https before making the query.

Is there any way around this? Can I make the package variable a constant (https), and either set up a http.Transport that "downgrades" to unencrypted HTTP, or use httptest.NewTLSServer() instead?

(When I try to use NewTLSServer() I get "http: TLS handshake error from 127.0.0.1:45678: tls: oversized record received with length 20037")

答案1

得分: 16

net/http中的大部分行为都可以进行模拟、扩展或修改。虽然http.Client是一个实现HTTP客户端语义的具体类型,但它的所有字段都是可导出的,可以进行自定义。

特别是Client.Transport字段可以被替换,使得Client可以执行各种操作,从使用自定义协议(如ftp://或file://)到直接连接到本地处理程序(而不生成HTTP协议字节或通过网络发送任何内容)。

客户端函数,例如http.Get,都使用导出的http.DefaultClient包变量(您可以修改它),因此使用这些便利函数的代码不需要更改为在自定义的Client变量上调用方法。请注意,虽然在公开可用的库中修改全局行为是不合理的,但在应用程序和测试中(包括库测试)这样做非常有用。

http://play.golang.org/p/afljO086iB 包含一个自定义的http.RoundTripper,它重写请求URL,以便将其路由到本地托管的httptest.Server,以及另一个示例,它直接将请求传递给http.Handler,并使用自定义的http.ResponseWriter实现来创建http.Response。第二种方法不像第一种方法那样仔细(它没有填充Response值中的许多字段),但更高效,并且应该与大多数处理程序和客户端调用者兼容。

上面链接的代码也包含在下面:

package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
	"net/http/httptest"
	"net/url"
	"os"
	"path"
	"strings"
)

func Handler(w http.ResponseWriter, r *http.Request) {
	fmt.Fprintf(w, "hello %s\n", path.Base(r.URL.Path))
}

func main() {
	s := httptest.NewServer(http.HandlerFunc(Handler))
	u, err := url.Parse(s.URL)
	if err != nil {
		log.Fatalln("failed to parse httptest.Server URL:", err)
	}
	http.DefaultClient.Transport = RewriteTransport{URL: u}
	resp, err := http.Get("https://google.com/path-one")
	if err != nil {
		log.Fatalln("failed to send first request:", err)
	}
	fmt.Println("[First Response]")
	resp.Write(os.Stdout)

	fmt.Print("\n", strings.Repeat("-", 80), "\n\n")

	http.DefaultClient.Transport = HandlerTransport{http.HandlerFunc(Handler)}
	resp, err = http.Get("https://google.com/path-two")
	if err != nil {
		log.Fatalln("failed to send second request:", err)
	}
	fmt.Println("[Second Response]")
	resp.Write(os.Stdout)
}

// RewriteTransport is an http.RoundTripper that rewrites requests
// using the provided URL's Scheme and Host, and its Path as a prefix.
// The Opaque field is untouched.
// If Transport is nil, http.DefaultTransport is used
type RewriteTransport struct {
	Transport http.RoundTripper
	URL       *url.URL
}

func (t RewriteTransport) RoundTrip(req *http.Request) (*http.Response, error) {
	// note that url.URL.ResolveReference doesn't work here
	// since t.u is an absolute url
	req.URL.Scheme = t.URL.Scheme
	req.URL.Host = t.URL.Host
	req.URL.Path = path.Join(t.URL.Path, req.URL.Path)
	rt := t.Transport
	if rt == nil {
		rt = http.DefaultTransport
	}
	return rt.RoundTrip(req)
}

type HandlerTransport struct{ h http.Handler }

func (t HandlerTransport) RoundTrip(req *http.Request) (*http.Response, error) {
	r, w := io.Pipe()
	resp := &http.Response{
		Proto:      "HTTP/1.1",
		ProtoMajor: 1,
		ProtoMinor: 1,
		Header:     make(http.Header),
		Body:       r,
		Request:    req,
	}
	ready := make(chan struct{})
	prw := &pipeResponseWriter{r, w, resp, ready}
	go func() {
		defer w.Close()
		t.h.ServeHTTP(prw, req)
	}()
	<-ready
	return resp, nil
}

type pipeResponseWriter struct {
	r     *io.PipeReader
	w     *io.PipeWriter
	resp  *http.Response
	ready chan<- struct{}
}

func (w *pipeResponseWriter) Header() http.Header {
	return w.resp.Header
}

func (w *pipeResponseWriter) Write(p []byte) (int, error) {
	if w.ready != nil {
		w.WriteHeader(http.StatusOK)
	}
	return w.w.Write(p)
}

func (w *pipeResponseWriter) WriteHeader(status int) {
	if w.ready == nil {
		// already called
		return
	}
	w.resp.StatusCode = status
	w.resp.Status = fmt.Sprintf("%d %s", status, http.StatusText(status))
	close(w.ready)
	w.ready = nil
}
英文:

Most of the behavior in net/http can be mocked, extended, or altered. Although http.Client is a concrete type that implements HTTP client semantics, all of its fields are exported and may be customized.

The Client.Transport field, in particular, may be replaced to make the Client do anything from using custom protocols (such as ftp:// or file://) to connecting directly to local handlers (without generating HTTP protocol bytes or sending anything over the network).

The client functions, such as http.Get, all utilize the exported http.DefaultClient package variable (which you may modify), so code that utilizes these convenience functions does not, for example, have to be changed to call methods on a custom Client variable. Note that while it would be unreasonable to modify global behavior in a publicly-available library, it's very useful to do so in applications and tests (including library tests).

http://play.golang.org/p/afljO086iB contains a custom http.RoundTripper that rewrites the request URL so that it'll be routed to a locally hosted httptest.Server, and another example that directly passes the request to an http.Handler, along with a custom http.ResponseWriter implementation, in order to create an http.Response. The second approach isn't as diligent as the first (it doesn't fill out as many fields in the Response value) but is more efficient, and should be compatible enough to work with most handlers and client callers.

The above-linked code is included below as well:

package main
import (
&quot;fmt&quot;
&quot;io&quot;
&quot;log&quot;
&quot;net/http&quot;
&quot;net/http/httptest&quot;
&quot;net/url&quot;
&quot;os&quot;
&quot;path&quot;
&quot;strings&quot;
)
func Handler(w http.ResponseWriter, r *http.Request) {
fmt.Fprintf(w, &quot;hello %s\n&quot;, path.Base(r.URL.Path))
}
func main() {
s := httptest.NewServer(http.HandlerFunc(Handler))
u, err := url.Parse(s.URL)
if err != nil {
log.Fatalln(&quot;failed to parse httptest.Server URL:&quot;, err)
}
http.DefaultClient.Transport = RewriteTransport{URL: u}
resp, err := http.Get(&quot;https://google.com/path-one&quot;)
if err != nil {
log.Fatalln(&quot;failed to send first request:&quot;, err)
}
fmt.Println(&quot;[First Response]&quot;)
resp.Write(os.Stdout)
fmt.Print(&quot;\n&quot;, strings.Repeat(&quot;-&quot;, 80), &quot;\n\n&quot;)
http.DefaultClient.Transport = HandlerTransport{http.HandlerFunc(Handler)}
resp, err = http.Get(&quot;https://google.com/path-two&quot;)
if err != nil {
log.Fatalln(&quot;failed to send second request:&quot;, err)
}
fmt.Println(&quot;[Second Response]&quot;)
resp.Write(os.Stdout)
}
// RewriteTransport is an http.RoundTripper that rewrites requests
// using the provided URL&#39;s Scheme and Host, and its Path as a prefix.
// The Opaque field is untouched.
// If Transport is nil, http.DefaultTransport is used
type RewriteTransport struct {
Transport http.RoundTripper
URL       *url.URL
}
func (t RewriteTransport) RoundTrip(req *http.Request) (*http.Response, error) {
// note that url.URL.ResolveReference doesn&#39;t work here
// since t.u is an absolute url
req.URL.Scheme = t.URL.Scheme
req.URL.Host = t.URL.Host
req.URL.Path = path.Join(t.URL.Path, req.URL.Path)
rt := t.Transport
if rt == nil {
rt = http.DefaultTransport
}
return rt.RoundTrip(req)
}
type HandlerTransport struct{ h http.Handler }
func (t HandlerTransport) RoundTrip(req *http.Request) (*http.Response, error) {
r, w := io.Pipe()
resp := &amp;http.Response{
Proto:      &quot;HTTP/1.1&quot;,
ProtoMajor: 1,
ProtoMinor: 1,
Header:     make(http.Header),
Body:       r,
Request:    req,
}
ready := make(chan struct{})
prw := &amp;pipeResponseWriter{r, w, resp, ready}
go func() {
defer w.Close()
t.h.ServeHTTP(prw, req)
}()
&lt;-ready
return resp, nil
}
type pipeResponseWriter struct {
r     *io.PipeReader
w     *io.PipeWriter
resp  *http.Response
ready chan&lt;- struct{}
}
func (w *pipeResponseWriter) Header() http.Header {
return w.resp.Header
}
func (w *pipeResponseWriter) Write(p []byte) (int, error) {
if w.ready != nil {
w.WriteHeader(http.StatusOK)
}
return w.w.Write(p)
}
func (w *pipeResponseWriter) WriteHeader(status int) {
if w.ready == nil {
// already called
return
}
w.resp.StatusCode = status
w.resp.Status = fmt.Sprintf(&quot;%d %s&quot;, status, http.StatusText(status))
close(w.ready)
w.ready = nil
}

答案2

得分: 2

你之所以会收到错误信息 http: TLS handshake error from 127.0.0.1:45678: tls: oversized record received with length 20037 是因为 HTTPS 需要一个域名(而不是 IP 地址)。SSL 证书是分配给域名的。

你可以使用自己的证书在 TLS 模式下启动 httptest 服务器。

cert, err := tls.LoadX509KeyPair("cert.pem", "key.pem")
if err != nil {
    log.Panic("bad server certs: ", err)
}
certs := []tls.Certificate{cert}

server = httptest.NewUnstartedServer(router)
server.TLS = &tls.Config{Certificates: certs}
server.StartTLS()
serverPort = ":" + strings.Split(server.URL, ":")[2] // it's always https://127.0.0.1:<port>
server.URL = "https://sub.domain.com" + serverPort

为了提供有效的 SSL 证书进行连接,有以下选项:

  1. 不提供证书和密钥
  2. 提供自签名的证书和密钥
  3. 提供真实有效的证书和密钥

没有证书

如果你不提供自己的证书,那么将默认加载一个 example.com 的证书。

自签名证书

要创建一个测试证书,可以使用附带的自签名证书生成器 $GOROOT/src/crypto/tls/generate_cert.go --host "*.domain.name"

由于是自签名证书,你会收到 x509: certificate signed by unknown authority 的警告,所以你需要在 http.Transport 字段中添加以下内容,以跳过这些警告:

TLSClientConfig: &tls.Config{InsecureSkipVerify: true}

有效的真实证书

最后,如果你要使用真实的证书,将有效的证书和密钥保存在可以加载的位置。


关键在于使用 server.URL = "https://sub.domain.com" 来提供你自己的域名。

英文:

The reason you're getting the error http: TLS handshake error from 127.0.0.1:45678: tls: oversized record received with length 20037 is because https requires a domain name (not an IP Address). Domain names are SSL certificates are assigned to.

Start the httptest server in TLS mode with your own certs

cert, err := tls.LoadX509KeyPair(&quot;cert.pem&quot;, &quot;key.pem&quot;)
if err != nil {
log.Panic(&quot;bad server certs: &quot;, err)
}
certs := []tls.Certificate{cert}
server = httptest.NewUnstartedServer(router)
server.TLS = &amp;tls.Config{Certificates: certs}
server.StartTLS()
serverPort = &quot;:&quot; + strings.Split(server.URL, &quot;:&quot;)[2] // it&#39;s always https://127.0.0.1:&lt;port&gt;
server.URL = &quot;https://sub.domain.com&quot; + serverPort

To provide a valid SSL certificate for a connection are the options of:

  1. Not supplying a cert and key
  2. Supplying a self-signed cert and key
  3. Supplying a real valid cert and key

No Cert

If you don't supply your own cert, then an example.com cert is loaded as default.

Self-Signed Cert

To create a testing cert can use the included self-signed cert generator at $GOROOT/src/crypto/tls/generate_cert.go --host &quot;*.domain.name&quot;

You'll get x509: certificate signed by unknown authority warnings because it's self-signed so you'll need to have your client skip those warnings, by adding the following to your http.Transport field:

 TLSClientConfig: &amp;tls.Config{InsecureSkipVerify: true}

Valid Real Cert

Finally, if you're going to use a real cert, then save the valid cert and key where they can be loaded.


The key here is to use server.URL = https://sub.domain.com to supply your own domain.

答案3

得分: 1

从Go 1.9+开始,你可以在httptest包中使用func (s *Server) Client() *http.Client

> Client返回一个配置好的HTTP客户端,用于向服务器发送请求。它配置为信任服务器的TLS测试证书,并在Server.Close时关闭空闲连接。

以下是该包的示例代码:

package main

import (
	"fmt"
	"io"
	"log"
	"net/http"
	"net/http/httptest"
)

func main() {
	ts := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintln(w, "Hello, client")
	}))
	defer ts.Close()

	client := ts.Client()
	res, err := client.Get(ts.URL)
	if err != nil {
		log.Fatal(err)
	}

	greeting, err := io.ReadAll(res.Body)
	res.Body.Close()
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf("%s", greeting)
}

英文:

From Go 1.9+ you can use func (s *Server) Client() *http.Client in the httptest package:

> Client returns an HTTP client configured for making requests to the server. It is configured to trust the server's TLS test certificate and will close its idle connections on Server.Close.

Example from the package:

package main

import (
	&quot;fmt&quot;
	&quot;io&quot;
	&quot;log&quot;
	&quot;net/http&quot;
	&quot;net/http/httptest&quot;
)

func main() {
	ts := httptest.NewTLSServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		fmt.Fprintln(w, &quot;Hello, client&quot;)
	}))
	defer ts.Close()

	client := ts.Client()
	res, err := client.Get(ts.URL)
	if err != nil {
		log.Fatal(err)
	}

	greeting, err := io.ReadAll(res.Body)
	res.Body.Close()
	if err != nil {
		log.Fatal(err)
	}

	fmt.Printf(&quot;%s&quot;, greeting)
}

huangapple
  • 本文由 发表于 2015年1月11日 04:32:17
  • 转载请务必保留本文链接:https://go.coder-hub.com/27880930.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定