英文:
Why is Go HTTPS Client not reusing connections?
问题
我有一个HTTP客户端,它可以创建多个连接到主机。我想设置它可以连接到特定主机的最大连接数。在Go的request.Transport中没有这样的选项。我的代码如下:
package main
import (
"fmt"
"net/http"
"net/url"
)
const (
endpoint_url_fmt = "https://blah.com/api1?%s"
)
func main() {
transport := http.Transport{DisableKeepAlives: false}
outParams := url.Values{}
outParams.Set("method", "write")
outParams.Set("message", "BLAH")
for {
// Encode as part of URI.
outboundRequest, err := http.NewRequest(
"GET",
fmt.Sprintf(endpoint_url_fmt, outParams.Encode()),
nil,
)
outboundRequest.Close = false
_, err = transport.RoundTrip(outboundRequest)
if err != nil {
fmt.Println(err)
}
}
}
我期望这将创建一个连接。因为我在一个for循环中调用它。但是它却不断地创建无限多的连接。
而类似的使用requests库的Python代码只创建一个连接。
#!/usr/bin/env python
import requests
endpoint_url_fmt = "https://something.com/restserver.php"
params = {}
params['method'] = 'write'
params['category'] = category_errors_scuba
params['message'] = "blah"
while True:
r = requests.get(endpoint_url_fmt, params=params)
}
由于某种原因,Go代码没有重用HTTP连接。
编辑:
Go代码需要关闭请求体以重用连接。
resp, err := transport.RoundTrip(outboundRequest)
resp.Close() // 这允许连接被重用
英文:
I have an http client which creates multiple connections to the host. I want to set a maximum number of connections it can set to a particular host.
There are no such options in go's request.Transport.
My code looks like
package main
import (
"fmt"
"net/http"
"net/url"
)
const (
endpoint_url_fmt = "https://blah.com/api1?%s"
)
func main() {
transport := http.Transport{ DisableKeepAlives : false }
outParams := url.Values{}
outParams.Set("method", "write")
outParams.Set("message", "BLAH")
for {
// Encode as part of URI.
outboundRequest, err := http.NewRequest(
"GET",
fmt.Sprintf(endpoint_url_fmt, outParams.Encode()),
nil
)
outboundRequest.Close = false
_ , err = transport.RoundTrip(outboundRequest)
if err != nil {
fmt.Println(err)
}
}
}
I would expect this to create 1 connection. As I am calling it in a for-loop. But this keeps creating an infinite number of connections.
Where as similar python code using the requests library creates only one connection.
#!/usr/bin/env python
import requests
endpoint_url_fmt = "https://something.com/restserver.php"
params = {}
params['method'] = 'write'
params['category'] = category_errors_scuba
params['message'] = "blah"
while True:
r = requests.get(endpoint_url_fmt, params = params)
For some reason the go code is not reusing http connections.
EDIT :
The go code needs the body to be closed to reuse the connection.
resp , err = transport.RoundTrip(outboundRequest)
resp.Close() // This allows the connection to be reused
答案1
得分: 20
根据OP的进一步澄清, 默认的客户端确实会重用连接。
确保关闭response。
> 当读取完毕后,调用者应该关闭resp.Body。如果不关闭resp.Body,客户端的底层RoundTripper(通常是Transport)可能无法重用持久的TCP连接以进行后续的“keep-alive”请求。
此外,我发现在调用Close()之前,我还需要读取完整的响应。
例如:
res, _ := client.Do(req)
io.Copy(ioutil.Discard, res.Body)
res.Body.Close()
为了确保http.Client连接的重用,请确保做两件事:
- 读取完整的响应(即
ioutil.ReadAll(resp.Body)
) - 调用
Body.Close()
旧答案,对于速率限制很有用,但不是OP想要的:
我认为通过golang 1.1的http API无法设置最大连接数。这意味着如果不小心的话,你可能会因为大量的TCP连接(直到用尽文件描述符或其他资源)而自讨苦吃。
话虽如此,你可以通过time.Tick来限制对特定主机的调用速率(从而限制出站请求和连接)。
例如:
import "time"
每秒请求数 := 5
限制 := time.Tick(1000000000 / 每秒请求数)
for i := 0; i < 16; i += 1 {
<-限制
go 服务队列()
}
英文:
Based on further clarification from the OP. The default client does reuse connections.
Be sure to close the response.
> Callers should close resp.Body when done reading from it. If resp.Body is not closed, the Client's underlying RoundTripper (typically Transport) may not be able to re-use a persistent TCP connection to the server for a subsequent "keep-alive" request.
Additionally, I've found that I also needed to read until the response was complete before calling Close().
e.g.
res, _ := client.Do(req)
io.Copy(ioutil.Discard, res.Body)
res.Body.Close()
To ensure http.Client connection reuse be sure to do two things:
- Read until Response is complete (i.e.
ioutil.ReadAll(resp.Body)
) - Call
Body.Close()
Old answer, useful for rate limiting, but not what the OP was after:
I don't think setting max connections is possible via the golang 1.1 http APIs. This means you can shoot yourself in the foot with tons of TCP connections (until you run out of file descriptors or whatever) if you aren't careful.
That said, you could limit the rate at which you call the go routine for a particular host (and therefore outbound requests and connections) via time.Tick.
For example:
import "time"
requests_per_second := 5
throttle := time.Tick(1000000000 / requests_per_second)
for i := 0; i < 16; i += 1 {
<-throttle
go serveQueue()
}
答案2
得分: 1
在http.Transport
中有一些有趣的改进:
// DisableKeepAlives为true时,禁用HTTP keep-alives,
// 并且只会在单个HTTP请求中使用与服务器的连接。
//
// 这与同名的TCP keep-alives无关。
DisableKeepAlives bool
// ...
// MaxIdleConns控制所有主机上的最大空闲(keep-alive)连接数。
// 0表示无限制。
MaxIdleConns int // Go 1.7
// 如果MaxIdleConnsPerHost非零,则控制每个主机保持的最大空闲(keep-alive)连接数。
// 如果为零,则使用DefaultMaxIdleConnsPerHost。
MaxIdleConnsPerHost int
// MaxConnsPerHost可选地限制每个主机的总连接数,
// 包括拨号、活动和空闲状态的连接。超过限制时,拨号将被阻塞。
//
// 0表示无限制。
MaxConnsPerHost int // Go 1.11
英文:
There are interesting improvements in http.Transport
:
// DisableKeepAlives, if true, disables HTTP keep-alives and
// will only use the connection to the server for a single
// HTTP request.
//
// This is unrelated to the similarly named TCP keep-alives.
DisableKeepAlives bool
// ...
// MaxIdleConns controls the maximum number of idle (keep-alive)
// connections across all hosts. Zero means no limit.
MaxIdleConns int // Go 1.7
// MaxIdleConnsPerHost, if non-zero, controls the maximum idle
// (keep-alive) connections to keep per-host. If zero,
// DefaultMaxIdleConnsPerHost is used.
MaxIdleConnsPerHost int
// MaxConnsPerHost optionally limits the total number of
// connections per host, including connections in the dialing,
// active, and idle states. On limit violation, dials will block.
//
// Zero means no limit.
MaxConnsPerHost int // Go 1.11
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论