英文:
Reusing http connections in Go
问题
我目前在寻找一种在Go中进行HTTP POST时重用连接的方法。
我已经创建了一个传输和客户端,如下所示:
// 创建一个新的传输和HTTP客户端
tr := &http.Transport{}
client := &http.Client{Transport: tr}
然后,我将这个客户端指针传递给一个goroutine,该goroutine正在对同一个端点进行多次POST请求,如下所示:
r, err := client.Post(url, "application/json", post)
通过查看netstat,这似乎导致每个POST请求都产生一个新的连接,从而导致大量并发连接处于打开状态。
在这种情况下,正确的重用连接的方法是什么?
英文:
I'm currently struggling to find a way to reuse connections when making HTTP posts in Go.
I've created a transport and client like so:
// Create a new transport and HTTP client
tr := &http.Transport{}
client := &http.Client{Transport: tr}
I'm then passing this client pointer into a goroutine which is making multiple posts to the same endpoint like so:
r, err := client.Post(url, "application/json", post)
Looking at netstat this appears to be resulting in a new connection for every post resulting in a large number of concurrent connections being open.
What is the correct way to reuse connections in this case?
答案1
得分: 132
确保你读取完整的响应并调用Close()
。
例如:
res, _ := client.Do(req)
io.Copy(ioutil.Discard, res.Body)
res.Body.Close()
再次强调... 为了确保http.Client
连接的重用,请确保:
- 读取完整的响应(例如
ioutil.ReadAll(resp.Body)
) - 调用
Body.Close()
英文:
Ensure that you read until the response is complete AND call Close()
.
e.g.
res, _ := client.Do(req)
io.Copy(ioutil.Discard, res.Body)
res.Body.Close()
Again... To ensure http.Client
connection reuse be sure to:
- Read until Response is complete (i.e.
ioutil.ReadAll(resp.Body)
) - Call
Body.Close()
答案2
得分: 51
如果有人仍然在寻找如何做到这一点的答案,这就是我正在做的方式。
package main
import (
"bytes"
"io/ioutil"
"log"
"net/http"
"time"
)
func httpClient() *http.Client {
client := &http.Client{
Transport: &http.Transport{
MaxIdleConnsPerHost: 20,
},
Timeout: 10 * time.Second,
}
return client
}
func sendRequest(client *http.Client, method string) []byte {
endpoint := "https://httpbin.org/post"
req, err := http.NewRequest(method, endpoint, bytes.NewBuffer([]byte("Post this data")))
if err != nil {
log.Fatalf("Error Occured. %+v", err)
}
response, err := client.Do(req)
if err != nil {
log.Fatalf("Error sending request to API endpoint. %+v", err)
}
// Close the connection to reuse it
defer response.Body.Close()
body, err := ioutil.ReadAll(response.Body)
if err != nil {
log.Fatalf("Couldn't parse response body. %+v", err)
}
return body
}
func main() {
c := httpClient()
response := sendRequest(c, http.MethodPost)
log.Println("Response Body:", string(response))
}
Go Playground: https://play.golang.org/p/cYWdFu0r62e
总之,我正在创建一个不同的方法来创建一个HTTP客户端,并将其分配给一个变量,然后使用它来发出请求。
请注意
defer response.Body.Close()
这将在请求完成后关闭连接,并且您可以多次重用客户端。
如果您想在循环中发送请求,请在循环中调用发送请求的函数。
如果您想更改客户端传输配置中的任何内容,比如添加代理配置,请更改客户端配置。
希望这对某人有所帮助。
英文:
If anyone is still finding answers on how to do it, this is how I am doing it.
package main
import (
"bytes"
"io/ioutil"
"log"
"net/http"
"time"
)
func httpClient() *http.Client {
client := &http.Client{
Transport: &http.Transport{
MaxIdleConnsPerHost: 20,
},
Timeout: 10 * time.Second,
}
return client
}
func sendRequest(client *http.Client, method string) []byte {
endpoint := "https://httpbin.org/post"
req, err := http.NewRequest(method, endpoint, bytes.NewBuffer([]byte("Post this data")))
if err != nil {
log.Fatalf("Error Occured. %+v", err)
}
response, err := client.Do(req)
if err != nil {
log.Fatalf("Error sending request to API endpoint. %+v", err)
}
// Close the connection to reuse it
defer response.Body.Close()
body, err := ioutil.ReadAll(response.Body)
if err != nil {
log.Fatalf("Couldn't parse response body. %+v", err)
}
return body
}
func main() {
c := httpClient()
response := sendRequest(c, http.MethodPost)
log.Println("Response Body:", string(response))
}
Go Playground: https://play.golang.org/p/cYWdFu0r62e
In summary, I am creating a different method to create an HTTP client and assigning it to a variable, and then using it to make requests.
Note the
defer response.Body.Close()
This will close the connection after the request is complete at the end of the function execution and you can reuse the client as many times.
If you want to send a request in a loop call the function that sends the request in a loop.
If you want to change anything in the client transport configuration, like add proxy config, make a change in the client config.
Hope this will help someone.
答案3
得分: 41
编辑:这更多是给那些为每个请求构建传输和客户端的人的一个注意事项。
编辑2:将链接更改为godoc。
Transport
是保存连接以供重用的结构体;请参阅https://godoc.org/net/http#Transport(“默认情况下,Transport会缓存连接以供将来重用。”)
因此,如果您为每个请求创建一个新的Transport,它将每次都创建新的连接。在这种情况下,解决方案是在客户端之间共享一个Transport实例。
英文:
Edit: This is more of a note for people that construct a Transport and Client for every request.
Edit2: Changed link to godoc.
Transport
is the struct that holds connections for re-use; see https://godoc.org/net/http#Transport ("By default, Transport caches connections for future re-use.")
So if you create a new Transport for each request, it will create new connections each time. In this case the solution is to share the one Transport instance between clients.
答案4
得分: 13
我记得,默认的客户端确实会重用连接。你是否关闭了response?
> 当读取完毕后,调用者应该关闭resp.Body。如果不关闭resp.Body,客户端的底层RoundTripper(通常是Transport)可能无法重用持久的TCP连接来处理后续的“keep-alive”请求。
英文:
IIRC, the default client does reuse connections. Are you closing the response?
> Callers should close resp.Body when done reading from it. If resp.Body is not closed, the Client's underlying RoundTripper (typically Transport) may not be able to re-use a persistent TCP connection to the server for a subsequent "keep-alive" request.
答案5
得分: 5
关于Body
// 关闭Body是调用者的责任。如果Body没有被完全读取和关闭, 默认的HTTP客户端的传输可能不会重用HTTP/1.x的“keep-alive”TCP连接。
因此,如果您想重用TCP连接,每次在读取完毕后都必须关闭Body。
此外,使用defer
,您可以确保在所有操作完成后调用Body.Close()
。
建议使用以下方式定义一个名为ReadBody的函数。
package main
import (
"fmt"
"io"
"io/ioutil"
"net/http"
"time"
)
func main() {
req, err := http.NewRequest(http.MethodGet, "https://github.com", nil)
if err != nil {
fmt.Println(err.Error())
return
}
client := &http.Client{}
i := 0
for {
resp, err := client.Do(req)
if err != nil {
fmt.Println(err.Error())
return
}
_, _ = readBody(resp.Body)
fmt.Println("done ", i)
time.Sleep(5 * time.Second)
}
}
func readBody(readCloser io.ReadCloser) ([]byte, error) {
defer readCloser.Close()
body, err := ioutil.ReadAll(readCloser)
if err != nil {
return nil, err
}
return body, nil
}
并且不要像下面这样调用Close:
res, _ := client.Do(req)
io.Copy(ioutil.Discard, res.Body) // 如果io.Copy发生恐慌,res.Body.Close()将不会被调用。
res.Body.Close()
英文:
about Body
// It is the caller's responsibility to
// close Body. The default HTTP client's Transport may not
// reuse HTTP/1.x "keep-alive" TCP connections if the Body is
// not read to completion and closed.
So if you want to reuse TCP connections, you have to close Body every time after read to completion.
Also, with defer
, you can make sure Body.Close()
is called after all.
An function ReadBody(io.ReadCloser) is suggested like this.
package main
import (
"fmt"
"io"
"io/ioutil"
"net/http"
"time"
)
func main() {
req, err := http.NewRequest(http.MethodGet, "https://github.com", nil)
if err != nil {
fmt.Println(err.Error())
return
}
client := &http.Client{}
i := 0
for {
resp, err := client.Do(req)
if err != nil {
fmt.Println(err.Error())
return
}
_, _ = readBody(resp.Body)
fmt.Println("done ", i)
time.Sleep(5 * time.Second)
}
}
func readBody(readCloser io.ReadCloser) ([]byte, error) {
defer readCloser.Close()
body, err := ioutil.ReadAll(readCloser)
if err != nil {
return nil, err
}
return body, nil
}
And don't call Close like below:
res, _ := client.Do(req)
io.Copy(ioutil.Discard, res.Body) // what if io.Copy panics, res.Body.Close() will not called.
res.Body.Close()
答案6
得分: 3
另一种处理init()
的方法是使用单例方法来获取HTTP客户端。通过使用sync.Once,您可以确保所有请求都只使用一个实例。
var (
once sync.Once
netClient *http.Client
)
func newNetClient() *http.Client {
once.Do(func() {
var netTransport = &http.Transport{
Dial: (&net.Dialer{
Timeout: 2 * time.Second,
}).Dial,
TLSHandshakeTimeout: 2 * time.Second,
}
netClient = &http.Client{
Timeout: time.Second * 2,
Transport: netTransport,
}
})
return netClient
}
func yourFunc() {
URL := "local.dev"
req, err := http.NewRequest("POST", URL, nil)
response, err := newNetClient().Do(req)
// ...
}
英文:
Another approach to init()
is to use a singleton method to get the http client. By using sync.Once you can be sure that only one instance will be used on all your requests.
var (
once sync.Once
netClient *http.Client
)
func newNetClient() *http.Client {
once.Do(func() {
var netTransport = &http.Transport{
Dial: (&net.Dialer{
Timeout: 2 * time.Second,
}).Dial,
TLSHandshakeTimeout: 2 * time.Second,
}
netClient = &http.Client{
Timeout: time.Second * 2,
Transport: netTransport,
}
})
return netClient
}
func yourFunc(){
URL := "local.dev"
req, err := http.NewRequest("POST", URL, nil)
response, err := newNetClient().Do(req)
// ...
}
答案7
得分: 2
这里缺少的是“goroutine”这个东西。Transport有自己的连接池,默认情况下,该池中的每个连接都会被重用(如果请求体已完全读取并关闭),但如果有多个goroutine发送请求,将会创建新的连接(池中的所有连接都忙碌,将会创建新的连接)。为了解决这个问题,您需要限制每个主机的最大连接数:Transport.MaxConnsPerHost
(https://golang.org/src/net/http/transport.go#L205)。
可能您还想设置IdleConnTimeout
和/或ResponseHeaderTimeout
。
英文:
The missing point here is the "goroutine" thing. Transport has its own connection pool, by default each connection in that pool is reused (if body is fully read and closed) but if several goroutines are sending requests, new connections will be created (the pool has all connections busy and will create new ones). To solve that you will need to limit the maximum number of connections per host: Transport.MaxConnsPerHost
(https://golang.org/src/net/http/transport.go#L205).
Probably you also want to setup IdleConnTimeout
and/or ResponseHeaderTimeout
.
答案8
得分: 0
你应该在你的http.Client
中显式设置MaxConnsPerHost
。Transport
会重用TCP连接,但你应该限制MaxConnsPerHost
(默认为0表示无限制)。
func init() {
// 单例http.Client
httpClient = createHTTPClient()
}
// 为了重用连接创建http.Client
func createHTTPClient() *http.Client {
client := &http.Client{
Transport: &http.Transport{
MaxConnsPerHost: 1,
// 其他选项字段
},
Timeout: time.Duration(RequestTimeout) * time.Second,
}
return client
}
英文:
https://golang.org/src/net/http/transport.go#L196
you should set MaxConnsPerHost
explicitly to your http.Client
. Transport
does reuse the TCP connection, but you should limit the MaxConnsPerHost
(default 0 means no limit).
func init() {
// singleton http.Client
httpClient = createHTTPClient()
}
// createHTTPClient for connection re-use
func createHTTPClient() *http.Client {
client := &http.Client{
Transport: &http.Transport{
MaxConnsPerHost: 1,
// other option field
},
Timeout: time.Duration(RequestTimeout) * time.Second,
}
return client
}
答案9
得分: 0
这是一个非常有用的功能,用于GO的HTTP调用,您可以保持连接活动并重用此连接。
var (
respReadLimit = int64(4096)
)
// 尝试读取响应体,以便我们可以重用此连接。
func (c *Client) drainBody(body io.ReadCloser) error {
defer body.Close()
_, err := io.Copy(ioutil.Discard, io.LimitReader(body, respReadLimit))
if err != nil {
return err
}
return nil
}
英文:
It is very useful function for GO http call, you can keep connection alive and resue this connection.
var (
respReadLimit = int64(4096)
)
// Try to read the response body so we can reuse this connection.
func (c *Client) drainBody(body io.ReadCloser) error {
defer body.Close()
_, err := io.Copy(ioutil.Discard, io.LimitReader(body, respReadLimit))
if err != nil {
return err
}
return nil
}
答案10
得分: -3
有两种可能的方法:
-
使用一个内部重用和管理与每个请求相关联的文件描述符的库。Http Client在内部执行相同的操作,但是您可以控制打开多少个并发连接以及如何管理资源。如果您感兴趣,请查看netpoll实现,它在内部使用epoll/kqueue来管理它们。
-
更简单的方法是,不使用连接池,而是为您的goroutine创建一个工作池。这将是一种简单且更好的解决方案,不会干扰您当前的代码库,并且只需要进行一些小的更改。
让我们假设您需要在收到请求后进行n个POST请求。您可以使用通道来实现这一点。
或者,您可以使用第三方库。例如:
https://github.com/ivpusic/grpool
英文:
There are two possible ways:
-
Use a library that internally reuses and manages the file descriptors, associated with each requests. Http Client does the same thing internally, but then you would have the control over how many concurrent connections to open, and how to manage your resources. If you are interested, look at the netpoll implementation, which internally uses epoll/kqueue to manage them.
-
The easy one would be, instead of pooling network connections, create a worker pool, for your goroutines. This would be easy, and
better solution, that would not hinder with your current codebase,
and would require minor changes.
Let's assume you need to make n POST request, after you recieve a request.
You could use channels, to implement this.
Or, simply you could use third party libraries.
Like:
https://github.com/ivpusic/grpool
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论