英文:
How to set timeout for http.Get() requests in Golang?
问题
我正在使用Go语言编写一个URL获取器,并且有一个要获取的URL列表。我向每个URL发送http.Get()
请求并获取它们的响应。
resp,fetch_err := http.Get(url)
我该如何为每个Get请求设置自定义超时时间?(默认时间非常长,这使得我的获取器非常慢。)我希望我的获取器在大约40-45秒后超时,并返回“请求超时”,然后继续下一个URL。
我该如何实现这个?
英文:
I'm making a URL fetcher in Go and have a list of URLs to fetch. I send http.Get()
requests to each URL and obtain their response.
resp,fetch_err := http.Get(url)
How can I set a custom timeout for each Get request? (The default time is very long and that makes my fetcher really slow.) I want my fetcher to have a timeout of around 40-45 seconds after which it should return "request timed out" and move on to the next URL.
How can I achieve this?
答案1
得分: 372
显然在Go 1.3中,http.Client有Timeout字段
client := http.Client{
Timeout: 5 * time.Second,
}
client.Get(url)
这对我来说已经解决了问题。
英文:
Apparently in Go 1.3 http.Client has Timeout field
client := http.Client{
Timeout: 5 * time.Second,
}
client.Get(url)
That's done the trick for me.
答案2
得分: 56
你需要使用自己的Client和自己的Transport来设置,该Transport使用一个包装DialTimeout的自定义Dial函数。
类似于(完全未经测试)这样:
var timeout = time.Duration(2 * time.Second)
func dialTimeout(network, addr string) (net.Conn, error) {
return net.DialTimeout(network, addr, timeout)
}
func main() {
transport := http.Transport{
Dial: dialTimeout,
}
client := http.Client{
Transport: &transport,
}
resp, err := client.Get("http://some.url")
}
英文:
You need to set up your own Client with your own Transport which uses a
custom Dial function which wraps around DialTimeout.
Something like (completely untested) this:
var timeout = time.Duration(2 * time.Second)
func dialTimeout(network, addr string) (net.Conn, error) {
return net.DialTimeout(network, addr, timeout)
}
func main() {
transport := http.Transport{
Dial: dialTimeout,
}
client := http.Client{
Transport: &transport,
}
resp, err := client.Get("http://some.url")
}
答案3
得分: 49
如果您想按请求执行此操作,为简洁起见忽略错误处理:
ctx, cncl := context.WithTimeout(context.Background(), time.Second*3)
defer cncl()
req, _ := http.NewRequestWithContext(ctx, http.MethodGet, "https://google.com", nil)
resp, _ := http.DefaultClient.Do(req)
英文:
If you want to do it per request, err handling ignored for brevity:
ctx, cncl := context.WithTimeout(context.Background(), time.Second*3)
defer cncl()
req, _ := http.NewRequestWithContext(ctx, http.MethodGet, "https://google.com", nil)
resp, _ := http.DefaultClient.Do(req)
答案4
得分: 36
要添加到Volker的答案中,如果您还想设置读/写超时,除了连接超时之外,可以按照以下方式进行操作
package httpclient
import (
"net"
"net/http"
"time"
)
func TimeoutDialer(cTimeout time.Duration, rwTimeout time.Duration) func(net, addr string) (c net.Conn, err error) {
return func(netw, addr string) (net.Conn, error) {
conn, err := net.DialTimeout(netw, addr, cTimeout)
if err != nil {
return nil, err
}
conn.SetDeadline(time.Now().Add(rwTimeout))
return conn, nil
}
}
func NewTimeoutClient(connectTimeout time.Duration, readWriteTimeout time.Duration) *http.Client {
return &http.Client{
Transport: &http.Transport{
Dial: TimeoutDialer(connectTimeout, readWriteTimeout),
},
}
}
这段代码经过测试,在生产环境中运行良好。完整的代码片段和测试代码可以在此处找到
https://gist.github.com/dmichael/5710968
请注意,由于conn.SetDeadline
引用了从time.Now()
开始的未来时间点,所以您需要为每个请求创建一个新的客户端。
英文:
To add to Volker's answer, if you would also like to set the read/write timeout in addition to the connect timeout you can do something like the following
package httpclient
import (
"net"
"net/http"
"time"
)
func TimeoutDialer(cTimeout time.Duration, rwTimeout time.Duration) func(net, addr string) (c net.Conn, err error) {
return func(netw, addr string) (net.Conn, error) {
conn, err := net.DialTimeout(netw, addr, cTimeout)
if err != nil {
return nil, err
}
conn.SetDeadline(time.Now().Add(rwTimeout))
return conn, nil
}
}
func NewTimeoutClient(connectTimeout time.Duration, readWriteTimeout time.Duration) *http.Client {
return &http.Client{
Transport: &http.Transport{
Dial: TimeoutDialer(connectTimeout, readWriteTimeout),
},
}
}
This code is tested and is working in production. The full gist with tests is available here
https://gist.github.com/dmichael/5710968
Be aware that you will need to create a new client for each request because of the conn.SetDeadline
which references a point in the future from time.Now()
答案5
得分: 16
Go http
模块中有几个客户端超时设置,当前答案中有一些关于这些超时的示例。
这里有一张图片来说明客户端超时,参考《Go net/http超时完全指南》:
有两种方法可以设置HTTP请求的超时时间:
- http.Client
client := http.Client{
Timeout: 3 * time.Second,
}
resp, err := client.Do(req)
- Context
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, URL)
它们之间的区别是:
> - 使用上下文是针对每个请求的,而使用客户端超时可能会应用于传递给Do
方法的所有请求。
> - 如果您想为每个请求专门设置截止时间/超时时间
,则使用上下文;否则,如果您希望每个出站请求都有一个超时时间,则使用客户端超时就足够了。
英文:
There are several client-side timeouts in the Go http
module, and there are some samples of those timeouts on current answers.
Here is one image to illustrate the client-side timeout refer to The complete guide to Go net/http timeouts
There are two methods to set the timeout for HTTP request
- http.Client
client := http.Client{
Timeout: 3 * time.Second,
}
resp, err := client.Do(req)
- Context
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, URL)
The difference between them is
> - Using context is request specific while using the Client timeout might be applied to all request pass to Do
method client has.
> - If you want to specialize your deadline/timeout
to each request then use context, otherwise, if you want 1 timeout for every outbound request then using client timeout is enough.
答案6
得分: 12
一个快速而不太规范的方法:
http.DefaultTransport.(*http.Transport).ResponseHeaderTimeout = time.Second * 45
这会改变全局状态而没有任何协调。但对于你的URL获取器来说可能是可以的。否则,创建一个私有的http.RoundTripper
实例:
var myTransport http.RoundTripper = &http.Transport{
Proxy: http.ProxyFromEnvironment,
ResponseHeaderTimeout: time.Second * 45,
}
var myClient = &http.Client{Transport: myTransport}
resp, err := myClient.Get(url)
...
上面的内容没有经过测试。
英文:
A quick and dirty way:
http.DefaultTransport.(*http.Transport).ResponseHeaderTimeout = time.Second * 45
This is mutating global state w/o any coordination. Yet it might be possibly okay for your url fetcher. Otherwise create a private instance of http.RoundTripper
:
var myTransport http.RoundTripper = &http.Transport{
Proxy: http.ProxyFromEnvironment,
ResponseHeaderTimeout: time.Second * 45,
}
var myClient = &http.Client{Transport: myTransport}
resp, err := myClient.Get(url)
...
Nothing above was tested.
答案7
得分: 0
timeout := time.Duration(5 * time.Second)
transport := &http.Transport{Proxy: http.ProxyURL(proxyUrl), ResponseHeaderTimeout:timeout}
这可能有所帮助,但请注意ResponseHeaderTimeout
只在连接建立后开始计时。
英文:
timeout := time.Duration(5 * time.Second)
transport := &http.Transport{Proxy: http.ProxyURL(proxyUrl), ResponseHeaderTimeout:timeout}
This may help, but notice that ResponseHeaderTimeout
starts only after the connection is established.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论