如何在Golang中为http.Get()请求设置超时时间?

huangapple go评论103阅读模式
英文:

How to set timeout for http.Get() requests in Golang?

问题

我正在使用Go语言编写一个URL获取器,并且有一个要获取的URL列表。我向每个URL发送http.Get()请求并获取它们的响应。

resp,fetch_err := http.Get(url)

我该如何为每个Get请求设置自定义超时时间?(默认时间非常长,这使得我的获取器非常慢。)我希望我的获取器在大约40-45秒后超时,并返回“请求超时”,然后继续下一个URL。

我该如何实现这个?

英文:

I'm making a URL fetcher in Go and have a list of URLs to fetch. I send http.Get() requests to each URL and obtain their response.

resp,fetch_err := http.Get(url)

How can I set a custom timeout for each Get request? (The default time is very long and that makes my fetcher really slow.) I want my fetcher to have a timeout of around 40-45 seconds after which it should return "request timed out" and move on to the next URL.

How can I achieve this?

答案1

得分: 372

显然在Go 1.3中,http.Client有Timeout字段

client := http.Client{
	Timeout: 5 * time.Second,
}
client.Get(url)

这对我来说已经解决了问题。

英文:

Apparently in Go 1.3 http.Client has Timeout field

client := http.Client{
	Timeout: 5 * time.Second,
}
client.Get(url)

That's done the trick for me.

答案2

得分: 56

你需要使用自己的Client和自己的Transport来设置,该Transport使用一个包装DialTimeout的自定义Dial函数。

类似于(完全未经测试这样

var timeout = time.Duration(2 * time.Second)

func dialTimeout(network, addr string) (net.Conn, error) {
    return net.DialTimeout(network, addr, timeout)
}

func main() {
    transport := http.Transport{
        Dial: dialTimeout,
    }

    client := http.Client{
        Transport: &transport,
    }

    resp, err := client.Get("http://some.url")
}
英文:

You need to set up your own Client with your own Transport which uses a
custom Dial function which wraps around DialTimeout.

Something like (completely untested) this:

var timeout = time.Duration(2 * time.Second)

func dialTimeout(network, addr string) (net.Conn, error) {
	return net.DialTimeout(network, addr, timeout)
}

func main() {
	transport := http.Transport{
		Dial: dialTimeout,
	}

	client := http.Client{
		Transport: &transport,
	}

	resp, err := client.Get("http://some.url")
}

答案3

得分: 49

如果您想按请求执行此操作,为简洁起见忽略错误处理:

ctx, cncl := context.WithTimeout(context.Background(), time.Second*3)
defer cncl()

req, _ := http.NewRequestWithContext(ctx, http.MethodGet, "https://google.com", nil)

resp, _ := http.DefaultClient.Do(req)
英文:

If you want to do it per request, err handling ignored for brevity:

ctx, cncl := context.WithTimeout(context.Background(), time.Second*3)
defer cncl()

req, _ := http.NewRequestWithContext(ctx, http.MethodGet, "https://google.com", nil)

resp, _ := http.DefaultClient.Do(req)

答案4

得分: 36

要添加到Volker的答案中,如果您还想设置读/写超时,除了连接超时之外,可以按照以下方式进行操作

package httpclient

import (
    "net"
    "net/http"
    "time"
)

func TimeoutDialer(cTimeout time.Duration, rwTimeout time.Duration) func(net, addr string) (c net.Conn, err error) {
    return func(netw, addr string) (net.Conn, error) {
        conn, err := net.DialTimeout(netw, addr, cTimeout)
        if err != nil {
            return nil, err
        }
        conn.SetDeadline(time.Now().Add(rwTimeout))
        return conn, nil
    }
}

func NewTimeoutClient(connectTimeout time.Duration, readWriteTimeout time.Duration) *http.Client {
    
    return &http.Client{
        Transport: &http.Transport{
            Dial: TimeoutDialer(connectTimeout, readWriteTimeout),
        },
    }
}

这段代码经过测试,在生产环境中运行良好。完整的代码片段和测试代码可以在此处找到
https://gist.github.com/dmichael/5710968

请注意,由于conn.SetDeadline引用了从time.Now()开始的未来时间点,所以您需要为每个请求创建一个新的客户端。

英文:

To add to Volker's answer, if you would also like to set the read/write timeout in addition to the connect timeout you can do something like the following

package httpclient

import (
    "net"
    "net/http"
    "time"
)

func TimeoutDialer(cTimeout time.Duration, rwTimeout time.Duration) func(net, addr string) (c net.Conn, err error) {
    return func(netw, addr string) (net.Conn, error) {
        conn, err := net.DialTimeout(netw, addr, cTimeout)
        if err != nil {
            return nil, err
        }
        conn.SetDeadline(time.Now().Add(rwTimeout))
        return conn, nil
    }
}

func NewTimeoutClient(connectTimeout time.Duration, readWriteTimeout time.Duration) *http.Client {
    
    return &http.Client{
        Transport: &http.Transport{
            Dial: TimeoutDialer(connectTimeout, readWriteTimeout),
        },
    }
}

This code is tested and is working in production. The full gist with tests is available here
https://gist.github.com/dmichael/5710968

Be aware that you will need to create a new client for each request because of the conn.SetDeadline which references a point in the future from time.Now()

答案5

得分: 16

Go http模块中有几个客户端超时设置,当前答案中有一些关于这些超时的示例。

这里有一张图片来说明客户端超时,参考《Go net/http超时完全指南》
如何在Golang中为http.Get()请求设置超时时间?

有两种方法可以设置HTTP请求的超时时间:

  • http.Client
client := http.Client{
    Timeout: 3 * time.Second,
}
resp, err := client.Do(req)
  • Context
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, URL)

它们之间的区别是:

> - 使用上下文是针对每个请求的,而使用客户端超时可能会应用于传递给Do方法的所有请求
> - 如果您想为每个请求专门设置截止时间/超时时间,则使用上下文;否则,如果您希望每个出站请求都有一个超时时间,则使用客户端超时就足够了。

英文:

There are several client-side timeouts in the Go http module, and there are some samples of those timeouts on current answers.

Here is one image to illustrate the client-side timeout refer to The complete guide to Go net/http timeouts
如何在Golang中为http.Get()请求设置超时时间?

There are two methods to set the timeout for HTTP request

  • http.Client
client := http.Client{
    Timeout: 3 * time.Second,
}
resp, err := client.Do(req)
  • Context
ctx, cancel := context.WithTimeout(context.Background(), 3*time.Second)
defer cancel()
req, err := http.NewRequestWithContext(ctx, http.MethodGet, URL)

The difference between them is

> - Using context is request specific while using the Client timeout might be applied to all request pass to Do method client has.
> - If you want to specialize your deadline/timeout to each request then use context, otherwise, if you want 1 timeout for every outbound request then using client timeout is enough.

答案6

得分: 12

一个快速而不太规范的方法:

http.DefaultTransport.(*http.Transport).ResponseHeaderTimeout = time.Second * 45

这会改变全局状态而没有任何协调。但对于你的URL获取器来说可能是可以的。否则,创建一个私有的http.RoundTripper实例:

var myTransport http.RoundTripper = &http.Transport{
        Proxy:                 http.ProxyFromEnvironment,
        ResponseHeaderTimeout: time.Second * 45,
}

var myClient = &http.Client{Transport: myTransport}

resp, err := myClient.Get(url)
...

上面的内容没有经过测试。

英文:

A quick and dirty way:

http.DefaultTransport.(*http.Transport).ResponseHeaderTimeout = time.Second * 45

This is mutating global state w/o any coordination. Yet it might be possibly okay for your url fetcher. Otherwise create a private instance of http.RoundTripper:

var myTransport http.RoundTripper = &http.Transport{
        Proxy:                 http.ProxyFromEnvironment,
        ResponseHeaderTimeout: time.Second * 45,
}

var myClient = &http.Client{Transport: myTransport}

resp, err := myClient.Get(url)
...

Nothing above was tested.

答案7

得分: 0

timeout := time.Duration(5 * time.Second)
transport := &http.Transport{Proxy: http.ProxyURL(proxyUrl), ResponseHeaderTimeout:timeout}

这可能有所帮助,但请注意ResponseHeaderTimeout只在连接建立后开始计时。

英文:
timeout := time.Duration(5 * time.Second)
transport := &http.Transport{Proxy: http.ProxyURL(proxyUrl), ResponseHeaderTimeout:timeout}

This may help, but notice that ResponseHeaderTimeout starts only after the connection is established.

huangapple
  • 本文由 发表于 2013年6月3日 19:06:41
  • 转载请务必保留本文链接:https://go.coder-hub.com/16895294.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定