Timeout in goroutines and Http requests

huangapple
go评论115阅读模式
英文:

Timeout in goroutines and Http requests

问题

我正在检查服务器的状态。服务器的休眠时间超过15秒，我正在检查超时情况。

  1. package main
  3. import (
  4. 	"fmt"
  5. 	"net/http"
  6. 	"time"
  7. )
  9. var urls = []string{
  10. 	"http://site-centos-64:8080/examples/abc1.jsp",
  11. }
  13. type HttpResponse struct {
  14. 	url      string
  15. 	response *http.Response
  16. 	err      error
  17. }
  19. var ch = make(chan *HttpResponse, 100) // buffered
  20. var count int
  22. func asyncHttpGets(urls []string) []*HttpResponse {
  23. 	responses := []*HttpResponse{}
  24. 	count := 0
  25. 	timeout := make(chan bool, 100)
  26. 	for i := 0; i < 500; i++ {
  27. 		go func() {
  28. 			for _, url := range urls {
  29. 				resp, err := http.Get(url)
  30. 				count++
  31. 				go func() {
  32. 					time.Sleep(1 * time.Second)
  33. 					timeout <- true
  34. 				}()
  35. 				ch <- &HttpResponse{url, resp, err}
  36. 				if err != nil {
  37. 					return
  38. 				}
  39. 				resp.Body.Close()
  40. 			}
  41. 		}()
  42. 	}
  43. 	for {
  44. 		select {
  45. 		case r := <-ch:
  46. 			responses = append(responses, r)
  47. 			if count == 500 {
  48. 				return responses
  49. 			}
  50. 		case <-timeout:
  51. 			fmt.Println("超时")
  52. 			if count == 500 {
  53. 				return responses
  54. 			}
  55. 		}
  56. 	}
  57. 	return responses
  58. }
  60. func main() {
  61. 	now := time.Now()
  62. 	results := asyncHttpGets(urls)
  63. 	for _, result := range results {
  64. 		fmt.Printf("%s 状态: %s\n", result.url, result.response.Status)
  65. 	}
  66. 	fmt.Println(time.Since(now))
  67. }

但是实际情况是，最初会打印出"超时"，但是最后150-200个请求显示为"200 OK"的状态，这是不应该的。而且当尝试执行1000次时，会显示"panic: runtime error: invalid memory address or nil pointer dereference"。

英文:

I am checking the status of a server. The server has a sleep of more than 15 seconds and I am checking for the timeout.

  1. package main
  2. import (
  3. &quot;fmt&quot;
  4. &quot;net/http&quot;
  5. &quot;time&quot;
  6. )
  7. var urls = []string{
  8. &quot;http://site-centos-64:8080/examples/abc1.jsp&quot;,
  9. }
  10. type HttpResponse struct {
  11. url      string
  12. response *http.Response
  13. err      error
  14. }
  15. var ch = make(chan *HttpResponse, 100) // buffered
  16. var count int	
  17. func asyncHttpGets(urls []string) []*HttpResponse {
  18. responses := []*HttpResponse{}
  19. count:=0
  20. timeout := make(chan bool, 100)
  21. for i:=0;i&lt;500;i++{
  22. go func(){
  23. for _, url := range urls {
  24. resp, err := http.Get(url)
  25. count++;
  26. go func() {
  27. time.Sleep(1 * time.Second)
  28. timeout &lt;- true
  29. }()	
  30. ch &lt;- &amp;HttpResponse{url, resp, err}
  31. if err != nil {
  32. return
  33. }
  34. resp.Body.Close()
  35. }
  36. }()
  37. }
  38. for {
  39. select {
  40. case r := &lt;-ch:
  41. responses = append(responses, r)
  42. if count == 500 {
  43. return responses
  44. }
  45. case &lt;-timeout:
  46. fmt.Println(&quot;Timed Out&quot;)
  47. if count == 500 {
  48. return responses
  49. }
  50. }
  51. }
  52. return responses
  53. }
  54. func main() {
  55. now:=time.Now()
  56. results := asyncHttpGets(urls)
  57. for _, result := range results {
  58. fmt.Printf(&quot;%s status: %s\n&quot;, result.url,result.response.Status)
  59. }
  60. fmt.Println( time.Since(now))
  61. }

But what is happening is that initially it prints a "Timed Out" but the last 150-200 requests show the "200 OK" status which it should not. Also when trying to do it for a 1000 times it shows "panic: runtime error: invalid memory address or nil pointer dereference"

答案1

得分: 2

你在发送请求之前执行了resp, err := http.Get(url)，这会导致一切都被阻塞，直到响应准备好，然后同时在两个通道上发送。

只需将启动超时 goroutine 的代码移到发送请求之前的那一行，问题就会解决。即：

  1. for _, url := range urls {
  2.     go func() {
  3.         time.Sleep(1 * time.Second)
  4.         timeout <- true
  5.         count++
  6.     }()
  8.     resp, err := http.Get(url)
  9.     count++ //我想这是你的意思，对吗？
  10.     ch <- &HttpResponse{url, resp, err}
  11.     if err != nil {
  12.         return
  13.     }
  14.     resp.Body.Close()
  15. }

顺便说一下，尝试使用原子递增来计数，也许可以使用一个 waitgroup，以及使用 time.After 通道来替代 sleep。

英文:

You are doing the resp, err := http.Get(url) before you initiate the timeout goroutine.
This will cause everything to block until the response is ready, then send on both channels simultaneously.

Just move starting the timeout goroutine to the line before sending the request, and it will be fine. i.e.:

  1.   for _, url := range urls {
  2. go func() {
  3. time.Sleep(1 * time.Second)
  4. timeout &lt;- true
  5. count++;
  6. }() 
  7. resp, err := http.Get(url)
  8. count++; //I think this is what you meant, right?
  9. ch &lt;- &amp;HttpResponse{url, resp, err}
  10. if err != nil {
  11. return
  12. }
  13. resp.Body.Close()
  14. }

BTW try to use atomic increments to count, and maybe use a waitgroup, and a time.After channel instead of sleep.

答案2

得分: 0

如果您想避免将并发逻辑与业务逻辑混合在一起，我写了这个库https://github.com/shomali11/parallelizer来帮助您解决这个问题。它封装了并发逻辑，因此您不必担心它。

所以在您的示例中：

  1. package main
  2. import (
  3. "github.com/shomali11/parallelizer"
  4. "fmt"
  5. )
  6. func main() {
  7. urls := []string{ ... }
  8. results = make([]*HttpResponse, len(urls)
  9. options := &Options{ Timeout: time.Second }
  10. group := parallelizer.NewGroup(options)
  11. for index, url := range urls {
  12. group.Add(func(index int, url string, results *[]*HttpResponse) {
  13. return func () {
  14. ...
  15. results[index] = &HttpResponse{url, response, err}
  16. }
  17. }(index, url, &results))
  18. }
  19. err := group.Run()
  20. fmt.Println("Done")
  21. fmt.Println(fmt.Sprintf("Results: %v", results))
  22. fmt.Printf("Error: %v", err) // nil if it completed, err if timed out
  23. }
英文:

If you would like to avoid mixing concurrency logic with business logic, I wrote this library https://github.com/shomali11/parallelizer to help you with that. It encapsulates the concurrency logic so you do not have to worry about it.

So in your example:

  1. package main
  2. import (
  3. &quot;github.com/shomali11/parallelizer&quot;
  4. &quot;fmt&quot;
  5. )
  6. func main() {
  7. urls := []string{ ... }
  8. results = make([]*HttpResponse, len(urls)
  9. options := &amp;Options{ Timeout: time.Second }
  10. group := parallelizer.NewGroup(options)
  11. for index, url := range urls {
  12. group.Add(func(index int, url string, results *[]*HttpResponse) {
  13. return func () {
  14. ...
  15. results[index] = &amp;HttpResponse{url, response, err}
  16. }
  17. }(index, url, &amp;results))
  18. }
  19. err := group.Run()
  20. fmt.Println(&quot;Done&quot;)
  21. fmt.Println(fmt.Sprintf(&quot;Results: %v&quot;, results))
  22. fmt.Printf(&quot;Error: %v&quot;, err) // nil if it completed, err if timed out
  23. }

huangapple
  • 本文由 发表于 2014年2月3日 14:17:28
  • 转载请务必保留本文链接：https://go.coder-hub.com/21520888.html
  • go
  • goroutine
  • timeout
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定