为什么这个并发的HTTP客户端会随机崩溃?

huangapple go评论71阅读模式
英文:

Why does this concurrent HTTP client randomly crash?

问题

我正在学习使用Go编写一个类似Apache的ab的HTTP测试客户端。下面的代码看起来相当简单:我创建了一定数量的goroutine,每个goroutine发送一部分HTTP请求并记录结果。我遍历resultChan通道并检查/记录每个结果。当消息数量增加时,例如100个,这个方法可以正常工作。然而,当我增加消息数量时,程序会挂起,htop显示该进程的VIRT为138G。

以下是相关的代码:

package main

import "net/http"
import "fmt"
import "time"

const (
    SUCCESS   = iota
    TOTAL = iota
    TIMEOUT = iota
    ERROR = iota
)

type Result struct {
    successful    int
    total         int
    timeouts      int
    errors        int
    duration      time.Duration
}

func makeRequests(url string, messages int, resultChan chan<- *http.Response) {
    for i := 0; i < messages; i++ {
        resp, _ := http.Get(url)
        if resp != nil {
            resultChan <- resp
        }
    }
}

func deployRequests(url string, threads int, messages int) *Result {
    results := new(Result)
    resultChan := make(chan *http.Response)
    start := time.Now()
    defer func() {
        fmt.Printf("%s\n", time.Since(start))
    }()
    for i := 0; i < threads; i++ {
        go makeRequests(url, (messages/threads) + 1, resultChan)
    }

    for response := range resultChan {
        if response.StatusCode != 200 {
            results.errors += 1
        } else {
            results.successful += 1
        }
        results.total += 1
        if results.total == messages {
            return results
        }
    }
    return results
}

func main() {
    results := deployRequests("http://www.google.com", 10, 1000)
    fmt.Printf("Total: %d\n", results.total)
    fmt.Printf("Successful: %d\n", results.successful)
    fmt.Printf("Error: %d\n", results.errors)
    fmt.Printf("Timeouts: %d\n", results.timeouts)
    fmt.Printf("%s", results.duration)
}

显然,代码中有一些缺失或愚蠢的地方(没有超时检查,通道是同步的等),但在修复这些问题之前,我想先让基本情况能够正常工作。请问,这个程序的编写方式导致了如此多的内存分配是什么原因呢?

据我所知,只有10个goroutine。如果每个HTTP请求创建一个goroutine,这是有道理的,那么如何在循环中执行会创建许多goroutine的操作呢?或者问题完全不相关?

英文:

I'm learning Go by writing an HTTP testing client like Apache's ab. The code below seems pretty straightforward: I create a configurable number of goroutines, each of which sends a portion of the overall HTTP requests and records the result. I iterate over the resultChan channel and inspect/record each result. This works find when the number of messages is, say, 100. When I increase the number of messages, however, it hangs and htop shows VIRT of 138G for the process.

Here's the code in question:

package main
import &quot;net/http&quot;
import &quot;fmt&quot;
import &quot;time&quot;
const (
SUCCESS   = iota
TOTAL = iota
TIMEOUT = iota
ERROR = iota
)
type Result struct {
successful    int
total         int
timeouts      int
errors        int
duration      time.Duration
}
func makeRequests(url string, messages int, resultChan chan&lt;- *http.Response) {
for i := 0; i &lt; messages; i++ {
resp, _ := http.Get(url)
if resp != nil {
resultChan &lt;- resp
}
}
}
func deployRequests(url string, threads int, messages int) *Result {
results := new (Result)
resultChan := make(chan *http.Response)
start := time.Now()
defer func() {
fmt.Printf(&quot;%s\n&quot;, time.Since(start))
}()
for i := 0; i &lt; threads; i++ {
go makeRequests(url, (messages/threads) + 1, resultChan)
}
for response := range resultChan {
if response.StatusCode != 200 {
results.errors += 1
} else {
results.successful += 1
}
results.total += 1
if results.total == messages {
return results
}
}
return results
}
func main () {
results := deployRequests(&quot;http://www.google.com&quot;, 10, 1000)
fmt.Printf(&quot;Total: %d\n&quot;, results.total)
fmt.Printf(&quot;Successful: %d\n&quot;, results.successful)
fmt.Printf(&quot;Error: %d\n&quot;, results.errors)
fmt.Printf(&quot;Timeouts: %d\n&quot;, results.timeouts)
fmt.Printf(&quot;%s&quot;, results.duration)
}

There are obviously some things missing or stupidly done (no timeout checking, channel is synchronous, etc) but I wanted to get the basic case working before fixing those. What is it about the program as written that causes so much memory allocation?

As far as I can tell, there are just 10 goroutines. If one is created per HTTP request, which would make sense, how does one perform operations that would create many goroutines in a loop? Or is the issue totally unrelated.

答案1

得分: 3

我认为导致程序出现问题的序列是:

  1. makeRequests 中的 http.Get 失败(连接被拒绝、请求超时等),返回一个 nil 响应和一个错误值。
  2. 错误被忽略,makeRequests 继续处理下一个请求。
  3. 如果发生任何错误,makeRequests 将少于预期数量的结果发送到 resultChan
  4. deployRequests 中的 for .. range .. chan 循环永远不会中断,因为 results.total 总是小于 messages

一个解决方法是:

如果 http.Get 返回一个错误值,将一个 nil 响应发送到 resultChan

resp, err := http.Get(url)
if err != nil {
    resultChan <- nil
} else if resp != nil {
    resultChan <- resp
}

deployRequests 中,如果 for 循环从 resultChan 读取到一个 nil 值,将其视为一个错误:

for response := range resultChan {
    if response == nil {
        results.errors += 1
    } else if response.StatusCode != 200 {
        // ...
}
英文:

I think the sequence leading to the hang is:

  1. http.Get in makeRequests fails (connection denied, request timeout, etc.), returning a nil response and an error value
  2. The error is ignored and makeRequests moves on to the next request
  3. If any errors occur, makeRequests posts less than the expected number of results to resultChan
  4. The for .. range .. chan loop in deployRequests never breaks because results.total is always less than messages

One workaround would be:

If http.Get returns an error value, post a nil response to resultChan:

    resp, err := http.Get(url)
if err != nil {
resultChan &lt;- nil
} else if resp != nil {
resultChan &lt;- resp
}

In deployRequests, if the for loop reads a nil value from resultChan, count that as an error:

for response := range resultChan {
if response == nil {
results.errors += 1
} else if response.StatusCode != 200 {
// ...

huangapple
  • 本文由 发表于 2014年3月20日 21:13:36
  • 转载请务必保留本文链接:https://go.coder-hub.com/22534113.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定