英文:
Why does this concurrent HTTP client randomly crash?
问题
我正在学习使用Go编写一个类似Apache的ab
的HTTP测试客户端。下面的代码看起来相当简单:我创建了一定数量的goroutine,每个goroutine发送一部分HTTP请求并记录结果。我遍历resultChan
通道并检查/记录每个结果。当消息数量增加时,例如100个,这个方法可以正常工作。然而,当我增加消息数量时,程序会挂起,htop显示该进程的VIRT为138G。
以下是相关的代码:
package main
import "net/http"
import "fmt"
import "time"
const (
SUCCESS = iota
TOTAL = iota
TIMEOUT = iota
ERROR = iota
)
type Result struct {
successful int
total int
timeouts int
errors int
duration time.Duration
}
func makeRequests(url string, messages int, resultChan chan<- *http.Response) {
for i := 0; i < messages; i++ {
resp, _ := http.Get(url)
if resp != nil {
resultChan <- resp
}
}
}
func deployRequests(url string, threads int, messages int) *Result {
results := new(Result)
resultChan := make(chan *http.Response)
start := time.Now()
defer func() {
fmt.Printf("%s\n", time.Since(start))
}()
for i := 0; i < threads; i++ {
go makeRequests(url, (messages/threads) + 1, resultChan)
}
for response := range resultChan {
if response.StatusCode != 200 {
results.errors += 1
} else {
results.successful += 1
}
results.total += 1
if results.total == messages {
return results
}
}
return results
}
func main() {
results := deployRequests("http://www.google.com", 10, 1000)
fmt.Printf("Total: %d\n", results.total)
fmt.Printf("Successful: %d\n", results.successful)
fmt.Printf("Error: %d\n", results.errors)
fmt.Printf("Timeouts: %d\n", results.timeouts)
fmt.Printf("%s", results.duration)
}
显然,代码中有一些缺失或愚蠢的地方(没有超时检查,通道是同步的等),但在修复这些问题之前,我想先让基本情况能够正常工作。请问,这个程序的编写方式导致了如此多的内存分配是什么原因呢?
据我所知,只有10个goroutine。如果每个HTTP请求创建一个goroutine,这是有道理的,那么如何在循环中执行会创建许多goroutine的操作呢?或者问题完全不相关?
英文:
I'm learning Go by writing an HTTP testing client like Apache's ab
. The code below seems pretty straightforward: I create a configurable number of goroutines, each of which sends a portion of the overall HTTP requests and records the result. I iterate over the resultChan
channel and inspect/record each result. This works find when the number of messages is, say, 100. When I increase the number of messages, however, it hangs and htop shows VIRT of 138G for the process.
Here's the code in question:
package main
import "net/http"
import "fmt"
import "time"
const (
SUCCESS = iota
TOTAL = iota
TIMEOUT = iota
ERROR = iota
)
type Result struct {
successful int
total int
timeouts int
errors int
duration time.Duration
}
func makeRequests(url string, messages int, resultChan chan<- *http.Response) {
for i := 0; i < messages; i++ {
resp, _ := http.Get(url)
if resp != nil {
resultChan <- resp
}
}
}
func deployRequests(url string, threads int, messages int) *Result {
results := new (Result)
resultChan := make(chan *http.Response)
start := time.Now()
defer func() {
fmt.Printf("%s\n", time.Since(start))
}()
for i := 0; i < threads; i++ {
go makeRequests(url, (messages/threads) + 1, resultChan)
}
for response := range resultChan {
if response.StatusCode != 200 {
results.errors += 1
} else {
results.successful += 1
}
results.total += 1
if results.total == messages {
return results
}
}
return results
}
func main () {
results := deployRequests("http://www.google.com", 10, 1000)
fmt.Printf("Total: %d\n", results.total)
fmt.Printf("Successful: %d\n", results.successful)
fmt.Printf("Error: %d\n", results.errors)
fmt.Printf("Timeouts: %d\n", results.timeouts)
fmt.Printf("%s", results.duration)
}
There are obviously some things missing or stupidly done (no timeout checking, channel is synchronous, etc) but I wanted to get the basic case working before fixing those. What is it about the program as written that causes so much memory allocation?
As far as I can tell, there are just 10 goroutines. If one is created per HTTP request, which would make sense, how does one perform operations that would create many goroutines in a loop? Or is the issue totally unrelated.
答案1
得分: 3
我认为导致程序出现问题的序列是:
makeRequests
中的http.Get
失败(连接被拒绝、请求超时等),返回一个nil
响应和一个错误值。- 错误被忽略,
makeRequests
继续处理下一个请求。 - 如果发生任何错误,
makeRequests
将少于预期数量的结果发送到resultChan
。 deployRequests
中的for .. range .. chan
循环永远不会中断,因为results.total
总是小于messages
。
一个解决方法是:
如果 http.Get
返回一个错误值,将一个 nil
响应发送到 resultChan
:
resp, err := http.Get(url)
if err != nil {
resultChan <- nil
} else if resp != nil {
resultChan <- resp
}
在 deployRequests
中,如果 for
循环从 resultChan
读取到一个 nil
值,将其视为一个错误:
for response := range resultChan {
if response == nil {
results.errors += 1
} else if response.StatusCode != 200 {
// ...
}
英文:
I think the sequence leading to the hang is:
http.Get
inmakeRequests
fails (connection denied, request timeout, etc.), returning anil
response and an error value- The error is ignored and
makeRequests
moves on to the next request - If any errors occur,
makeRequests
posts less than the expected number of results toresultChan
- The
for .. range .. chan
loop indeployRequests
never breaks becauseresults.total
is always less thanmessages
One workaround would be:
If http.Get
returns an error value, post a nil
response to resultChan
:
resp, err := http.Get(url)
if err != nil {
resultChan <- nil
} else if resp != nil {
resultChan <- resp
}
In deployRequests
, if the for
loop reads a nil value from resultChan
, count that as an error:
for response := range resultChan {
if response == nil {
results.errors += 1
} else if response.StatusCode != 200 {
// ...
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论