英文:
Golang HTTP Concurrent Requests POST EOF
问题
我正在运行一个使用Go语言编写的小型客户端。目前我们遇到了一些误报问题,似乎是由于client.Do()
在num>=1000
时返回了EOF错误。
以下是我的代码核心部分:
func DoCreate(js string, cli *http.Client) {
url2 := "http://xvz.myserver.com:9000/path/create"
postBytesReader := bytes.NewReader([]byte(js))
request, _ := http.NewRequest("POST", url2, postBytesReader)
resp, err := cli.Do(request)
if err != nil {
fmt.Println(err) // Post http://xvz.myserver.com:9000/path/create: EOF
return
}
body, _ := ioutil.ReadAll(resp.Body)
fmt.Println(string(body))
defer resp.Body.Close()
}
func main() {
client := &http.Client{
CheckRedirect: nil,
}
for i := 0; i < num; i++ {
var pq CreateReqInfo
pq.Uid = strconv.Itoa(bid + i)
petstr, _ := json.Marshal(pq)
go DoCreate(string(petstr), client)
}
}
问题是关于文件句柄数量或最大连接数吗?
英文:
I'm running a small client in Go. We're getting a number of false alarms at the moment, which seem to come down to client.Do()
returning an EOF error when num>=1000
.
This is the essence of my code:
func DoCreate(js string, cli *http.Client) {
url2 := "http://xvz.myserver.com:9000/path/create"
postBytesReader := bytes.NewReader([]byte(js))
request, _ := http.NewRequest("POST", url2, postBytesReader)
resp, err := cli.Do(request)
if err != nil {
fmt.Println(err) // Post http://xvz.myserver.com:9000/path/create: EOF
return
}
body, _ := ioutil.ReadAll(resp.Body)
fmt.Println(string(body))
defer resp.Body.Close()
}
func main() {
client := &http.Client{
CheckRedirect: nil,
}
for i := 0; i < num; i++ {
var pq CreateReqInfo
pq.Uid = strconv.Itoa(bid + i)
petstr, _ := json.Marshal(pq)
go DoCreate(string(petstr), client)
}
}
Is problem about number of file handles or max connections?
答案1
得分: 20
我觉得你可能遇到了这个答案中描述的问题。基本上,Go的http客户端会尝试重用连接,除非你在你的Transport
或请求本身中明确指示不这样做:
request.Close = true
当服务器在没有在响应中使用Connection: close
头部指示的情况下关闭连接时,这就成为一个问题。net.http
代码假设连接仍然打开,所以下一个尝试使用该连接的请求会遇到连接被关闭的EOF
错误。
你需要检查在第1000个请求周围连接上发生了什么。接收服务器是否设置了每个客户端的连接数限制?你的代码是否遇到了连接超时?无论哪种情况,如果连接以Go的Client
代码无法预测的方式被服务器关闭,那么你就会遇到EOF
错误。
英文:
It looks to me that you might be having the issue described in this answer. Basically, the Go http client will try to reuse connections unless you've specifically indicated that it shouldn't either in the Transport
you've set your Client
instance to use or on the request itself with:
request.Close = true
This becomes a problem when the server on the other end closes the connection without indicating so with a Connection: close
header in a response. The net.http
code assumes that the connection is still open and so the next request that tries to use the connection next, encounters the EOF
from the connection being closed the other time.
You would need to check what is happening on the connection around the 1000th request. Is the receiving server set to limit the number of connections per client? Is your code running into a connection timeout? In either case, if it so happens the connection is closed by the server in a way that the Go Client
code can't predict, then you'll run into the EOF
.
答案2
得分: 1
首先,为什么你在循环内的DoCreate()
函数中每次都实例化一个新的客户端client := &http.Client{...
?客户端是可以被重复使用的,甚至可以并发使用,所以你可以在更全局的地方构建它,比如在main()
函数中。
其次,对我来说,这样的错误看起来像是由RoundTrip
产生的,所以可能是由服务器端引起的连接问题。你能否尝试使用模拟服务器进行测试?
最后,如果所有这些都没有帮助,那么是的,一些系统对每个net.Conn
所需的打开文件描述符数量有限制。这个限制只能在操作系统级别上消除。
英文:
- First, what for you instantiate new client each time
client := &http.Client{...
insideDoCreate()
inside a loop? Client can be
reusable even concurrently, so you can construct it more globally,
say inmain()
seems to me. - Then for me such error looks like
produced by RoundTrip, so by connection, so possibly from server site. Can you
test with mocking server? - Finally if all that would not help, then
yes, some systems have restriction on number of opened
FileDescriptors which everynet.Conn
wants to have own. This
restriction can be eliminated only on OS level.
答案3
得分: 1
EOF通常是由于服务器未返回完整的标头(包括两个CRLF
)或在标头完整之前连接被关闭而引起的。更有可能的是,您的服务器被并发请求过载,尽管您仍然应确保本地资源足够以处理您正在进行的并发请求的数量。如果num
足够大,您将耗尽某些资源。
尽管该错误并不真正具有描述性,但它并不比其他请求错误更令人担忧。它是一个错误条件,您应该像处理其他错误一样处理它。如果您想要确定,您可能需要对导致EOF
的连接进行数据包捕获。
英文:
The EOF is usually from a server not returning a complete header (including both CRLF
s), or the connection is closed before the header is complete. It's more likely that your overloading your server with concurrent requests, though you should still make sure you have adequate resources locally for the number of concurrent requests you're making. If num
is large enough, you're going to run out of something.
Though the error isn't really descriptive, it's not anything to worry about more so than any other request error. It's an error condition, and handle it like you would any others. If you want to know for certain, you'll likely have to have a packet capture of the connection that cause the EOF
.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论