Golang HTTP并发请求POST EOF

huangapple go评论79阅读模式
英文:

Golang HTTP Concurrent Requests POST EOF

问题

我正在运行一个使用Go语言编写的小型客户端。目前我们遇到了一些误报问题,似乎是由于client.Do()num>=1000时返回了EOF错误。

以下是我的代码核心部分:

func DoCreate(js string, cli *http.Client) {

	url2 := "http://xvz.myserver.com:9000/path/create"

	postBytesReader := bytes.NewReader([]byte(js))
	request, _ := http.NewRequest("POST", url2, postBytesReader)

	resp, err := cli.Do(request)
	if err != nil {
		fmt.Println(err) // Post http://xvz.myserver.com:9000/path/create: EOF 
		return
	}
	body, _ := ioutil.ReadAll(resp.Body)
	fmt.Println(string(body))

	defer resp.Body.Close()
}

func main() {
    client := &http.Client{
		CheckRedirect: nil,
	}

	for i := 0; i < num; i++ {
		var pq CreateReqInfo
		pq.Uid = strconv.Itoa(bid + i)
		petstr, _ := json.Marshal(pq)
		go DoCreate(string(petstr), client)
	}
}

问题是关于文件句柄数量或最大连接数吗?

英文:

I'm running a small client in Go. We're getting a number of false alarms at the moment, which seem to come down to client.Do() returning an EOF error when num&gt;=1000.

This is the essence of my code:

func DoCreate(js string, cli *http.Client) {

	url2 := &quot;http://xvz.myserver.com:9000/path/create&quot;

	postBytesReader := bytes.NewReader([]byte(js))
	request, _ := http.NewRequest(&quot;POST&quot;, url2, postBytesReader)

	resp, err := cli.Do(request)
	if err != nil {
		fmt.Println(err) // Post http://xvz.myserver.com:9000/path/create: EOF 
		return
	}
	body, _ := ioutil.ReadAll(resp.Body)
	fmt.Println(string(body))

	defer resp.Body.Close()
}

func main() {
    client := &amp;http.Client{
		CheckRedirect: nil,
	}

	for i := 0; i &lt; num; i++ {
		var pq CreateReqInfo
		pq.Uid = strconv.Itoa(bid + i)
		petstr, _ := json.Marshal(pq)
		go DoCreate(string(petstr), client)
	}
}

Is problem about number of file handles or max connections?

答案1

得分: 20

我觉得你可能遇到了这个答案中描述的问题。基本上,Go的http客户端会尝试重用连接,除非你在你的Transport或请求本身中明确指示不这样做:

request.Close = true

当服务器在没有在响应中使用Connection: close头部指示的情况下关闭连接时,这就成为一个问题。net.http代码假设连接仍然打开,所以下一个尝试使用该连接的请求会遇到连接被关闭的EOF错误。

你需要检查在第1000个请求周围连接上发生了什么。接收服务器是否设置了每个客户端的连接数限制?你的代码是否遇到了连接超时?无论哪种情况,如果连接以Go的Client代码无法预测的方式被服务器关闭,那么你就会遇到EOF错误。

英文:

It looks to me that you might be having the issue described in this answer. Basically, the Go http client will try to reuse connections unless you've specifically indicated that it shouldn't either in the Transport you've set your Client instance to use or on the request itself with:

request.Close = true

This becomes a problem when the server on the other end closes the connection without indicating so with a Connection: close header in a response. The net.http code assumes that the connection is still open and so the next request that tries to use the connection next, encounters the EOF from the connection being closed the other time.

You would need to check what is happening on the connection around the 1000th request. Is the receiving server set to limit the number of connections per client? Is your code running into a connection timeout? In either case, if it so happens the connection is closed by the server in a way that the Go Client code can't predict, then you'll run into the EOF.

答案2

得分: 1

首先,为什么你在循环内的DoCreate()函数中每次都实例化一个新的客户端client := &amp;http.Client{...?客户端是可以被重复使用的,甚至可以并发使用,所以你可以在更全局的地方构建它,比如在main()函数中。

其次,对我来说,这样的错误看起来像是由RoundTrip产生的,所以可能是由服务器端引起的连接问题。你能否尝试使用模拟服务器进行测试?

最后,如果所有这些都没有帮助,那么是的,一些系统对每个net.Conn所需的打开文件描述符数量有限制。这个限制只能在操作系统级别上消除。

英文:
  • First, what for you instantiate new client each time client := &amp;http.Client{... inside DoCreate() inside a loop? Client can be
    reusable even concurrently, so you can construct it more globally,
    say in main() seems to me.
  • Then for me such error looks like
    produced by RoundTrip, so by connection, so possibly from server site. Can you
    test with mocking server?
  • Finally if all that would not help, then
    yes, some systems have restriction on number of opened
    FileDescriptors which every net.Conn wants to have own. This
    restriction can be eliminated only on OS level.

答案3

得分: 1

EOF通常是由于服务器未返回完整的标头(包括两个CRLF)或在标头完整之前连接被关闭而引起的。更有可能的是,您的服务器被并发请求过载,尽管您仍然应确保本地资源足够以处理您正在进行的并发请求的数量。如果num足够大,您将耗尽某些资源。

尽管该错误并不真正具有描述性,但它并不比其他请求错误更令人担忧。它是一个错误条件,您应该像处理其他错误一样处理它。如果您想要确定,您可能需要对导致EOF的连接进行数据包捕获。

英文:

The EOF is usually from a server not returning a complete header (including both CRLFs), or the connection is closed before the header is complete. It's more likely that your overloading your server with concurrent requests, though you should still make sure you have adequate resources locally for the number of concurrent requests you're making. If num is large enough, you're going to run out of something.

Though the error isn't really descriptive, it's not anything to worry about more so than any other request error. It's an error condition, and handle it like you would any others. If you want to know for certain, you'll likely have to have a packet capture of the connection that cause the EOF.

huangapple
  • 本文由 发表于 2015年1月20日 21:05:49
  • 转载请务必保留本文链接:https://go.coder-hub.com/28046100.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定