英文:
GoLang net/http memory keeps increasing on contineous requests
问题
我在GoLang中有以下代码：
package main
import (
"bytes"
"encoding/json"
"io/ioutil"
"log"
"net/http"
"time"
)
func httpClient() *http.Client {
var transport http.RoundTripper = &http.Transport{
DisableKeepAlives: false,
}
client := &http.Client{Timeout: 60 * time.Second, Transport: transport}
return client
}
func sendRequest(client *http.Client, method string) []byte {
endpoint := "https://httpbin.org/post"
values := map[string]string{"foo": "baz"}
jsonData, err := json.Marshal(values)
req, err := http.NewRequest(method, endpoint, bytes.NewBuffer(jsonData))
if err != nil {
log.Fatalf("Error Occurred. %+v", err)
}
resp, err := client.Do(req)
if err != nil {
defer resp.Body.Close()
log.Fatalf("Error sending request to API endpoint. %+v", err)
}
// Close the connection to reuse it
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatalf("Couldn't parse response body. %+v", err)
}
return body
}
func main() {
// c should be re-used for further calls
c := httpClient()
for i := 1; i <= 60; i++ {
response := sendRequest(c, http.MethodPost)
log.Println("Response Body:", string(response))
response = nil
time.Sleep(time.Millisecond * 1000)
}
}
执行时，内存大小不断增加，增长速度在一个小时内可以达到90MB。垃圾回收器(gc)是否工作不正常？即使我在多个请求中使用相同的http客户端，但似乎仍然有一些东西在增加内存占用。
英文:
I have the following code in GoLang
package main
import (
"bytes"
"encoding/json"
"io/ioutil"
"log"
"net/http"
"time"
)
func httpClient() *http.Client {
var transport http.RoundTripper = &http.Transport{
DisableKeepAlives: false,
}
client := &http.Client{Timeout: 60 * time.Second, Transport: transport}
return client
}
func sendRequest(client *http.Client, method string) []byte {
endpoint := "https://httpbin.org/post"
values := map[string]string{"foo": "baz"}
jsonData, err := json.Marshal(values)
req, err := http.NewRequest(method, endpoint, bytes.NewBuffer(jsonData))
if err != nil {
log.Fatalf("Error Occurred. %+v", err)
}
resp, err:= client.Do(req)
if err != nil {
defer resp.Body.Close()
log.Fatalf("Error sending request to API endpoint. %+v", err)
}
// Close the connection to reuse it
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
log.Fatalf("Couldn't parse response body. %+v", err)
}
return body
}
func main() {
// c should be re-used for further calls
c := httpClient()
for i := 1; i <= 60; i++ {
response := sendRequest(c, http.MethodPost)
log.Println("Response Body:", string(response))
response = nil
time.Sleep(time.Millisecond * 1000)
}
}
When executed, it keeps the memory size increasing and the growth goes to as much as 90mb in one hour. is the gc not working properly. Even though i am using same httpclient for multiple requests but it still looks like theres something thats increasing the size of memory footprint.
答案1
得分: 0
我建议你使用像 pprof 这样的工具，这些工具在精确排查这类问题时非常有用。
你将 DisableKeepAlives 字段设置为 false，这意味着即使请求已经完成，它仍会保持打开的连接，导致进一步的内存泄漏。在调用 ioutil.ReadAll(resp.Body) 后，你还应该调用 defer resp.Body.Close()。这正是 defer 关键字的目的——防止内存泄漏。垃圾回收并不意味着绝对的内存安全。
此外，在主函数之外避免使用 log.Fatal。使用分级日志记录器，如 zap 或 zerolog，而不是 log.Fatal，因为 log.Fatal 调用 os.Exit(1)，立即生效，这意味着你的 defer 语句将不起作用，或者调用普通的 panic。参考：https://stackoverflow.com/questions/33885235/should-a-go-package-ever-use-log-fatal-and-when
英文:
I advice you to use tools like pprof, these are very useful at troubleshooting precisely this kind of issues.
You have set DisableKeepAlives field to false, which means that it will keep open connections even after the requests have been made, leading to further memory leaks. You should also call defer resp.Body.Close() after calling ioutil.ReadAll(resp.Body). This is precisely the purpose of the defer keyword - preventing memory leaks. GC does not mean absolute memory safety.
Also, outside of main avoid using log.Fatal. Use leveled logger, like zap or zerolog instead, since log.Fatal calls os.Exit(1) with an immediate effect, which means your defer statements will take no effect, or call plain panic. See https://stackoverflow.com/questions/33885235/should-a-go-package-ever-use-log-fatal-and-when
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论