在生产环境中使用FreeOSMemory()函数。

huangapple go评论83阅读模式
英文:

FreeOSMemory() in production

问题

我正在一个包中使用goroutines,其中包含一个tcp服务器。大部分时间响应都非常重,但是当协程结束时,内存中并没有清除。

func Handle() {
    service := ":7777"
    tcpAddr, err := net.ResolveTCPAddr("tcp4", service)
    checkError(err)
    listener, err := net.ListenTCP("tcp", tcpAddr)
    checkError(err)
    defer listener.Close()

    for {
        conn, err := listener.Accept()
        checkError(err)
        go handleRequest(conn, db)

    }
}

func handleRequest(conn net.Conn, db *sql.DB) {
    message := make([]byte, 0, 4096)
    tmp := make([]byte, 256)
    n, err := conn.Read(tmp)
    if err != nil {
        if err != io.EOF {
            fmt.Println("read error:", err)
        }
    }
    message = append(message, tmp[:n]...)
    fmt.Println("Message Received:", string(message))
    // do something to get resp
    conn.Write(append(resp, []byte("\n")...))
    conn.Close()
    debug.FreeOSMemory()
    return
}

在这种情况下,响应很大,一个协程使用了10%的内存,这是可以接受的,因为我从数据库中获取了170,000个用户并将结果解析为JSON。但是当handleRequest结束时,它仍然在内存中,如果我不使用debug.FreeOSMemory(),我对此是否有疑问,因为它在调试包中,所以我的问题是清空协程使用的内存是否是一个好方法?我测试过了,对系统没有影响,而且工作得非常好。如果不是,请问什么是正确的方法?我不能等待垃圾回收来清理它吗?我阅读了这个,这就是为什么我开始使用它,在第一个答案中有最后的建议。

英文:

I am using goroutines in a package where there is a tcp server. The response most of the time is very heavy, but when the routines end it is not cleared from the memory.

func Handle() {
	service := ":7777"
	tcpAddr, err := net.ResolveTCPAddr("tcp4", service)
	checkError(err)
	listener, err := net.ListenTCP("tcp", tcpAddr)
	checkError(err)
	defer listener.Close()

	for {
		conn, err := listener.Accept()
		checkError(err)
		go handleRequest(conn, db)

	}
}

func handleRequest(conn net.Conn, db *sql.DB) {
	message := make([]byte, 0, 4096)
	tmp := make([]byte, 256)
	n, err := conn.Read(tmp)
	if err != nil {
		if err != io.EOF {
			fmt.Println("read error:", err)
		}
	}
	message = append(message, tmp[:n]...)
	fmt.Println("Message Received:", string(message))
	// do something to get resp
	conn.Write(append(resp, []byte("\n")...))
	conn.Close()
	debug.FreeOSMemory()
	return
}

So in this case the response is big, and a goroutine using 10% of the memory, thats okay, because I'm getting 170.000 users from the database and parse the result to JSON. But when the handleRequest and it is still in the memory, if I'm not using debug.FreeOsMemory(). I have doubts it is a good way to do it because it is in the debug pacakge so my question is it a good way to empty the memory what the goroutines are using? I tested it so it is not affected to the system and working very well. If not, what is the good way? I can't wait the GC for clear it up?! I read this and this is why I started to use it, in the first answer there is the last suggestion.

答案1

得分: 10

Go运行时不会立即将空闲内存释放回操作系统,这样做效率会很低。如果你的应用程序在不调用debug.FreeOsMemory()的情况下不稳定,那么可能存在更大的问题,即使这样做似乎有所帮助也不应该掩盖这些问题。这甚至可能会使情况变得更糟,因为如果处理请求确实需要大量内存(在请求完成后由垃圾回收器正确释放),调用FreeOsMemory()将会将内存返回给操作系统,而运行时在处理另一个请求时将不得不再次请求/分配该内存。如果你没有将内存还给操作系统,它将可用于下一个请求...

尝试减少请求处理程序的内存需求。如果不可能(值得怀疑),那么限制可能同时处理的需要大内存的请求的数量。

参考以下问题+答案了解如何做到这一点:https://stackoverflow.com/questions/37529511/process-management-for-the-go-webserver/37531953#37531953

还可以参考:https://stackoverflow.com/questions/38170852/is-this-an-idiomatic-worker-thread-pool-in-go/38172204#38172204

英文:

The Go runtime does not release free memory back to the OS "immediately", it would be inefficient. Read more about it here: Golang - Cannot free memory once occupied by bytes.Buffer.

You should let the Go runtime handle this. If your app is unstable without calling debug.FreeOsMemory(), there are bigger problems which you shouldn't cover up with this even if it "seemingly" helps. It may even make things worse, as if serving a request does require a large amount of memory (which is properly freed by the GC when done with the request), calling FreeOsMemory() will just return it to the OS which the runtime will have to ask for / allocate again when serving another request. Should you have not handed it back to the OS, it would be available for the next request...

Try to decrease the memory requirement of the request handler. If it is not possible (questionable), then limit the number of requests requiring large memory that may be served concurrently.

See this question+answer how to do that: https://stackoverflow.com/questions/37529511/process-management-for-the-go-webserver/37531953#37531953

Also: https://stackoverflow.com/questions/38170852/is-this-an-idiomatic-worker-thread-pool-in-go/38172204#38172204

答案2

得分: 4

作为偶尔调用debug.FreeOsMemory()的替代方法,您可以使用GOGC环境变量或debug.SetGCPercent()方法来控制垃圾收集器的侵略性。

根据经验,我发现将值设置为10时,它能够完成其工作。

请参阅Package runtime

英文:

As an alternative to occasionally invoking debug.FreeOsMemory(), you can control the aggressiveness of the garbage collector with the GOGC environment variable or with the debug.SetGCPercent() method.

Empirically, I've seen that with a value as low as 10, it does its job.

See Package runtime

答案3

得分: 3

GO语言清理所有未使用但不是立即使用的内存。你可以阅读另一个问题:

Golang - 无法释放被bytes.Buffer占用的内存

英文:

GO lang clean all memory that is not in use, but is not inmediate. You can read the other question:

Golang - Cannot free memory once occupied by bytes.Buffer

答案4

得分: 0

这是疯狂的,我认为这是golang的一个bug。我的代码包括:

go func() {
	for {
		debug.FreeOSMemory()
		log4sys.Warn("NumGoroutine:", runtime.NumGoroutine())
		time.Sleep(1 * time.Minute)
	}
}()

解决这个问题的方法是...

英文:

That is crazy, I think is a bug for golang,My code include:

go func() {
	for {
		debug.FreeOSMemory()
		log4sys.Warn("NumGoroutine:",runtime.NumGoroutine())
		time.Sleep(1 * time.Minute)
	}
}()

to solution this question

huangapple
  • 本文由 发表于 2017年2月20日 20:50:44
  • 转载请务必保留本文链接:https://go.coder-hub.com/42345060.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定