释放未使用的内存。

huangapple go评论98阅读模式
英文:

Freeing unused memory?

问题

我正在使用以下函数来下载小于20MB的文件。它将整个内容读入内存,因为另一个函数需要在将其写入磁盘之前对字节进行处理。

func getURL(url string) ([]byte, error) {
    resp, err := http.Get(url)
    if err != nil {
        return nil, fmt.Errorf("getURL: %s", err)
    }
    defer resp.Body.Close()

    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
        return nil, fmt.Errorf("getURL: %s", err)
    }

    return body, nil
}

这个函数运行良好,但会消耗系统上的所有内存。

是否有可能在另一个函数处理完body后释放它所使用的内存,这样内存使用量就不会超过当前正在处理的字节数了?

英文:

I'm using the following function for downloading files smaller than 20MB. It read the entire content to memory as another function has to perform work on the bytes before it can be written to disk.

func getURL(url string) ([]byte, error) {
    resp, err := http.Get(url)
        if err != nil {
            return nil, fmt.Errorf("getURL: %s", err)
    }
    defer resp.Body.Close()

    body, err := ioutil.ReadAll(resp.Body)
    if err != nil {
            return nil, fmt.Errorf("getURL: %s", err)
    }

    return body, nil
}

This works fine, but all memory is consumed on the system.

Is it possible to release memory used by body after it has been processed by another function, so memory use won't be larger than the bytes currently being processed?

答案1

得分: 10

首先,我建议阅读以下问题/答案:

https://stackoverflow.com/questions/42345060/freeosmemory-in-production/42345554#42345554

https://stackoverflow.com/questions/37382600/golang-cannot-free-memory-once-occupied-by-bytes-buffer/37383604#37383604

你可以使用runtime.GC()触发垃圾回收以释放未使用的对象,并可以使用debug.FreeOSMemory()来促使Go运行时将内存释放回操作系统,但这些只是临时应对措施。一个良好编写的Go应用程序不应该需要调用这些函数。

你应该做的是避免运行时分配大量内存。

你可以通过以下方式实现这一点(你甚至可以结合这些解决方案):

  • 限制需要大量内存的请求的服务,更多信息请参考:https://stackoverflow.com/questions/37529511/process-management-for-the-go-webserver/37531953#37531953;还有https://stackoverflow.com/questions/38170852/is-this-an-idiomatic-worker-thread-pool-in-go/38172204#38172204

  • 使用内存/缓冲池,不要一直分配大数组/切片,更多信息请参考:https://stackoverflow.com/questions/38505830/how-to-implement-memory-pooling-in-golang/38506367#38506367

  • 创建/更改处理单元,使其不是操作字节切片,而是操作io.Reader,这样你就不需要将所有内容读入内存,只需传递resp.Body即可。请注意,即使多个单元需要读取/检查主体,仍然可以只读取和处理一次,而不将其保存在内存中。可以使用io.Pipe()io.TeeReader()或自定义解决方案。

英文:

First I recommend to read the following questions / answers:

https://stackoverflow.com/questions/42345060/freeosmemory-in-production/42345554#42345554

https://stackoverflow.com/questions/37382600/golang-cannot-free-memory-once-occupied-by-bytes-buffer/37383604#37383604

You may trigger a gc to free unused objects with runtime.GC() and you may urge your Go runtime to release memory back to OS with debug.FreeOSMemory(), but all these are just fire fighting. A well-written Go app should never have to call these.

What you should do is prevent the runtime having to allocate large amount of memory.

How may you achieve this? Some means (you can even combine these solutions):

huangapple
  • 本文由 发表于 2017年8月4日 22:36:29
  • 转载请务必保留本文链接:https://go.coder-hub.com/45509538.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定