英文:
Freeing unused memory?
问题
我正在使用以下函数来下载小于20MB的文件。它将整个内容读入内存,因为另一个函数需要在将其写入磁盘之前对字节进行处理。
func getURL(url string) ([]byte, error) {
resp, err := http.Get(url)
if err != nil {
return nil, fmt.Errorf("getURL: %s", err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, fmt.Errorf("getURL: %s", err)
}
return body, nil
}
这个函数运行良好,但会消耗系统上的所有内存。
是否有可能在另一个函数处理完body
后释放它所使用的内存,这样内存使用量就不会超过当前正在处理的字节数了?
英文:
I'm using the following function for downloading files smaller than 20MB. It read the entire content to memory as another function has to perform work on the bytes before it can be written to disk.
func getURL(url string) ([]byte, error) {
resp, err := http.Get(url)
if err != nil {
return nil, fmt.Errorf("getURL: %s", err)
}
defer resp.Body.Close()
body, err := ioutil.ReadAll(resp.Body)
if err != nil {
return nil, fmt.Errorf("getURL: %s", err)
}
return body, nil
}
This works fine, but all memory is consumed on the system.
Is it possible to release memory used by body
after it has been processed by another function, so memory use won't be larger than the bytes currently being processed?
答案1
得分: 10
首先,我建议阅读以下问题/答案:
https://stackoverflow.com/questions/42345060/freeosmemory-in-production/42345554#42345554
你可以使用runtime.GC()
触发垃圾回收以释放未使用的对象,并可以使用debug.FreeOSMemory()
来促使Go运行时将内存释放回操作系统,但这些只是临时应对措施。一个良好编写的Go应用程序不应该需要调用这些函数。
你应该做的是避免运行时分配大量内存。
你可以通过以下方式实现这一点(你甚至可以结合这些解决方案):
-
限制需要大量内存的请求的服务,更多信息请参考:https://stackoverflow.com/questions/37529511/process-management-for-the-go-webserver/37531953#37531953;还有https://stackoverflow.com/questions/38170852/is-this-an-idiomatic-worker-thread-pool-in-go/38172204#38172204
-
使用内存/缓冲池,不要一直分配大数组/切片,更多信息请参考:https://stackoverflow.com/questions/38505830/how-to-implement-memory-pooling-in-golang/38506367#38506367
-
创建/更改处理单元,使其不是操作字节切片,而是操作
io.Reader
,这样你就不需要将所有内容读入内存,只需传递resp.Body
即可。请注意,即使多个单元需要读取/检查主体,仍然可以只读取和处理一次,而不将其保存在内存中。可以使用io.Pipe()
、io.TeeReader()
或自定义解决方案。
英文:
First I recommend to read the following questions / answers:
https://stackoverflow.com/questions/42345060/freeosmemory-in-production/42345554#42345554
You may trigger a gc to free unused objects with runtime.GC()
and you may urge your Go runtime to release memory back to OS with debug.FreeOSMemory()
, but all these are just fire fighting. A well-written Go app should never have to call these.
What you should do is prevent the runtime having to allocate large amount of memory.
How may you achieve this? Some means (you can even combine these solutions):
-
Limit serving the requests requiring large memory, more about it: https://stackoverflow.com/questions/37529511/process-management-for-the-go-webserver/37531953#37531953; also https://stackoverflow.com/questions/38170852/is-this-an-idiomatic-worker-thread-pool-in-go/38172204#38172204
-
Use memory / buffer pools, do not allocate big arrays / slices all the time, more about it: https://stackoverflow.com/questions/38505830/how-to-implement-memory-pooling-in-golang/38506367#38506367
-
Create / change your processing units not to operate on byte slices but on
io.Reader
s, so you don't need to read all content into memory, you can just passresp.Body
on. Note that even if multiple units have to read / inspect the body, it is still possible to only read and process it once, and not keep it in memory. Means may beio.Pipe()
,io.TeeReader()
or custom solutions.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库,让每个人都能够通过互相帮助和分享经验来进步。
评论