英文:
io.Copy cause out of memory in golang
问题
我使用io.Copy()函数来复制一个大约700MB的文件，但是它导致了内存溢出的问题。
bodyBuf := &bytes.Buffer{}
bodyWriter := multipart.NewWriter(bodyBuf)
//关键步骤
fileWriter, err := bodyWriter.CreateFormFile(paramName, fileName)
if err != nil {
return nil, err
}
file, err := os.Open(fileName) //文件大小约为700MB
if err != nil {
return nil, err
}
defer file.Close()
//复制文件
copyLen, err := io.Copy(fileWriter, file) //这导致了内存溢出
if err != nil {
fmt.Println("io.copy(): ", err)
return nil, err
}
错误信息如下：
runtime: memory allocated by OS (0x752cf000) not in usable range [0x18700000,0x98700000)
runtime: out of memory: cannot allocate 1080229888-byte block (1081212928 in use)
fatal error: out of memory
我为buf分配了足够的内存，但在bodyWriter.CreateFormFile()函数中导致了内存溢出。
buf := make([]byte, 766509056)
bodyBuf := bytes.NewBuffer(buf)
bodyWriter := multipart.NewWriter(bodyBuf)
fileWriter, err := bodyWriter.CreateFormFile(paramName, fileName) //内存溢出
if err != nil {
return nil, err
}
英文:
I use io.Copy() to copy a file, about 700Mb, but it cause out of memory
bodyBuf := &bytes.Buffer{}
bodyWriter := multipart.NewWriter(bodyBuf)
//key step
fileWriter, err := bodyWriter.CreateFormFile(paramName, fileName)
if err != nil {
return nil, err
}
file, err := os.Open(fileName) //the file size is about 700Mb
if err != nil {
return nil, err
}
defer file.Close()
//iocopy
copyLen, err := io.Copy(fileWriter, file) // this cause out of memory
if err != nil {
fmt.Println("io.copy(): ", err)
return nil, err
}
The error message as follow:
runtime: memory allocated by OS (0x752cf000) not in usable range [0x18700000,0x98700000)
runtime: out of memory: cannot allocate 1080229888-byte block (1081212928 in use)
fatal error: out of memory
I allocate enough memory for buf, it cause out of memory in bodyWriter.CreateFormFile()
buf := make([]byte, 766509056)
bodyBuf := bytes.NewBuffer(buf)
bodyWriter := multipart.NewWriter(bodyBuf)
fileWriter, err := bodyWriter.CreateFormFile(paramName, fileName) // out of memory
if err != nil {
return nil, err
}
答案1
得分: 7
这是因为你正在将文件内容复制到一个内存缓冲区bodyBuf中，这会强制Go尝试分配一个与整个文件大小相同的内存块。
根据你使用multipart的方式，看起来你是想通过HTTP流式传输文件？在这种情况下，不要将bytes.Buffer传递给multipart.NewWriter，直接将你的HTTP连接传递给它即可。
英文:
That's because you are 'copying', to bodyBuf, which is an in-memory buffer, forcing Go to try an allocate a block of memory as big as the entire file.
Based on your use of multipart it looks like you are trying to stream the file over http? In that case, don't pass a bytes.Buffer to multipart.NewWriter, directly pass your http connection instead.
通过集体智慧和协作来改善编程学习和解决问题的方式。致力于成为全球开发者共同参与的知识库，让每个人都能够通过互相帮助和分享经验来进步。
评论