编码为Base64时的内存消耗

huangapple go评论118阅读模式
英文:

memory consumption at encoding base64

问题

我在使用golang的encoding/base64库时遇到了内存消耗的问题。

我的软件将一个视频文件分割成单独的图像(gocv mat),将它们转换为base64字符串,并以json格式保存到文件中。

在测试过程中,我发现内存使用量一直在增加,直到oom-reaper杀死进程。

通过使用pprof进行调查,发现encoding/base64的内存似乎在不断增加。

我在每个图像帧之后进行了pprof快照,并发现在oom-reaper杀死进程之前,encoding/base64的分配内存从976.89kB(flat)增加到4633.54kB(flat)。

开始时:
flat flat% sum% cum cum%
976.89kB 32.29% 32.29% 976.89kB 32.29% encoding/base64.(*Encoding).EncodeToString
512.50kB 16.94% 49.23% 512.50kB 16.94% runtime.allocm
512.20kB 16.93% 66.15% 512.20kB 16.93% runtime.malg
512.05kB 16.92% 83.08% 1488.94kB 49.21% runtime.main
512kB 16.92% 100% 512kB 16.92% time.resetTimer (inline)
0 0% 100% 976.89kB 32.29% main.Process

结束时:
显示占用6170.44kB的节点,占总内存的100%
flat flat% sum% cum cum%
4633.54kB 75.09% 75.09% 4633.54kB 75.09% encoding/base64.(*Encoding).EncodeToString
1024.41kB 16.60% 91.69% 1024.41kB 16.60% runtime.malg
512.50kB 8.31% 100% 512.50kB 8.31% runtime.allocm
0 0% 100% 4633.54kB 75.09% main.Process

列表显示了相应的代码:

(pprof) list encoding/base64
Total: 2.95MB
ROUTINE ======================== encoding/base64.(*Encoding).EncodeToString in /usr/local/go/src/encoding/base64/base64.go
976.89kB 976.89kB(flat, cum)占总内存的32.29%
. . 175:
. . 176:// EncodeToString返回src的base64编码。
. . 177:func (enc *Encoding) EncodeToString(src []byte) string {
. . 178: buf := make([]byte, enc.EncodedLen(len(src)))
. . 179: enc.Encode(buf, src)
976.89kB 976.89kB 180: return string(buf)
. . 181:}
. . 182:
. . 183:type encoder struct {
. . 184: err error
. . 185: enc *Encoding

所以在我的golang代码中,相应的代码行是:

func Process(img gocv.Mat) (myImage Images) {

detectImg, detectClass, detectBoxes := Detect(&net,
    img.Clone(),
    0.45, 0.5,
    OutputNames, classes)
defer detectImg.Close()

// 将gocv.Mat转换为[]bytes
myImg, _ := detectImg.ToImage()
myJPG := new(bytes.Buffer)
jpeg.Encode(myJPG, myImg, &jpeg.Options{95})
myBytes := myJPG.Bytes()

// 内存消耗
encodedString := base64.StdEncoding.EncodeToString(myBytes)

// [...]

return myImage

}

在这种情况下,我如何释放"encodedString"的内存,以防止内存堆积?
(更新:回答说这不是必要的,也不可能)

或者,也许不是我的错误编码,而是lib base64的内存泄漏?
(更新:回答说这肯定不是问题)

英文:

I have problems with memory consumption at my software using golangs lib encoding/base64

My software is splitting a videofile to separate images, (gocv mat)
converting them to base64 string and saving it to file in json format.

During testing I found that the memory usage is piling up until
the oom-reaper is killing the process.

Investigation with pprof showed that the encoding/base64 memory seems to pile up.

I did pprof snapshots after each image frame, and allocated mem of
encoding/base64 is raising from 976.89kB(flat) to 4633.54kB(flat) shortly before oom-reaper was killing the process.

Beginning:
      flat  flat%   sum%        cum   cum%
  976.89kB 32.29% 32.29%   976.89kB 32.29%  encoding/base64.(*Encoding).EncodeToString
  512.50kB 16.94% 49.23%   512.50kB 16.94%  runtime.allocm
  512.20kB 16.93% 66.15%   512.20kB 16.93%  runtime.malg
  512.05kB 16.92% 83.08%  1488.94kB 49.21%  runtime.main
     512kB 16.92%   100%      512kB 16.92%  time.resetTimer (inline)
         0     0%   100%   976.89kB 32.29%  main.Process

End:
Showing nodes accounting for 6170.44kB, 100% of 6170.44kB total
      flat  flat%   sum%        cum   cum%
 4633.54kB 75.09% 75.09%  4633.54kB 75.09%  encoding/base64.(*Encoding).EncodeToString
 1024.41kB 16.60% 91.69%  1024.41kB 16.60%  runtime.malg
  512.50kB  8.31%   100%   512.50kB  8.31%  runtime.allocm
         0     0%   100%  4633.54kB 75.09%  main.Process

list shows me the code acoording to it:

(pprof) list encoding/base64
Total: 2.95MB
ROUTINE ======================== encoding/base64.(*Encoding).EncodeToString in /usr/local/go/src/encoding/base64/base64.go
  976.89kB   976.89kB (flat, cum) 32.29% of Total
         .          .    175:
         .          .    176:// EncodeToString returns the base64 encoding of src.
         .          .    177:func (enc *Encoding) EncodeToString(src []byte) string {
         .          .    178:   buf := make([]byte, enc.EncodedLen(len(src)))
         .          .    179:   enc.Encode(buf, src)
  976.89kB   976.89kB    180:   return string(buf)
         .          .    181:}
         .          .    182:
         .          .    183:type encoder struct {
         .          .    184:   err  error
         .          .    185:   enc  *Encoding

So in my golang code the according line of code was:

func Process(img gocv.Mat) ( myImage Images  ){

	detectImg, detectClass, detectBoxes := Detect(&net, 
                                           img.Clone(), 
                                           0.45, 0.5, 
                                           OutputNames, classes)
	defer detectImg.Close()

	// convert gocv.Mat to []bytes
	myImg , _ := detectImg.ToImage()
	myJPG := new(bytes.Buffer)
	jpeg.Encode(myJPG, myImg, &jpeg.Options{95})
	myBytes := myJPG.Bytes()


	// memory consuming
	encodedString := base64.StdEncoding.EncodeToString(myBytes)

// [...]

	return myImage

}

How can I release the memory of "encodedString" in this case that it does not pile up?
(Update: Answers say this is not necessary and not possible)

Or is it maybe not my wrong coding, and the mem-leak is at the lib base64 ??
(Update: Answers say this is surely not the case)

答案1

得分: 0

回答你的问题:

> 在这种情况下,我如何释放“encodedString”的内存,以防止堆积?

你不能也不需要释放未使用的内存。未使用的内存会被“释放”。

> 或者可能不是我的错误编码,而是lib base64中的内存泄漏吗?

不,encoding/base64包没有内存泄漏。(在垃圾回收语言的标准库中,一个简单函数的内存泄漏几乎不可能被检测到。)

为了指导你找到解决方案:

你的应用程序使用了大量的内存,这是因为a)处理视频和图像确实需要大量内存,b)你似乎没有采取任何措施来保持内存低:例如,你将整个图像编码为一个bytes.Buffer,然后将整个bytes.Buffer编码为一个字符串,然后在该字符串上进行操作,依此类推。你可能应该将图像编码为一个“流”,将这个“流”编码为base64,并将这个输出“流”进一步传输到目标位置。在Go语言中,这是非常简单的,因为所有这些编码器都可以在io.Writer上工作,可以非常容易地链接起来。

英文:

To answer your questions:

> How can I release the memory of "encodedString" in this case that it does not pile up?

You cannot and you need not. Unused memory is "freed".

> Or is it maybe not my wrong coding, and the mem-leak is at the lib base64 ?

No, package encoding/base64 has no memory leak. (Chances are 0 that you detect a memory leak in a trivial function in the standard library of a garbage collected language.)

To guide you towards a solution:

Your application uses absurd amount of memory but that's because a) processing video and images is memory hungry and b) you seem to do nothing to keep memory low: E.g. you encode the whole image into a bytes.Buffer, then encode the whole bytes.Buffer to a string then work on that string and so on. You probably should encode the image into a stream, encode this stream to base64 and stream this output further to where it deposited. This is totally painless in Go as all these encoders work on io.Writers which can be chained very easily.

答案2

得分: -1

我的问题之前完全错了。

Base64根本不是问题,它只是在pprof中显示的内存消耗最多的部分,导致我错误地认为base64是问题所在。

我猜想pprof会告诉我我的Go程序的所有内存消耗,包括gocv。gocv是opencv的一个C语言封装,但它的内存消耗对pprof来说是不可见的,因为它是C代码!(在提问时我不知道这一点)。
pprof可见的内存消耗并没有显示出go语言的c封装库(如gocv)使用的内存。
大部分内存消耗对于golang来说根本不可见。
所以JimB给出的有用提示是:

>看到你在使用一个围绕opencv的Go封装,你关心的内存可能根本不是由Go分配的。在这种情况下,你确实需要确保根据它们的文档关闭或释放所有东西,因为大部分工作是在C++中完成的,而不是在Go中。即使你正确清理了,你仍然需要注意你的内存限制,并确保你在任何给定的时间点上没有尝试保存太多的数据。

当我清理gocv对象时,内存消耗显著下降。我使用了以下方式关闭对象:

defer obj.close()
英文:

My question above was at a totaly wrong way.

Base64 was not at all the problem, it was just the top-consumer of memory
displayed at pprof, leading me to the faulty conclusion that base64 is the problem.

I guessed that pprof would tell me all mem-consumption
of my go programm, including gocv.
gocv is a c wrapper around opencv, but the mem-consumption of it is not visible to pprof, as it is c-code! (I did not know that at asking the question).
The memory consumption that was visible due pprof was not showing
the used memory by c-wrapper libs for go like gocv.
The big part of memory consumption was not visible to golang at all.
So the helping hint of JimB was:

> Seeing how you are using a Go wrapper around opencv, the memory you
> are concerned with is probably not even allocated by Go. In that case
> you do need to ensure that everything is probably closed or released
> according to their documentation, because the bulk of the work is done
> in C++, not Go. Even if you are cleaning up properly however, you
> still need to be aware of your memory limitations and ensure you are
> not trying to hold too much data at any given point.

As I was cleaning up the gocv objects, mem consumtion was going down significantly. I used to close objects:

defer obj.close()

huangapple
  • 本文由 发表于 2022年1月26日 21:04:23
  • 转载请务必保留本文链接:https://go.coder-hub.com/70863870.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定