重复调用image.png.Decode()会导致内存不足错误。

huangapple go评论85阅读模式
英文:

Repeated calls to image.png.Decode() leads to out of memory errors

问题

我正在尝试做一件我最初认为很简单的事情。具体来说:

对于输入文件列表中的每个文件:

  1. 使用png.Decode()打开文件。
  2. 扫描文件中的每个像素,并测试其是否为“灰色”。
  3. 返回图像中“灰色”像素的百分比。

这是我调用的函数:

func greyLevel(fname string) (float64, string) {
    f, err := os.Open(fname)
    if err != nil {
            return -1.0, "无法打开文件"
    }
    defer f.Close()

    i, err := png.Decode(f)
    if err != nil {
            return -1.0, "无法解码"
    }

    bounds := i.Bounds()

    var lo uint32 = 122 // 低灰色RGB值。
    var hi uint32 = 134 // 高灰色RGB值。
    var gpix float64    // 灰色像素计数。
    var opix float64    // 其他(非灰色)像素计数。
    var tpix float64    // 总像素数。

    for x := bounds.Min.X; x < bounds.Max.X; x++ {
            for y := bounds.Min.Y; y < bounds.Max.Y; y++ {
                    r, g, b, _ := i.At(x, y).RGBA()
                    if ((r/255)-1 > lo && (r/255)-1 < hi) &&
                            ((g/255)-1 > lo && (g/255)-1 < hi) &&
                            ((b/255)-1 > lo && (b/255)-1 < hi) {
                            gpix++
                    } else {
                            opix++
                    }
                    tpix++
            }
    }
    return (gpix / tpix) * 100, ""
} 

func main() {
    srcDir := flag.String("s", "", "包含图像文件的目录。")
    threshold := flag.Float64("t", 65.0, "灰色像素的阈值(百分比)。")
    flag.Parse()

    dirlist, direrr := ioutil.ReadDir(*srcDir)
    if direrr != nil {
            log.Fatalf("读取%s时出错:%s\n", *srcDir, direrr)
    }

    for f := range dirlist {
            src := path.Join(*srcDir, dirlist[f].Name())

            level, msg := greyLevel(src)

            if msg != "" {
                    log.Printf("处理%s时出错:%s\n", src, msg)
                    continue
            }

            if level >= *threshold {
                    log.Printf("%s是灰色的(%2.2f%%)\n", src, level)
            } else {
                    log.Printf("%s不是灰色的(%2.2f%%)\n", src, level)
            }
    }
}

这些文件相对较小(960x720,8位RGB)

我调用ioutil.ReadDir()生成文件列表,循环遍历切片并调用greyLevel()。

在大约155个文件(超过4000个文件的列表)之后,脚本会发生崩溃:

runtime: 内存分配超出可用范围
runtime: 内存不足:无法分配2818048字节的块(已使用534708224字节)
throw: 内存不足

我觉得我可能遗漏了一些简单的东西。我以为Go会释放在greyLevels()中分配的内存,但看来不是这样?

后续:

在每次调用greyLevels之后插入runtime.GC()后,内存使用量变得稳定。昨晚我测试了大约800张图像,然后停止了。今天我让它在整个输入集上运行,大约6800张图像。

处理了1500张图像后,top命令显示如下:

top - 10:30:11 up 41 days, 11:47,  2 users,  load average: 1.46, 1.25, 0.88
Tasks: 135 total,   2 running, 131 sleeping,   1 stopped,   1 zombie
Cpu(s): 49.8%us,  5.1%sy,  0.2%ni, 29.6%id, 15.0%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:   3090304k total,  2921108k used,   169196k free,     2840k buffers
Swap:  3135484k total,    31500k used,  3103984k free,   640676k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
28474 mtw       20   0 2311m 1.8g  412 R   99 60.5  16:48.52 8.out

在处理另外5000张图像后,内存使用量保持稳定。

英文:

I am attempting to do what I originally thought would be pretty simple. To wit:

For every file in a list of input files:

  1. open the file with png.Decode()
  2. scan every pixel in the file and test to see if it is "grey".
  3. Return the percentage of "grey" pixels in the image.

This is the function I am calling:

func greyLevel(fname string) (float64, string) {
    f, err := os.Open(fname)
    if err != nil {
            return -1.0, &quot;can&#39;t open file&quot;
    }
    defer f.Close()

    i, err := png.Decode(f)
    if err != nil {
            return -1.0, &quot;unable to decode&quot;
    }

    bounds := i.Bounds()

    var lo uint32 = 122 // Low grey RGB value.
    var hi uint32 = 134 // High grey RGB value.
    var gpix float64    // Grey pixel count.
    var opix float64    // Other (non-grey) pixel count.
    var tpix float64    // Total pixels.

    for x := bounds.Min.X; x &lt; bounds.Max.X; x++ {
            for y := bounds.Min.Y; y &lt; bounds.Max.Y; y++ {
                    r, g, b, _ := i.At(x, y).RGBA()
                    if ((r/255)-1 &gt; lo &amp;&amp; (r/255)-1 &lt; hi) &amp;&amp;
                            ((g/255)-1 &gt; lo &amp;&amp; (g/255)-1 &lt; hi) &amp;&amp;
                            ((b/255)-1 &gt; lo &amp;&amp; (b/255)-1 &lt; hi) {
                            gpix++
                    } else {
                            opix++
                    }
                    tpix++
            }
    }
    return (gpix / tpix) * 100, &quot;&quot;
} 

func main() {
    srcDir := flag.String(&quot;s&quot;, &quot;&quot;, &quot;Directory containing image files.&quot;)
    threshold := flag.Float64(&quot;t&quot;, 65.0, &quot;Threshold (in percent) of grey pixels.&quot;)
    flag.Parse()

    dirlist, direrr := ioutil.ReadDir(*srcDir)
    if direrr != nil {
            log.Fatalf(&quot;Error reading %s: %s\n&quot;, *srcDir, direrr)
    }

    for f := range dirlist {
            src := path.Join(*srcDir, dirlist[f].Name())

            level, msg := greyLevel(src)

            if msg != &quot;&quot; {
                    log.Printf(&quot;error processing %s: %s\n&quot;, src, msg)
                    continue
            }

            if level &gt;= *threshold {
                    log.Printf(&quot;%s is grey (%2.2f%%)\n&quot;, src, level)
            } else {
                    log.Printf(&quot;%s is not grey (%2.2f%%)\n&quot;, src, level)
            }
    }
}

The files are relatively small (960x720, 8-bit RGB)

I am calling ioutil.ReadDir() to generate a list of files, looping over the slice and calling greyLevel().

After about 155 files (out of a list of >4000) the script panics with:

runtime: memory allocated by OS not in usable range
runtime: out of memory: cannot allocate 2818048-byte block (534708224 in use)
throw: out of memory

I figure there is something simple I am missing. I thought that Go would de-allocate the memory allocated in greyLevels() but I guess not?

Follow up:

After inserting runtime.GC() after every call to greyLevels, the memory usage evens out. Last night I was teting to about 800 images then stopped. Today I let it run over the entire input set, approximately 6800 images.

After 1500 images, top looks like this:

top - 10:30:11 up 41 days, 11:47,  2 users,  load average: 1.46, 1.25, 0.88
Tasks: 135 total,   2 running, 131 sleeping,   1 stopped,   1 zombie
Cpu(s): 49.8%us,  5.1%sy,  0.2%ni, 29.6%id, 15.0%wa,  0.0%hi,  0.3%si,  0.0%st
Mem:   3090304k total,  2921108k used,   169196k free,     2840k buffers
Swap:  3135484k total,    31500k used,  3103984k free,   640676k cached

  PID USER      PR  NI  VIRT  RES  SHR S %CPU %MEM    TIME+  COMMAND
28474 mtw       20   0 2311m 1.8g  412 R   99 60.5  16:48.52 8.out

And remained steady after processing another 5000 images.

答案1

得分: 2

看起来你正在使用一个32位的机器。由于Go的垃圾回收器是保守的,程序很可能会因为内存不足而运行失败。保守的垃圾回收器可能无法检测到某些内存区域不再使用。目前在Go程序中没有其他解决方法,除了避免使用垃圾回收器无法处理的数据结构(例如:struct {...; binaryData [256]byte})。

尝试在调用greyLevel函数的循环的每次迭代中调用runtime.GC()。也许这样可以帮助程序处理更多的图像。

如果调用runtime.GC()无法改善情况,你可能需要改变策略,使程序每次运行处理更少数量的PNG文件。

英文:

It appears that you are using a 32-bit machine. It is likely that the program runs out of memory because Go's garbage collector is conservative. A conservative garbage collector may fail to detect that some region of memory is no longer in use. There is currently no workaround for this in Go programs other than avoiding data structures that the garbage collector cannot handle (such as: struct {...; binaryData [256]byte})

Try to call runtime.GC() in each iteration of the loop in which you are calling function greyLevel. Maybe it will help the program to process more images.

If calling runtime.GC() fails to improve the situation you may want to change your strategy so that the program processes a smaller number of PNG files per run.

答案2

得分: 0

似乎是最近修复的问题3173。请尝试使用最新的每周版本进行重试(假设您现在使用的是2012-03-07之前的某个版本)。

英文:

Seems like issue 3173 which was recently fixed. Could you please retry with latest weekly? (Assuming you now use some pre 2012-03-07 version).

huangapple
  • 本文由 发表于 2012年3月26日 15:31:18
  • 转载请务必保留本文链接:https://go.coder-hub.com/9868214.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定