意外的 OutOfMemoryError 在分配比堆更大的数组时发生

huangapple go评论59阅读模式
英文:

Unexpected OutOfMemoryError when allocating an array larger than the heap

问题

我今天在处理OOM(Out of Memory)错误时遇到了一些我无法解释的情况。

我尝试分配一个比堆更大的数组,期望出现 "请求的数组大小超过VM限制" 错误,但实际上我得到了一个 "Java堆空间不足" 错误。

根据 JDK 11 文档 "3 Troubleshoot Memory Leaks > Understand the OutOfMemoryError Exception"

线程 thread_name 中的异常: java.lang.OutOfMemoryError: 请求的数组大小超过了 VM 限制

原因: 详细消息 "请求的数组大小超过 VM 限制" 表示应用程序(或应用程序使用的 API)尝试分配一个大于堆大小的数组。例如,如果应用程序尝试分配一个大小为 512 MB 的数组,但最大堆大小为 256 MB,则会抛出 OutOfMemoryError,原因是“请求的数组大小超过 VM 限制”。


代码:

public class MemOverflow {

    public static void main(final String[] args) {
        System.out.println("堆最大大小: " + (Runtime.getRuntime().maxMemory() / 1024 / 1024) + "MB");
        long[] array = new long[100_000_000]; // ~800MB
        System.out.println(array.length);
    }
}

当堆大小足够大以存储我的数组时,这段代码按预期工作:

$ javac MemOverflow.java && java -Xmx800m MemOverflow
堆最大大小: 800MB
100000000

然而,当我减小堆大小时,我得到了错误的OOM错误:

$ javac MemOverflow.java && java -Xmx100m MemOverflow
堆最大大小: 100MB
Exception in thread "main" java.lang.OutOfMemoryError: Java堆空间不足
	at MemOverflow.main(MemOverflow.java:5)
# 我预期会在这里看到 "请求的数组大小超过 VM 限制" 错误

我是不是漏掉了什么? 我知道我可以使用 Integer.MAX_VALUE 作为数组大小来生成我想要的错误,但我希望通过调整堆大小来触发这个错误。

Java 版本:

openjdk version "11.0.8" 2020-07-14
OpenJDK Runtime Environment (build 11.0.8+10-post-Ubuntu-0ubuntu120.04)
OpenJDK 64-Bit Server VM (build 11.0.8+10-post-Ubuntu-0ubuntu120.04, mixed mode)

编辑:
根据 @cdalxndr 的回答,我还尝试了 Oracle 最新的 JDK 15,但结果是一样的。

java version "15" 2020-09-15
Java(TM) SE Runtime Environment (build 15+36-1562)
Java HotSpot(TM) 64-Bit Server VM (build 15+36-1562, mixed mode, sharing)

编辑 2:
我已经提交了此问题:JDK-8254804


编辑 3:2021-01-19 更新
文档已修复。 在 v16-ea(早期访问版)中进行了修复。

线程 thread_name 中的异常: java.lang.OutOfMemoryError: 请求的数组大小超过了 VM 限制

原因: 详细消息 "请求的数组大小超过 VM 限制" 表示应用程序(或应用程序使用的 API)尝试分配的数组大小超过了 JVM 实现限制,与可用的堆大小无关。


编辑 4:2021-03-20 更新
JDK 16 已发布,修复了文档问题。

英文:

I was playing with OOM errors today and I found something I can't explain myself.

I try to allocate an array bigger than the heap, expecting a "Requested array size exceeds VM limit" error, but I get a "Java heap space" error instead.

According to the JDK 11 documentation "3 Troubleshoot Memory Leaks > Understand the OutOfMemoryError Exception" :

> Exception in thread thread_name: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
>
> Cause: The detail message "Requested array size exceeds VM limit"
> indicates that the application (or APIs used by that application)
> attempted to allocate an array that is largerthan the heap size. For
> example, if an application attempts to allocate an array of 512 MB,
> but the maximum heap size is 256 MB, then OutOfMemoryError will be
> thrown with the reason “Requested array size exceeds VM limit."


Code :

public class MemOverflow {

    public static void main(final String[] args) {
        System.out.println("Heap max size: " + (Runtime.getRuntime().maxMemory() / 1024 / 1024) + "MB");
        long[] array = new long[100_000_000]; // ~800MB
        System.out.println(array.length);
    }
}

This code works as expected with a heap big enough to store my array :

$ javac MemOverflow.java && java -Xmx800m MemOverflow
Heap max size: 800MB
100000000

However, when I reduce my heap size I get the wrong OOM error :

$ javac MemOverflow.java && java -Xmx100m MemOverflow
Heap max size: 100MB
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
	at MemOverflow.main(MemOverflow.java:5)
# I'm expected the "Requested array size exceeds VM limit" error here

Am I missing something here ? I know I can generate the error I want using Integer.MAX_VALUE as the array size, but I would like it to be thrown by adjusting the heap size.

Java version :

openjdk version "11.0.8" 2020-07-14
OpenJDK Runtime Environment (build 11.0.8+10-post-Ubuntu-0ubuntu120.04)
OpenJDK 64-Bit Server VM (build 11.0.8+10-post-Ubuntu-0ubuntu120.04, mixed mode)

Edit:
Based on @cdalxndr answer, I also tried with the Oracle latest JDK 15 but I get the same result.

java version "15" 2020-09-15
Java(TM) SE Runtime Environment (build 15+36-1562)
Java HotSpot(TM) 64-Bit Server VM (build 15+36-1562, mixed mode, sharing)

Edit 2:
I reported it: JDK-8254804.


Edit 3: 2021-01-19 update

The documentation has been fixed. in the v16-ea (early access).

> Exception in thread thread_name: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
>
> Cause: The detail message "Requested array size exceeds VM limit"
> indicates that the application (or APIs used by that application)
> attempted to allocate an array with a size larger than the JVM
> implementation limit, irrespective of how much heap size is available.


Edit 4 : 2021-03-20 update

JDK 16 is now released with the fixed docs

答案1

得分: 16

引用的文档,理解 OutOfMemoryException

> 线程 thread_name 中的异常: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
>
> 原因: 详细信息中的“Requested array size exceeds VM limit”表示应用程序(或应用程序使用的 API)尝试分配大于堆大小的数组。例如,如果应用程序尝试分配大小为 512 MB 的数组,但最大堆大小为 256 MB,则会抛出 OutOfMemoryError,并显示“Requested array size exceeds VM limit”的原因。
>
> 操作: 通常问题要么是配置问题(堆大小太小),要么是导致应用程序尝试创建巨大数组的 bug(例如,当数组中的元素数使用计算错误大小的算法计算时)。

...是不正确的。

实际上,“requested array size exceeds VM limit” 消息的真正含义是 JVM 实现施加了一个限制。超过此限制将始终导致失败,无论可用的堆空间多少。如果尝试分配接近 Integer.MAX_VALUE 元素的数组,则会出现带有此消息的 OutOfMemoryError。 (这不是固定限制。详见下文。)

相比之下,带有“Java 堆空间”消息的 OutOfMemoryError 意味着如果情况不同,请求本质上是可以满足的:例如,如果其他对象使用的内存较少,或者如果 JVM 的最大堆大小增加了。

我已经重新分配了错误 JDK-8254804 来修复相关的文档。

ArraysSupport.java 中有一个看似相关的注释:

/**
 * 要分配的数组的最大长度(除非必要)。
 * 一些虚拟机在数组中保留了一些头部字。
 * 尝试分配较大的数组可能会导致
 * {@code OutOfMemoryError: Requested array size exceeds VM limit}
 */
public static final int MAX_ARRAY_LENGTH = Integer.MAX_VALUE - 8;

请注意,这是库代码,而不是 Hotspot JVM 代码。这是库代码对不会超过 JVM 可能施加的限制的数组大小的“保守猜测”。问题是 JVM 的最大数组大小限制可能会因不同情况而异,例如使用的垃圾收集器,JVM 是否在 32 位或 64 位模式下运行,甚至是运行在其上的 JVM(Hotspot 或其他版本)。库在这里需要保守一些,因为它需要在不同的 JVM 和不同的情况下运行,且没有标准化的接口返回“允许的最大数组大小”。(实际上,这样的概念可能定义不清,因为不同的数组类型的最大数组大小可能不同,或者甚至可能在不同时刻发生变化。)

英文:

The documentation cited, Understand the OutOfMemoryException,

> Exception in thread thread_name: java.lang.OutOfMemoryError: Requested array size exceeds VM limit
>
> Cause: The detail message "Requested array size exceeds VM limit" indicates that the application (or APIs used by that application) attempted to allocate an array that is larger than the heap size. For example, if an application attempts to allocate an array of 512 MB, but the maximum heap size is 256 MB, then OutOfMemoryError will be thrown with the reason “Requested array size exceeds VM limit."
>
> Action: Usually the problem is either a configuration issue (heap size too small) or a bug that results in an application attempting to create a huge array (for example, when the number of elements in the array is computed using an algorithm that computes an incorrect size).

...is incorrect.

What "requested array size exceeds VM limit" message really means is that there is an limit imposed by the implementation of the JVM. Exceeding this limit will always cause the failure, no matter how much heap space is available. An OutOfMemoryError with this message occurs if an there is an attempt to allocate an array with close to Integer.MAX_VALUE elements. (This is not a fixed limit. See below.)

By contrast, an OutOfMemoryError with the "Java heap space" message means that the request could have been fulfilled if circumstances were different: for example, if less memory were used by other objects, or if the JVM's maximum heap size were increased.

I've repurposed bug JDK-8254804 to fix the documentation in question.

There is a comment that seems relevant in ArraysSupport.java:

/**
 * The maximum length of array to allocate (unless necessary).
 * Some VMs reserve some header words in an array.
 * Attempts to allocate larger arrays may result in
 * {@code OutOfMemoryError: Requested array size exceeds VM limit}
 */
public static final int MAX_ARRAY_LENGTH = Integer.MAX_VALUE - 8;

Note that this is in library code, not in the Hotspot JVM code. It's the library code's conservative guess for an array size that doesn't exceed the limit that might be imposed by the JVM. The problem is that the JVM's maximum array size limit might vary depending on different circumstances, such as the garbage collector in use, whether the JVM is running in 32-bit or 64-bit mode, or even the JVM (Hotspot or other) upon which it is running. The library needs to be a bit conservative here because it needs to run on different JVMs in different circumstances, and there's no standardized interface to the JVM that returns the "maximum allowed array size." (Indeed, such a concept is potentially ill-defined, as the maximum array size might differ for different array types, or it might change from one moment to the next.)

答案2

得分: 4

这可能是 JDK 实现的差异。

您的文档指向 Oracle 网站,但您并未使用 Oracle JDK
您的 OpenJDK 实现可能会以不同的方式处理此异常,因此您不会收到 Requested array size exceeds VM limit 异常。

从我的 zulu-11 jdk ArrayList.java 实现中获取的一些信息:

/**
 * 分配数组的最大大小(除非有必要)。
 * 一些虚拟机在数组中保留一些标题字。
 * 尝试分配更大的数组可能会导致
 * OutOfMemoryError:请求的数组大小超过了虚拟机限制
 */
private static final int MAX_ARRAY_SIZE = Integer.MAX_VALUE - 8;
英文:

This may be a jdk implementation difference.

Your docs point to Oracle site, but you are not using Oracle JDK.
Your OpenJDK implementation may handle this exception differently, so you don't get Requested array size exceeds VM limit exception.

Some info from my zulu-11 jdk ArrayList.java implementation:

/**
 * The maximum size of array to allocate (unless necessary).
 * Some VMs reserve some header words in an array.
 * Attempts to allocate larger arrays may result in
 * OutOfMemoryError: Requested array size exceeds VM limit
 */
private static final int MAX_ARRAY_SIZE = Integer.MAX_VALUE - 8;

huangapple
  • 本文由 发表于 2020年10月14日 00:16:39
  • 转载请务必保留本文链接:https://go.coder-hub.com/64339159.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定