检测Java中的低可用内存

huangapple go评论84阅读模式
英文:

Detect low available memory in Java

问题

我想检测空闲内存何时低于一定水平(比如15%的空闲内存),然后采取一些操作。
我尝试使用Runtime内存指示器,但似乎得到了不一致的行为。

以下是我的测试应用程序:

import java.util.*;

public class Memory
{
    static final int BYTES_PER_MB = 1024*1024;

    public static void main(String[] args) {

        Queue<byte[]> q = new LinkedList<>();
        int allocated = 0;
        while (true) {
            q.add(new byte[BYTES_PER_MB * 100]);
            allocated += 100;
            System.out.printf("Allocated 100m (total allocated=%d)%n", allocated);
            printMem();
        }
    }

    public static void printMem() {
        long max = Runtime.getRuntime().maxMemory() / BYTES_PER_MB;
        long total = Runtime.getRuntime().totalMemory() / BYTES_PER_MB;
        long free = Runtime.getRuntime().freeMemory() / BYTES_PER_MB;
        long totalFree = free+(max-total); // allocated free + memory that can be allocated
        System.out.printf("MEM: max=%dm, total=%dm, free=%d, (total free=%d)%n", max, total, free, totalFree);
    }
}

以下是我得到的结果:

JVM                                      | 命令                         | OOM前的总空闲
--------------------------------------------------------------------------------------------------
Oracle Java 1.8.0_191-b12 64位服务器      | java -Xmx1024m -Xms10m Memory   | 309
Oracle Java 1.8.0_191-b12 64位服务器      | java -Xmx1024m Memory           | 205
Oracle Java 11.0.2+9-LTS  64位服务器      | java -Xmx1024m -Xms10m Memory   | 12
Oracle Java 11.0.2+9-LTS  64位服务器      | java -Xmx1024m Memory          | 12

这些结果对于 Java 11 是有意义的,但我不理解 Java 8 的结果。

  1. 为什么我无法分配内存,即使我有足够的内存?(我认为这不是碎片化问题,因为堆主要有100M的块,即使我以10M的块分配,它也无法成功)
  2. 为什么如果我指定一个较低的-Xmx,我可以分配的内存更少?它不应该只影响初始堆大小吗?
英文:

I'd like to detect when free memory is low (say 15% free) and take some action.
I tried using the Runtime memory indicators, but I seem to be getting inconsistent behavior.

Here's my test app:

import java.util.*;

public class Memory
{
    static final int BYTES_PER_MB = 1024*1024;
    
	public static void main(String[] args) {
        
        Queue&lt;byte[]&gt; q = new LinkedList&lt;&gt;();
        int allocated = 0;
        while (true) {
            q.add(new byte[BYTES_PER_MB * 100]);
            allocated += 100;
            System.out.printf(&quot;Allocated 100m (total allocated=%d)%n&quot;, allocated);
            printMem();
        }
    }
        
	public static void printMem() {
        long max = Runtime.getRuntime().maxMemory() / BYTES_PER_MB;
        long total = Runtime.getRuntime().totalMemory() / BYTES_PER_MB;
        long free = Runtime.getRuntime().freeMemory() / BYTES_PER_MB;
        long totalFree = free+(max-total); // allocated free + memory that can be allocated
        System.out.printf(&quot;MEM: max=%dm, total=%dm, free=%d, (total free=%d)%n&quot;, max, total, free, totalFree);
    }
}

Here are the results I'm getting:

JVM                                      | Command                         | Total free before OOM
--------------------------------------------------------------------------------------------------
Oracle Java 1.8.0_191-b12 64-Bit Server  | java -Xmx1024m -Xms10m Memory   | 309
Oracle Java 1.8.0_191-b12 64-Bit Server  | java -Xmx1024m Memory           | 205
Oracle Java 11.0.2+9-LTS  64-Bit Server  | java -Xmx1024m -Xms10m Memory   | 12
Oracle Java 11.0.2+9-LTS  64-Bit Server  | java -Xmx1024m Memory&quot;          | 12

The results make sense for Java 11, but I don't understand the results for Java 8.

  1. Why am I unable to allocate memory, even though I have more than enough? (I don't think it's a fragmentation issue, since the heap has mostly 100m chunks + it also fails if I allocate in 10m chunks)
  2. Why can I allocate even less if I specify a low -Xmx? Shouldn't it only affect the initial heap size?

答案1

得分: 4

  1. 在Java 8和Java 11之间,默认的垃圾回收(GC)行为是不同的。调整GC参数是提高内存效率的一种方法。在本地测试中,将 '-XX:+UseAdaptiveGCBoundary' 设置在Java 8中导致了接近堆内存限制时的OOM。

  2. -Xmx 控制允许为堆分配的最大内存量。初始堆大小可以通过:-Xms 来控制。如果这两个值相同,那么堆大小将固定为设置的值。

...

为了更好地理解不同JVM版本之间的内存分配情况,查看一些可视化图表可能会有帮助。下面的图表展示了在Java 8和Java 15中为类似上面所示的程序分配内存时的内存分配情况(源代码在下面包含):

检测Java中的低可用内存

检测Java中的低可用内存

检测Java中的低可用内存

图例说明:

  1. Total Free - 包括已分配的空闲内存和仍可从系统中请求的内存,直到达到最大内存。 (test中的 totFre)
  2. JVM Allocated (Free) - 可在已为JVM分配的内存(上面的总内存)中使用的内存。 (test中的 freMem)
  3. JVM Allocated - 当前为JVM分配的内存。随着对象的创建或需要更多空间,这可能会有所变化,最多达到最大内存。 (test中的 totMem)
  4. Max Memory - JVM允许自己使用的最大内存。可以通过 -Xmx 标志进行配置。 (test中的 maxMem)

...

为什么Java 8和15之间会有差异?

当Java 15配置为使用Java 8中使用的相同垃圾回收器(Parallel GC)时,堆的增长方式类似于Java 8。不同之处在于,在Java 15中,在耗尽所有内存之前会发生内存不足的情况。这表明GC机制和使用的参数会影响在需要更多堆空间时堆增长的数量,以及可以使用的总有效堆容量。这在Java 8中运行相同的测试时进一步得到证明,此时使用串行GC时会正确使用所有内存。在示例测试中,将并行GC配置为使用 '-XX:+UseAdaptiveGCBoundary' 可以在失败之前改善最大内存利用率。

...

测试源代码:

public static void main(String[] args)
{
    long totalProgramAllocated = 0;
    Queue<byte[]> dataQueue = new LinkedList<>();
    while (true)
    {
        System.out.println("Total Allocated: " + (totalProgramAllocated >> 20));
        byte[] nextAllocatedChunk = createArray(25);
        dataQueue.add(nextAllocatedChunk);
        totalProgramAllocated += nextAllocatedChunk.length;
        System.out.println(new MemoryUsage());
    }
}

private static final Random random = new Random();
private static byte[] createArray(int megabytes){ byte[] data = new byte[(1 << 20) * megabytes]; random.nextBytes(data); return data; }

public static final class MemoryUsage
{
    public final long maxMem = Runtime.getRuntime().maxMemory();
    public final long totMem = Runtime.getRuntime().totalMemory();
    public final long freMem = Runtime.getRuntime().freeMemory();
    public final long totFre = maxMem - (totMem - freMem);
    @Override public String toString(){ return String.join(",", Long.toString(maxMem), Long.toString(totMem), Long.toString(freMem), Long.toString(totFre)); }
}
英文:
  1. The default GC behavior between Java 8 and Java 11 is different. Tweaking GC parameters is one way to improve memory efficiency. In local tests, setting '-XX:+UseAdaptiveGCBoundary' in Java 8 caused OOM closer to heap memory limits.
  2. -Xmx controls the maximum amount of memory which is allowed to be allocated for the heap. The initial heap size can be controlled via: -Xms. If both values are the same, then the heap size will be fixed to the value set.

..

To better understand the memory breakdown between JVM versions it may be helpful to look at some visualizations. The below charts show the memory breakdown as memory is allocated by a similar program to the one included above in Java 8 and Java 15 (source included below):

检测Java中的低可用内存

检测Java中的低可用内存

检测Java中的低可用内存

Legend Description for Charts:
1) Total Free - Free memory including already allocated free memory and memory which may still be requested from the system up to Max memory. (totFre in test)
2) JVM Allocated (Free) - The memory which is available for usage within the memory already allocated for the JVM (Total Memory above). (freMem in test)
3) JVM Allocated - The memory currently allocated for the JVM. This may vary as objects are created or more space is needed up to Max Memory. (totMem in test)
4) Max Memory - The maximum memory the JVM will allow itself to use. Can be configured via -Xmx flag. (maxMem in test)

..

Why the discrepancy between Java 8 and 15?

When Java 15 is configured to use the same garbage collector used in Java 8 (Parallel GC), the heap grows similarly to how it does in Java 8. One difference is that in Java 15, all memory is consumed before failing with Out of Memory. This suggests that both the GC mechanism and parameters used impact the amount by which the heap is grown when more heap space is needed and the total effective heap capacity which can be used. This is further evidenced by running the same test in Java 8 with Serial GC in which case all memory is properly used. In the sample test, configuring the Parallel GC to use '-XX:+UseAdaptiveGCBoundary' improved max memory utilization before failure.

..

Test Source:

    public static void main(String[] args)
	{
		long totalProgramAllocated = 0;
		Queue&lt;byte[]&gt; dataQueue = new LinkedList&lt;&gt;();
		while (true)
		{
			System.out.println(&quot;Total Allocated: &quot; + (totalProgramAllocated &gt;&gt; 20));
			byte[] nextAllocatedChunk = createArray(25);
			dataQueue.add(nextAllocatedChunk);
			totalProgramAllocated += nextAllocatedChunk.length;
			System.out.println(new MemoryUsage());
		}
	}

	private static final Random random = new Random();
	private static byte[] createArray(int megabytes){ byte[] data = new byte[(1 &lt;&lt; 20) * megabytes]; random.nextBytes(data); return data; }
	
	public static final class MemoryUsage
	{
		public final long maxMem = Runtime.getRuntime().maxMemory();
		public final long totMem = Runtime.getRuntime().totalMemory();
		public final long freMem = Runtime.getRuntime().freeMemory();
		public final long totFre = maxMem - (totMem - freMem);
		@Override public String toString(){ return String.join(&quot;,&quot;, Long.toString(maxMem), Long.toString(totMem), Long.toString(freMem), Long.toString(totFre)); }
	}

huangapple
  • 本文由 发表于 2020年9月23日 20:21:26
  • 转载请务必保留本文链接:https://go.coder-hub.com/64027855.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定