如何配置Kudu测试工具以避免“块缓存容量超过内存压力阈值”错误。

huangapple go评论71阅读模式
英文:

How can I configure the Kudu test harness to avoid "Block cache capacity exceeds the memory pressure threshold"

问题

我正试图遵循入门指南中关于使用 KuduTestHarness 的指引。我已创建了以下简单的测试用例。

import org.apache.kudu.test.KuduTestHarness;

import static org.junit.Assert.assertTrue;
import org.junit.Rule;
import org.junit.Test;

public class DemoTest {
    @Rule
    public KuduTestHarness harness = new KuduTestHarness();

    @Test
    public void testDemo() {
        assertTrue(true);
    }
}

但是我在控制台日志中收到以下错误信息。

2020-10-07 11:50:01.060 [cluster stderr printer] INFO  org.apache.kudu.test.cluster.MiniKuduCluster - E1007 11:50:01.059237 17257 block_cache.cc:99] Block cache capacity exceeds the memory pressure threshold (536870912 bytes vs. 498776800 bytes). This will cause instability and harmful flushing behavior. Lower --block_cache_capacity_mb or raise --memory_limit_hard_bytes.

2020-10-07 11:50:01.060 [cluster stderr printer] INFO  org.apache.kudu.test.cluster.MiniKuduCluster - E1007 11:50:01.059262 17257 flags.cc:441] Detected inconsistency in command-line flags; exiting

2020-10-07 11:50:01.100 [main] DEBUG org.apache.kudu.test.cluster.MiniKuduCluster - Response: error {
  code: RUNTIME_ERROR
  message: "failed to start masters: Unable to start Master at index 0: /tmp/kudu-binary-jar1893943400146501302/kudu-binary-1.13.0-linux-x86_64/bin/kudu-master: process exited with non-zero status 1"
}

我尝试向基本构建器添加标志,但没有任何影响。新标志不会显示在日志的标志列表中。

import org.apache.kudu.test.cluster.MiniKuduCluster.MiniKuduClusterBuilder;
...
static {
    MiniKuduClusterBuilder builder = KuduTestHarness.getBaseClusterBuilder();
    builder.addMasterServerFlag("--block_cache_capacity_mb=498776800");
}
...

有人可以指导我正确配置测试工具吗?

英文:

I am trying to follow the guide to using the KuduTestHarness in the Getting Started guide. I have created the following simple test case.

import org.apache.kudu.test.KuduTestHarness;

import static org.junit.Assert.assertTrue;
import org.junit.Rule;
import org.junit.Test;

public class DemoTest{
    @Rule
    public KuduTestHarness harness=new KuduTestHarness();

    @Test
    public void testDemo(){
	assertTrue(true);
    }
}

But I get the following errors in the console log.

2020-10-07 11:50:01,060 [cluster stderr printer] INFO  org.apache.kudu.test.cluster.MiniKuduCluster - E1007 11:50:01.059237 17257 block_cache.cc:99] Block cache capacity exceeds the memory pressure threshold (536870912 bytes vs. 498776800 bytes). This will cause instability and harmful flushing behavior. Lower --block_cache_capacity_mb or raise --memory_limit_hard_bytes.

2020-10-07 11:50:01,060 [cluster stderr printer] INFO  org.apache.kudu.test.cluster.MiniKuduCluster - E1007 11:50:01.059262 17257 flags.cc:441] Detected inconsistency in command-line flags; exiting

2020-10-07 11:50:01,100 [main] DEBUG org.apache.kudu.test.cluster.MiniKuduCluster - Response: error {
  code: RUNTIME_ERROR
  message: "failed to start masters: Unable to start Master at index 0: /tmp/kudu-binary-jar1893943400146501302/kudu-binary-1.13.0-linux-x86_64/bin/kudu-master: process exited with non-zero status 1"
}

I tried adding a flag to the base builder, but it does not have any affect. The new flag does not show up in the list of flags in the logs.

import org.apache.kudu.test.cluster.MiniKuduCluster.MiniKuduClusterBuilder;
...
    static{
	MiniKuduClusterBuilder builder=KuduTestHarness.getBaseClusterBuilder();
	builder.addMasterServerFlag("--block_cache_capacity_mb=498776800");
    }
...

Can someone point me in the right direction for correctly configuring the test harness.

答案1

得分: 0

OK,我通过阅读源代码自己解决了这个问题。我对为什么MiniKuduCluster的相关javdocs不在线上感到困惑。

答案是getBaseClusterBuilder()是一个工厂方法,而不是我之前认为的访问方法。每次都会获得一个新的实例,测试工具类在创建新的规则实例时将使用一个新的构建器,因此您需要在那时注入您的自定义构建器。有一个接受构建器对象的构造函数。

以下代码展示了如何实现。

    public static MiniKuduClusterBuilder builder;

    @BeforeClass
    public static void classInit(){
	builder=KuduTestHarness.getBaseClusterBuilder()
	    .addMasterServerFlag("--block_cache_capacity_mb=475")
	    .addTabletServerFlag("--block_cache_capacity_mb=475");
    }

    @Rule
    public KuduTestHarness harness=new KuduTestHarness(builder);

根本问题是因为我在我的笔记本上使用了内存有限的虚拟机。在这种情况下,我几乎无法做任何事情来解决这个问题,因此在这一点上自定义构建器的能力非常有用。

英文:

OK, I managed to solve this myself by reading the source code. I was puzzled by why the relevant javdocs for the MiniKuduCluster were not online.

The answer is that getBaseClusterBuilder() is a factory method, not an accessor method as I had assumed. You get a new one every time and the test harness class will use a fresh builder when you create a new rule instance, so you need to inject your custom builder at that point. There is a constructor that takes the builder object.

This code shows how it can be done.

    public static MiniKuduClusterBuilder builder;

    @BeforeClass
    public static void classInit(){
	builder=KuduTestHarness.getBaseClusterBuilder()
	    .addMasterServerFlag("--block_cache_capacity_mb=475")
	    .addTabletServerFlag("--block_cache_capacity_mb=475");
    }

    @Rule
    public KuduTestHarness harness=new KuduTestHarness(builder);

The underlying problem was due to using a VM on my laptop with limited memory. There was not much I could do to get around that so the ability to customise the builder at this point is very useful.

huangapple
  • 本文由 发表于 2020年10月7日 19:25:07
  • 转载请务必保留本文链接:https://go.coder-hub.com/64243032.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定