Certain JUnit test take insane amounts of time in Ant

Related: https://stackoverflow.com/questions/123127/ant-junit-tests-are-running-much-slower-via-ant-than-via-ide-what-to-look-at

I am having the same problem as the other question, but I have narrowed the scope a lot.

I have a large suite of unit test (namely 800 test cases) which should be run by Atlassian Bamboo CI. I have rewritten my build script entirely.

When running all tests in my test folder in IntelliJ IDEA, everything takes a 7-minutes reasonable time, including Spring-context-based tests requiring a new instance of H2 populated with tables.

When running in Ant, both locally and on CI, the build takes an insane 1-hour time to test. Putting 8 threads on duty reduces the time to 20 minutes (the slowest job).

I had suspected that the heaviest tests were related to Spring and in memory database. I have investigated a lot today and found a shortlits of bad tests that perform awfully, but I am here because I can't understand the reason.

Most of awfully-performing tests are entirely in-memory with little to none system output. While there is a relatively "large" number of assertion operations, iterations, etc. the same test performs awfully on Ant and performs great on IDE.

Pick the following tests:

• SortedLinkedListTests: 1.163 seconds

I have a SortedLinkedList component (linked list with O(n) sorted insertion) along with its unit tests. No logging framework, no Spring context. Nothing.

Basically the unit tests operate on 10000 iterations. The following very single test took 346 seconds.

@Test
public void testAddWithoutComparator()
{
    final SortedLinkedList<Integer> uut = new SortedLinkedList<>();

    final Random random = new Random();

    for (int i = 0; i < ITERATIONS; i++)
    {
        uut.add(random.nextInt());

        Integer previous = null;
        if (i % 50 == 0)
            for (Integer candidate : uut)
            {
                assertNotNull(candidate);
                if (previous != null)
                    assertTrue(previous.compareTo(candidate) <= 0);
                previous = candidate;
            }
    }
}

Maybe the code is a bit dumb, can be optimized a lot, but it's not so bad, not that bad. On 1 to 10000 iterations, it checks that the list is sorted at every step, not just at the beginning or at the end. Think about 10000 incremental tests.
This test has a complexity O(n^2), and I am happy with that. I am happy that the test, not writing anything to IO, runs in 4 seconds in memory.

Again, a lot can be simplified, or optimized, but 4s vs 346s is insane IMO

• BeanUtilsTests

I have a BeanUtils utility that operates on properties and reflections. This very single test, including an AES encryption operation, takes 25 seconds to run in Ant

@Test
public void testMarshal_Password()
{
    String passwordPlaintext = RandomStringUtils.randomAlphanumeric(25);
    assertThat(BeanUtils.marshal(passwordPlaintext, PASSWORD), is(not(equalTo(passwordPlaintext))));
}

No iterations!!! Internally, it is a switch statement that redirects to an encryption utility. The key is hardcoded in the code, no need to generate random secure keys. Cipher.Init and then encrypt the random string.

It takes 1s in IDE, probably a lot more time to initialize testing runtime than perform the test.

The Ant build

<!-- JUNIT TARGETS -->
<target name="debug-junit">
    <property name="junit.include" value="**/*.java" />
    <condition property="jarOk">
        <resourcecount count="1">
            <fileset id="fs" dir="${project.artifactsDirectory}" includes="${artifact.name}-junit.jar"/>
        </resourcecount>
    </condition>
    <fail message="No jar file. Please package the project first" unless="jarOk"/>

    <mkdir dir="${junit.destDir}"/>

    <junit printsummary="yes" haltonfailure="yes" dir="${basedir}" forkmode="once" logfailedtests="true" maxmemory="1024m" fork="true" showoutput="false">
        <classpath refid="project.classpath.test"/>

        <jvmarg value="-Djava.compiler=NONE"/>
        <jvmarg value="-agentlib:jdwp=transport=dt_socket,address=${junit.debugPort},server=y,suspend=y"/>

        <batchtest todir="${junit.destDir}" haltonerror="false" haltonfailure="false" skipnontests="true">
            <fileset dir="${junit.srcDir}" includes="${junit.include}"/>
            <formatter type="xml" />
        </batchtest>
    </junit>
</target>

<target name="run-junit">
    <property name="junit.include" value="**/*.java" />
    <condition property="jarOk">
        <resourcecount count="1">
            <fileset id="fs" dir="${project.artifactsDirectory}" includes="${artifact.name}-junit.jar"/>
        </resourcecount>
    </condition>
    <fail message="No jar file. Please package the project first" unless="jarOk"/>

    <mkdir dir="${junit.destDir}"/>

    <junit printsummary="yes" haltonfailure="yes" dir="${basedir}" forkmode="once" logfailedtests="true" maxmemory="1024m" fork="true" showoutput="false">
        <classpath refid="project.classpath.test"/>

        <jvmarg value="-Djava.compiler=NONE"/>
        <jvmarg value="-ea"/>

        <batchtest todir="${junit.destDir}" haltonerror="false" haltonfailure="false" skipnontests="true">
            <fileset dir="${junit.srcDir}" includes="${junit.include}"/>
            <formatter type="xml" />
        </batchtest>
    </junit>
</target>

Explanations:

• junit.debugargs is overwritten when I want to attach a debugger to debug tests. Here you don't have debug-specific properties
• fork="true" just to make sure that classpath is not <strike>infected</strike> affected by Ant-related classes
• showoutput="false" to reduce IO. In particular, Bamboo remote agents transmit stdout to master, reducing performance
• forkmode I also have tried perTest mode with parallelism. Now I reverted to single-threading mode because Atlassian Bamboo runs on parallel agents
• logfailedtests="true" failures are important, not all tests fail

Question

Somebody help please? I need my build to run in a barely decent time

Updates

> Update 1: narrowing the scope

I am now trying to use arguments appropriately to run a single test class on my local Ant. I found SortedLinkedListTests to be running slow, so I am now trying to run it alone. It still takes a lot of time.

I have managed to reduce the execution time of BeanUtilsTests by reducting the number of iterations of another test method by a magnitude. And then, even the password test runs like a charm.

> Update 2: suite helps

I was trying to recall what has changed since I rewrote my build from scratch, and replicate.

My previous build had a similar run-junit Ant task that only invoked a single @Suite-annotated class file, which referenced all tests to run manually. This is one of the reasons I rewrote the build. I have checked Bamboo for past logs of tests. SortedLinkedListTests executed in a few seconds with the previous build

This is the old runner. I can't see anything that would drop the performance so hard and bad

<target name="run-junit">
    <condition property="jarOk">
        <resourcecount count="1">
            <fileset id="fs" dir="${dir.publish}" includes="${artifact.name}-junit-*.jar"/>
        </resourcecount>
    </condition>
    <fail message="No jar file. Please package the project first" unless="jarOk"/>

    <mkdir dir="${dir.junitResult}"/>
    <property name="junit.srcDir" value="test"/>
    <fail message="Must set -Djunit.testSuite" unless:set="junit.testSuite"/>

    <junit printsummary="yes" haltonfailure="yes" dir="${basedir}" fork="true" showoutput="true">
        <classpath refid="project.classpath.test"/>
        <classpath location="build/compiled-test"/>

        <batchtest todir="${dir.junitResult}" haltonerror="false" haltonfailure="false" fork="true">
            <fileset dir="${junit.srcDir}" includes="**/${junit.testSuite}.java"/>
            <formatter type="xml"/>
        </batchtest>
        <formatter type="xml"/>
    </junit>
</target>

The following worked for me after hours of try-and-fail

Remove

<jvmarg value="-Djava.compiler=NONE"/>

I am marking this as a community wiki in order for anyone to be able to expand with an explanation!