Changing JUnit test order breaks tests using validation messages from multiple JAR files per Hibernate Validator's PlatformResourceBundleLocator

I have a Gradle Spring Boot app running on Java 11 using Hibernate Validator. The app uses multiple custom library JARs with custom validation constraint annotations, each with its own ValidationMessages.properties files containing default messages for those annotations. This is supported using the built-in functionality in Hibernate's PlatformResourceBundleLocator to aggregate ValidationMessages.properties files from multiple JAR files into a single bundle:

@Configuration
public class ValidationConfig {
    @Bean
    public LocalValidatorFactoryBean validator() {
        PlatformResourceBundleLocator resourceBundleLocator =
                new PlatformResourceBundleLocator(ResourceBundleMessageInterpolator.USER_VALIDATION_MESSAGES, null, true);

        LocalValidatorFactoryBean factoryBean = new LocalValidatorFactoryBean();
        factoryBean.setMessageInterpolator(new ResourceBundleMessageInterpolator(resourceBundleLocator));
        return factoryBean;
    }
}

jar1: ValidationMessages.properties

com.example.CustomValidation1.message=My custom message 1

jar2: ValidationMessages.properties

com.example.CustomValidation2.message=My custom message 2

There are a fair number of unit & integration tests in the project which test the validation functionality. Some of these tests autowire in the Spring aggregated message validator bean. Some of the tests (in particular ones which predate the application's usage of multiple ValidationMessages.properties) do not depend on the exact messages returned, and use the default Spring validator without the message aggregation. While it may make sense to update the older tests, in the interest of priorities and available time that task has been deferred to the future.

The message aggregation functionality is working as expected when I run the application. It is also working as expected when I run the tests on my local machine. However, when my project's tests are run on the build server via the Jenkins continuous integration tool, some of the validation tests fail.

I have determined the failure occurs based on the order in which the JUnit test classes are run. The tests are being run in a different order locally than they are on Jenkins (which is allowed since the Gradle JUnit plugin does not guarantee test class execution order). Specifically, whether any tests fail vary based on whether or not a test which uses a message aggregation validator runs before one that uses a validator without message aggregation.

I have been able to boil down the issue to a simple re-creatable failure within a single test class, as follows. In the interest of example simplicity, the @CustomValidation1 & @CustomValidation2 validation annotations have been written to always fail validation.

import com.example.CustomValidation1;
import com.example.CustomValidation2;
import org.hibernate.validator.messageinterpolation.ResourceBundleMessageInterpolator;
import org.hibernate.validator.resourceloading.PlatformResourceBundleLocator;
import org.junit.FixMethodOrder;
import org.junit.Test;
import org.junit.runners.MethodSorters;
import org.springframework.validation.beanvalidation.LocalValidatorFactoryBean;

import javax.validation.ConstraintViolation;
import javax.validation.Validator;
import java.util.Set;
import java.util.stream.Collectors;

import static org.junit.Assert.assertEquals;

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class ValidationMessageTests {
    private Validator aggregateMessageValidator = createAggregateMessageValidator();

    private Validator standardValidator = createBasicValidator();

    private Validator createBasicValidator() {
        LocalValidatorFactoryBean factoryBean = new LocalValidatorFactoryBean();
        factoryBean.afterPropertiesSet();
        return factoryBean;
    }

    private Validator createAggregateMessageValidator() {
        PlatformResourceBundleLocator resourceBundleLocator =
                new PlatformResourceBundleLocator(ResourceBundleMessageInterpolator.USER_VALIDATION_MESSAGES, null, true);

        LocalValidatorFactoryBean factoryBean = new LocalValidatorFactoryBean();
        factoryBean.setMessageInterpolator(new ResourceBundleMessageInterpolator(resourceBundleLocator));
        factoryBean.afterPropertiesSet();
        return factoryBean;
    }

    @Test
    public void test1() {
        Set<ConstraintViolation<MyInput>> violations = aggregateMessageValidator.validate(new MyInput());
        assertEquals(Set.of("My custom message 1", "My custom message 2"),
                violations.stream().map(ConstraintViolation::getMessage).collect(Collectors.toSet()));
    }

    @Test
    public void test2() {
        Set<ConstraintViolation<MyInput>> violations = standardValidator.validate(new MyInput());
        assertEquals(2, violations.size());
    }

    @CustomValidation1
    @CustomValidation2
    private static class MyInput {
    }
}

When test1 is run before test2, both tests pass. However, when test2 is renamed to test0 so that the it runs before test1, test0 passes but test1 fails with the following error:

java.lang.AssertionError: 
Expected :[My custom message 1, My custom message 2]
Actual   :[My custom message 1, {com.example.CustomValidation2.message}]

Why does changing the test order cause these tests to fail, and how do I fix it?

The code is currently using Spring Boot 2.2.4.RELEASE with Hibernate Validator 6.0.18.Final.

This underlying cause of this issue is that ResourceBundles are cached by default. Per the ResourceBundle JavaDocs:

> Resource bundle instances created by the getBundle factory methods are cached by default, and the factory methods return the same resource bundle instance multiple times if it has been cached.

Due to this caching, whichever test first causes Hibernate Validator to load the ValidationMessages.properties file(s) will determine which ResourceBundle is used for all subsequent tests. When a test that is not using the customized validation configuration is run first, a non-aggregated PropertyResourceBundle will be cached for ValidationMessages, rather than the desired AggregateResourceBundle. When the PlatformResourceBundleLocator loads the resource bundle, the aggregation logic is ignored since the already cached ResourceBundle is used instead.

The fix is straightforward. ResourceBundle has a clearCache method to clear the cache, which can be called before Hibernate Validator would retrieve the bundle to create a validation message:

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class ValidationMessageTests {
    @Before
    public void setup() {
        ResourceBundle.clearCache();
    }

    // The rest of this test class is unchanged
}