ProblemDetails don't work when the application is started via a WebApplicationFactory (i.e in an integration test)

I have an ASP.NET Core application that is correctly configured to return ProblemDetails upon any unhandled exaception. This works as expected, exception is thrown and the response is a HTTP500 with the expected ProblemDetails JSON body. However, if the same application is spun up as part of a integration test using WebApplicationFactory<Program>, ProblemDetails are not returned in the same failed request body. The whole ProblemDetails mechanism just seems to stop functioning. This seems like a bug but I'm curious if anyone has this working or if there are any workarounds?

For reference, ProblemDetails are enabled in Program.cs with;

services.AddProblemDetails()
app.UseExceptionHandler()

Maybe you're using XUnit, which doesn't directly support console output, but you can use try...catch... to get error messages.

For example:

BasicTests:

public class BasicTests 
    : IClassFixture<WebApplicationFactory<Program>>
{
    private readonly WebApplicationFactory<Program> _factory;
    ITestOutputHelper output;

    public BasicTests(WebApplicationFactory<Program> factory, ITestOutputHelper output)
    {
        _factory = factory;
        this.output= output;
    }

    [Theory]
    [InlineData("/About")]
    public async Task Get_EndpointsReturnSuccessAndCorrectContentType(string url)
    {
        // Arrange
        var client = _factory.CreateClient();
        // Act
        var response = await client.GetAsync(url);
        // Assert
        try
        {
            //don't use this line
            //response.EnsureSuccessStatusCode(); // Status Code 200-299
            Assert.Equal(HttpStatusCode.OK, response.StatusCode);
            
        }
        catch (XunitException e)
        {
            output.WriteLine(e.InnerException.Message);
        }
    }
}

About:

public async Task<IActionResult> OnGet()
{
    throw new NullReferenceException();
}

Test Result:

Another way is to not use XUnit, use NUnit or MSTest. Take MSTest as an example:

UnitTest1:

[TestClass]
public class UnitTest1
{
    private HttpClient _httpClient;

    public UnitTest1()
    {
        var webAppFactory = new WebApplicationFactory<Program>();
        _httpClient = webAppFactory.CreateDefaultClient();
    }

    [TestMethod]
    public async Task WeatherForcast()
    {
        var response = await _httpClient.GetAsync("/WeatherForecast");
        var stringResult = await response.Content.ReadAsStringAsync();
        response.EnsureSuccessStatusCode();
    }
}

Api:

[ApiController]
[Route("[controller]")]
public class WeatherForecastController : ControllerBase
{
    private static readonly string[] Summaries = new[]
    {
    "Freezing", "Bracing", "Chilly", "Cool", "Mild", "Warm", "Balmy", "Hot", "Sweltering", "Scorching"
};

    private readonly ILogger<WeatherForecastController> _logger;

    public WeatherForecastController(ILogger<WeatherForecastController> logger)
    {
        _logger = logger;
    }

    [HttpGet(Name = "GetWeatherForecast")]
    public IEnumerable<WeatherForecast> Get()
    {
        throw new NullReferenceException();
    }
}

Test Result:

This is a bug in .NET 7, fixed in 8.