Blazor WASM project forces to authenticate every time the project is started

I have a Blazor WASM and a ASP.NET Core Web API setup to work with Azure Active Directory for Authentication (setup according to the docs with additional scenarios since in my case the two are separate projects).

Authentication on the WASM project as well as API via accessed toke works quite fine, however the login procedure has to be done every time the projects are started (with MFA this becomse quite a hassle).

To avoid this I have added the following line to Program.cs of the WASM project:

builder.Services.AddMsalAuthentication(options =>
{
    // MSAL setup config with AAD here

    if(builder.HostEnvironment.IsEnvironment("Development"))
        options.ProviderOptions.Cache.CacheLocation = "localStorage";
});

Cool, now I don't need to login everytime anymore, however unfortunately with this the Authorization on the API will fail:

> info: Microsoft.AspNetCore.Authorization.DefaultAuthorizationService2
Authorization failed. These requirements were not met:
DenyAnonymousAuthorizationRequirement: Requires an authenticated user.

Is it possible to solve this issue? Will users in production be forced to login every time they access the site as well? Shouldn't a token be valid for a certain amount of time?

Edit
Some additional Info. The HttpClient is configured with a CustomAuthorizationMessageHandler as follows:

Program.cs

builder.Services.AddHttpClient("MyAPI", client =>
        client.BaseAddress = new Uri("https://localhost:7124"))
            .AddHttpMessageHandler<CustomAuthorizationMessageHandler>();

CustomAuthorizationMessageHandler.cs

public class CustomAuthorizationMessageHandler : AuthorizationMessageHandler
{
    public CustomAuthorizationMessageHandler(IAccessTokenProvider provider,
        NavigationManager navigation)
        : base(provider, navigation)
    {
        ConfigureHandler(
            authorizedUrls: new[] { "https://localhost:7124" },
            scopes: new[] { "api://<<some id>>/API.Access" });
    }
}

I have a test in my side with your configuration options.ProviderOptions.Cache.CacheLocation = "localStorage"; which makes me no need to sign in again, and I tried to call the web api which is protected by Azure AD and it worked as well. But I don't have MFA in my App.

Here's my code about calling API, I'm afraid you can check if the token generated correctly when the issue happened.

@page "/profile"
@using Microsoft.AspNetCore.Authorization
@using Microsoft.AspNetCore.Components.WebAssembly.Authentication;
@attribute [Authorize]
@inherits UserProfileBase
@inject IAccessTokenProvider TokenProvider

<h3>User Profile</h3>
<button @onclick="call">
    call api
</button>
<div>@result</div>


@code {

    private string result = "no data now";

    private async Task call()
    {
        var http = new HttpClient();
        http.BaseAddress = new Uri("https://localhost:7018/WeatherForecast");

        var tokenResult = await TokenProvider.RequestAccessToken(
            new AccessTokenRequestOptions
                {
                    Scopes = new[] { "api://aad_Client_id/Tiny.Read" }
                });

        if (tokenResult.TryGetToken(out var token))
        {
            http.DefaultRequestHeaders.Add("Authorization",
                $"Bearer {token.Value}");
            result = await http.GetStringAsync("https://localhost:7018/WeatherForecast");
        }
    }
}