Azure Function: Error building configuration in an external startup class

I am using Dependency Injection to load the Azure App Config Connection string and it's KV references credentials using a DefaultAzureCredential. This config works on all other functions we have developed and deployed in the same way. Only this one is causing me issues.

NOTE: The function works as expected in Azure despite the error message. What does this error even mean if the app works.

AZFD0005
Diagnostic event
Error code
AZFD0005
Level
Error
Message
Error building configuration in an external startup class.
Details
Microsoft.Azure.WebJobs.Script.ExternalStartupException : Error building configuration in an external startup class. ---> Microsoft.Extensions.Configuration.AzureAppConfiguration.KeyVaultReferenceException : ManagedIdentityCredential authentication failed: Service request failed. Status: 400 (Bad Request) Content: Headers: Date: Wed, 17 May 2023 08:33:29 GMT Server: Kestrel Transfer-Encoding: chunked X-CORRELATION-ID: REDACTED Content-Type: application/json; charset=utf-8 See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/managedidentitycredential/troubleshoot. ErrorCode:, Key:AADCosmosDbPrimaryKeyKV, Label:, Etag:RkLTY1--bNFdppGrHvS5fQZQmcxXnmgd_yTEv82Vhbc, SecretIdentifier:https://uksgmdaiaadkvdev.vault.azure.net/secrets/AADCosmosDbPrimaryKeyKVDEV ---> Azure.Identity.AuthenticationFailedException : ManagedIdentityCredential authentication failed: Service request failed. Status: 400 (Bad Request) Content: Headers: Date: Wed, 17 May 2023 08:33:29 GMT Server: Kestrel Transfer-Encoding: chunked X-CORRELATION-ID: REDACTED Content-Type: application/json; charset=utf-8 See the troubleshooting guide for more information. https://aka.ms/azsdk/net/identity/managedidentitycredential/troubleshoot ---> Azure.RequestFailedException : Service request failed. Status: 400 (Bad Request) Content: Headers: Date: Wed, 17 May 2023 08:33:29 GMT Server: Kestrel Transfer-Encoding: chunked X-CORRELATION-ID: REDACTED Content-Type: application/json; charset=utf-8 at async Azure.Identity.ManagedIdentitySource.HandleResponseAsync(Boolean async,TokenRequestContext context,Response response,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Identity.ManagedIdentitySource.AuthenticateAsync(Boolean async,TokenRequestContext context,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Identity.ManagedIdentityClient.AuthenticateAsync(Boolean async,TokenRequestContext context,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async,TokenRequestContext requestContext,CancellationToken cancellationToken) End of inner exception at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex,String additionalMessage) at async Azure.Identity.ManagedIdentityCredential.GetTokenImplAsync(Boolean async,TokenRequestContext requestContext,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Identity.ManagedIdentityCredential.GetTokenAsync(TokenRequestContext requestContext,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Identity.DefaultAzureCredential.GetTokenFromSourcesAsync(TokenCredential[] sources,TokenRequestContext requestContext,Boolean async,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async,TokenRequestContext requestContext,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Azure.Identity.CredentialDiagnosticScope.FailWrapAndThrow(Exception ex,String additionalMessage) at async Azure.Identity.DefaultAzureCredential.GetTokenImplAsync(Boolean async,TokenRequestContext requestContext,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Identity.DefaultAzureCredential.GetTokenAsync(TokenRequestContext requestContext,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.AuthenticateRequestAsync(HttpMessage message,Boolean async) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Security.KeyVault.ChallengeBasedAuthenticationPolicy.ProcessCoreAsync(HttpMessage message,ReadOnlyMemory1 pipeline,Boolean async) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Core.Pipeline.RedirectPolicy.ProcessAsync(??) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Core.Pipeline.RetryPolicy.ProcessAsync(??) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Core.Pipeline.RetryPolicy.ProcessAsync(??) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Core.Pipeline.HttpPipeline.SendRequestAsync(Request request,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Security.KeyVault.KeyVaultPipeline.SendRequestAsync(??) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Security.KeyVault.KeyVaultPipeline.SendRequestAsync[TResult](??) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Azure.Security.KeyVault.Secrets.SecretClient.GetSecretAsync(String name,String version,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureKeyVault.AzureKeyVaultSecretProvider.GetSecretValue(Uri secretUri,String key,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureKeyVault.AzureKeyVaultKeyValueAdapter.ProcessKeyValue(ConfigurationSetting setting,CancellationToken cancellationToken) End of inner exception at async Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureKeyVault.AzureKeyVaultKeyValueAdapter.ProcessKeyValue(ConfigurationSetting setting,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.ProcessAdapters(ConfigurationSetting setting,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.SetData(IDictionary2 data,Boolean ignoreFailures,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at async Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.LoadAll(Boolean ignoreFailures,CancellationToken cancellationToken) at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw() at Microsoft.Extensions.Configuration.AzureAppConfiguration.AzureAppConfigurationProvider.Load() at Microsoft.Extensions.Configuration.ConfigurationRoot..ctor(IList`1 providers) at Microsoft.Extensions.Configuration.ConfigurationBuilder.Build() at IVCE.Azure.DayForce_PMS_Housekeeping.Startup.ConfigureAppConfiguration(IFunctionsConfigurationBuilder builder) at D:\a\1\s\DayForce-PMS-Housekeeping\Startup.cs : 41 at Microsoft.Azure.Functions.Extensions.DependencyInjection.FunctionsStartup.Configure(WebJobsBuilderContext context,IWebJobsConfigurationBuilder builder) at Microsoft.Azure.WebJobs.WebJobsBuilderExtensions.ConfigureAndLogUserConfigurationProviders(IWebJobsConfigurationStartup startup,WebJobsBuilderContext context,IWebJobsConfigurationBuilder builder,ILoggerFactory loggerFactory) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsBuilderExtensions.cs : 342 at Microsoft.Azure.WebJobs.WebJobsBuilderExtensions.UseWebJobsConfigurationStartup(IWebJobsConfigurationBuilder builder,Type startupType,WebJobsBuilderContext context,ILoggerFactory loggerFactory) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsBuilderExtensions.cs : 327 at Microsoft.Azure.WebJobs.WebJobsBuilderExtensions.UseExternalConfigurationStartup(IWebJobsConfigurationBuilder builder,IWebJobsStartupTypeLocator startupTypeLocator,WebJobsBuilderContext context,ILoggerFactory loggerFactory) at C:\projects\azure-webjobs-sdk-rqm4t\src\Microsoft.Azure.WebJobs.Host\Hosting\WebJobsBuilderExtensions.cs : 367 at Microsoft.Azure.WebJobs.Script.ScriptHostBuilderExtensions.<>c__DisplayClass7_3.<AddScriptHostCore>b__8(IWebJobsStartupTypeLocator locator) at /_/src/WebJobs.Script/ScriptHostBuilderExtensions.cs : 246 End of inner exception
Hit count
281
Timestamp
May 17, 2023 at 3:30:45 PM GMT+1
Help link
https://go.microsoft.com/fwlink/?linkid=2224847

Here is the DI Startup.cs file used.

public override void ConfigureAppConfiguration(IFunctionsConfigurationBuilder builder)
{
FunctionsHostBuilderContext context = builder.GetContext();

        var settings = builder.ConfigurationBuilder
            .AddJsonFile(Path.Combine(context.ApplicationRootPath, "appsettings.json"), optional: true, reloadOnChange: false)
            .AddJsonFile(Path.Combine(context.ApplicationRootPath, $"appsettings.{context.EnvironmentName}.json"), optional: true, reloadOnChange: false)
                .AddEnvironmentVariables()
                    .Build();


        var o = new DefaultAzureCredentialOptions();
        o.VisualStudioTenantId = settings["AzureAd:TenantId"];

        _azureConfiguration = new ConfigurationBuilder()
          .AddEnvironmentVariables()
          .AddAzureAppConfiguration(options =>
          {
              options.Connect(settings["ConnectionStrings:AzureAppConfig"])
                       // .Select(KeyFilter.Any, settings["DOTNET_ENVIRONMENT"])
                       .ConfigureKeyVault(kv =>
                       {
                        kv.SetCredential(new DefaultAzureCredential(o));
                       })
                      .UseFeatureFlags()
                      .ConfigureRefresh(refresh =>
                      {
                          refresh.Register(".appconfig.featureflag/IBERIA_AAD_FEATURE", settings["DOTNET_ENVIRONMENT"], true)
                                 .SetCacheExpiration(TimeSpan.FromSeconds(1));
                      });
              _refresher = options.GetRefresher();
          })
          .Build();


    }

    public override void Configure(IFunctionsHostBuilder builder)
    {
        var moduleSettingsListTask = _azureConfiguration.DeserialiseForAsync<ModuleSettingsContext>("PMS_Integration").GetAwaiter();
        var moduleSettingsList = moduleSettingsListTask.GetResult();
        _moduleSettingsContext = moduleSettingsList.FirstOrDefault(m => m.AppTag == "PMS_ESP");
        builder.Services.AddScoped(c => _azureConfiguration);
        builder.Services.AddSingleton(new CosmosClient(_moduleSettingsContext.CosmosDbEndpoint, _azureConfiguration["AADCosmosDbPrimaryKeyKV"], new CosmosClientOptions()));
        builder.Services.AddScoped<ICosmosDbService, CosmosDbService>();
    }

Ok resolved - spotted that in the appsettings.json file I had set the ClientId to the wrong guid. Should be the ClientId of the USer-Assigned-Managed Identity and not the System-Assigned-Managed-Identity.