Databricks - not allowing to write output to folder using key stored in azure keyvault

I'm currently facing a weird issue.

I recently changed the way i handle the secrets (storage key) inside my databricks project, using azure keyvault and secret scope.

When i load the input file from a blob storage it works fine.
But when i try to write the output i get the following error:

shaded.databricks.org.apache.hadoop.fs.azure.AzureException: Unable to access container analysis in account [REDACTED].blob.core.windows.net using anonymous credentials, and no credentials found for them in the configuration.

I did set on the spark conf level the key: spark.conf.set("fs.azure.account.key.ACCOUNTNAME.blob.core.windows.net", dbutils.secrets.get('scope', 'STORAGE-KEY'))

Now when i hardcode in the cluster configuration in the Spark config section this fs.azure.account.key.sparkstorageprod.blob.core.windows.net STORAGE-KEY, it works fine and i'm abble to write the output.

I would like to know how to avoid hardcoding the key value in the spark conf section in the cluster configuration, and only rely on the secret scope and the azure key vault.

I tried the same in my environment and able to write it to storage account without any issue.

Recheck all the Secret scope steps again.

>I would like to know how to avoid hardcoding the key value in the spark conf section in the cluster configuration, and only rely on the secret scope and the azure key vault.

You can use the Secret scope in the cluster level without hardcoding the secret like below. Specify the secret name and scope in {{}}.

Ex: spark.hadoop.fs.azure.account.key.<account_name>.blob.core.windows.net {{secrets/<secret-scope-name>/<secret-name>}}

My Demo:

Cluster configurations:

You can see that I am able to read and write after using secret scope at cluster level.