Create new databricks cluster from ADF linked service with InitScripts from abfss (azure blob )

Recently databricks depreciated DBFS init script, And I tried to set up databricks linked service with InitScripts from abfss in ADF, I am getting a file not found error.

The new cluster configuration is like below

But same tried from Databricks its worked, because we have option to configure which is init script location type.

But in ADF , I couldn't see any option to do the same ,

Please help me on this to resolve the issue.

I need to create new databricks cluster with init script read from azure blob storage(abfss) for every pipeline excecution.

• You can use the rest call to create the cluster with init script as required with abfss and then use this cluster in the databricks notebook directly.
• You can use the we activity to call the Clusters 2.0 REST API to create a cluster as specified in this document with authentication header where you specify bearer token (access token). The following is the body that you can use (You might have to add clusted configuration as well):

{
    "num_workers": null,
    "autoscale": {
        "min_workers": 2,
        "max_workers": 8
    },
    "cluster_name": "cluster1",
    "spark_version": "7.3.x-scala2.12",
    "spark_conf": {},
    "node_type_id": "Standard_D3_v2",
    "custom_tags": {},
    "spark_env_vars": {
        "PYSPARK_PYTHON": "/databricks/python3/bin/python3"
    },
    "autotermination_minutes": 120,
    "init_scripts": <script_path_here_as_specified_in_document>
}

• Use a wait activity until the cluster is created. I have used wait time as 300 seconds.

• Finally, use the cluster_id returned by the web activity for newly created cluster to run the notebook as @activity('Web1').output.cluster_id.

If you specified init script on abfss://... then you need to specify corresponding spark.hadoop.fs... configurations in the "Cluster Spark conf" section (fill the data in <...> with corresponding values):

• spark.hadoop.fs.azure.account.auth.type set to OAuth
• spark.hadoop.fs.azure.account.oauth.provider.type set to org.apache.hadoop.fs.azurebfs.oauth2.ClientCredsTokenProvider
• spark.hadoop.fs.azure.account.oauth2.client.endpoint set to https://login.microsoftonline.com/<tenant_id>/oauth2/token
• spark.hadoop.fs.azure.account.oauth2.client.id set to <client_id>
• spark.hadoop.fs.azure.account.oauth2.client.secret set to {{secrets/<secret-scope>/<secret-key-name>}} (it will fetch the key from the secret scope that contains client secret)