Databricks dbutils.fs.ls() error Py4JSecurityException: Constructor public com.databricks.backend.daemon.dbutils.FSUtilsParallel is not whitelisted

When executing dbtuils.fs.ls() to list files in Databricks I get the error:

py4j.security.Py4JSecurityException: Constructor public com.databricks.backend.daemon.dbutils.FSUtilsParallel(org.apache.spark.SparkContext) is not whitelisted.

I also get the error when I try dbtuils.fs.ls().

Unfortunately, the following failed spark.databricks.pyspark.enablePy4JSecurity False

I have done some research and can someone advise if any of the following will fix the issue:

• Enabling credential passthrough for standard and high-concurrency clusters.

• Configuring credential passthrough and initializing storage resources in ADLS accounts.

• Accessing ADLS resources directly when credential passthrough is enabled.

• Accessing ADLS resources through a mount point when credential passthrough is enable

Thanks

As per MS Document

> The error you are facing is because, when utilizing a High Concurrency cluster and enabling credential pass through, this problem occurs with various library operations.
> When you visit a method that Azure Databricks hasn't expressly designated as safe for Azure Data Lake Storage credential passthrough clusters, this issue is raised.

To workaround you can:

• Use a different cluster mode as a workaround if it describes your situation.
• Update spark.databricks.pyspark.enableProcessIsolation to false for this your cluster needs to be with no isolation shared access mode
• Use standard clusters in your case as a workaround.