How to Authenitcate Access to ADLS from Databricks without creating Service Principle with Using AD App Registrations to Mount a Drive

huangapple go评论53阅读模式
英文:

How to Authenitcate Access to ADLS from Databricks without creating Service Principle with Using AD App Registrations to Mount a Drive

问题

It is possible to mount an ADLS Gen2 account/container without configuring service principal credentials to access storage. The following article states that it's possible to access an ADLS storage account without first configuring a service principal.

https://learn.microsoft.com/en-us/azure/databricks/data-governance/credential-passthrough/adls-passthrough

You can mount a drive in Databricks without registering an application by using the following code:

mount = "/mnt/lake"

if check(mount) == 1:
    resultMsg = "<div>%s is already mounted. </div>" % mount
else:
    dbutils.fs.mount(
        source = f"abfss://root@{Lake}.dfs.core.windows.net/",
        mount_point = mount,
        extra_configs = configs)
    resultMsg = "<div>%s was mounted. </div>" % mount

However, if you encounter the error "IllegalArgumentException: Unsupported Azure Scheme: abfss," it might be due to incorrect configuration or a problem with the URL you are using. Double-check your configuration and ensure that your storage account and container details are correct.

英文:

Is it possible to mount ADLS Gen2 account/container without configuring service principle credentials to access storage?

The following article states that it's possible to access a ADLS storage account with first configuring a service principle.

https://learn.microsoft.com/en-us/azure/databricks/data-governance/credential-passthrough/adls-passthrough

I normally mount a drive in Databricks by first creating a service principle by Registering an Azure AD application using App Registrations. I would use the following code to mount:

if check(mount)==1:
  resultMsg = &quot;&lt;div&gt;%s is already mounted. &lt;/div&gt;&quot; % mount
else:
  dbutils.fs.mount(
  source = f&quot;abfss://root@{Lake}.dfs.core.windows.net/&quot;,
  mount_point = mount,
  extra_configs = configs)
  resultMsg = &quot;&lt;div&gt;%s was mounted. &lt;/div&gt;&quot; % mount

At present I am waiting for the administrator at my place of work to give me the permissions to register applications.

So, I was wondering if I could mount a drive from Databricks without first registering an application?

I tried @JayashankarGS suggestion as follows:

mount = &quot;/mnt/lake&quot;

if check(mount)==1:
  resultMsg = &quot;&lt;div&gt;%s is already mounted. &lt;/div&gt;&quot; % mount
else:
  dbutils.fs.mount(
  source = f&quot;abfss://root@{Lake}.dfs.core.windows.net/&quot;,
  mount_point = mount,
  extra_configs = configs)
  resultMsg = &quot;&lt;div&gt;%s was mounted. &lt;/div&gt;&quot; % mount

But I got the error;

IllegalArgumentException: Unsupported Azure Scheme: abfss

答案1

得分: 1

你可以尝试使用账户密钥来挂载存储。

我在我的项目中也做了类似的事情!

你可以使用下面的代码作为参考:

Container_name = "root"
storage_account = "lake"
key = ""

url = "wasbs://" + container_name + "@" + storage_account + ".blob.core.windows.net/"
config = "fs.azure.account.key." + storage_account + ".blob.core.windows.net"

mount_folder = "/mnt/lake"
mounted_list = dbutils.fs.mounts()

mounted_exist = False
for item in mounted_list:
    if mount_folder in item[0]:
        mounted_exist = True
        break

if not mounted_exist:
    dbutils.fs.mount(source=url, mount_point=mount_folder, extra_configs={config: key})
英文:

You can try to mount storage using account key

I did similar thing in my project as well!!

You can use below code as a reference

    Container_name = &quot;root&quot;
    storage_account = &quot;lake&quot;
	key = &quot;&quot;

	url = &quot;wasbs://&quot; + container_name + &quot;@&quot; + storage_account + &quot;.blob.core.windows.net/&quot;
	config = &quot;fs.azure.account.key.&quot; + storage_account + &quot;.blob.core.windows.net&quot;

	mount_folder = &quot;/mnt/lake&quot;
	mounted_list = dbutils.fs.mounts()

    mounted_exist = False
	for item in mounted_list:
		if mount_folder in item[0]:
			mounted_exist = True
			break

	if not mounted_exist:
		dbutils.fs.mount(source = url, mount_point = mount_folder, extra_configs = {config : key})

答案2

得分: 0

你使用存储帐户密钥来访问adls2中的数据。
下面是代码示例:

spark.conf.set("fs.azure.account.key.<存储帐户名称>.dfs.core.windows.net","你的存储帐户密钥")
dbutils.fs.ls("abfss://<容器名称>@<存储帐户名称>.dfs.core.windows.net/<目录>")

输出:

How to Authenitcate Access to ADLS from Databricks without creating Service Principle with Using AD App Registrations to Mount a Drive

或者你也可以按照以下方法操作:

dbutils.fs.mount(source = "wasbs://data@jgsadls.blob.core.windows.net",
		    mount_point = "/mnt/iotdata",
		    extra_configs = {"fs.azure.account.key.jgsadls.blob.core.windows.net":"你的帐户密钥"})
dbutils.fs.ls("/mnt/iotdata/json_data/")

这是结果:

How to Authenitcate Access to ADLS from Databricks without creating Service Principle with Using AD App Registrations to Mount a Drive

英文:

You use storage account key to access data in adls2.
Below is code for it.

    spark.conf.set(&quot;fs.azure.account.key.&lt;storage account name&gt;.dfs.core.windows.net&quot;,&quot;Your stoarge account key&quot;)
    dbutils.fs.ls(&quot;abfss://&lt;container name&gt;@&lt;storage account name&gt;.dfs.core.windows.net/&lt;dir&gt;&quot;)

Output:

How to Authenitcate Access to ADLS from Databricks without creating Service Principle with Using AD App Registrations to Mount a Drive

Or you can follow below approach.

    dbutils.fs.mount(source = &quot;wasbs://data@jgsadls.blob.core.windows.net&quot;,
		    mount_point = &quot;/mnt/iotdata&quot;,
		    extra_configs = {&quot;fs.azure.account.key.jgsadls.blob.core.windows.net&quot;:&quot;Your account key&quot;})
	dbutils.fs.ls(&quot;/mnt/iotdata/json_data/&quot;)

This is the result.

How to Authenitcate Access to ADLS from Databricks without creating Service Principle with Using AD App Registrations to Mount a Drive

huangapple
  • 本文由 发表于 2023年5月22日 18:28:34
  • 转载请务必保留本文链接:https://go.coder-hub.com/76305251.html
匿名

发表评论

匿名网友

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen:

确定