如何在笔记本以外的pyspark作业中使用dbutils命令 [英] How to use dbutils command in pyspark job other than NoteBook

问题描述

我想使用dbutils命令来访问pyspark作业中的秘密，该作业是通过Databricks上Jobs中的Spark-Submit提交的.使用dbutils命令时，它会给出未定义的错误dbutils.

除了笔记本以外，是否有办法在pyspark作业中使用dbutils?

尝试了以下解决方案:

1)根据此解决方案导入DBUtils.但这不是Databricks dbutils.

2)根据此解决方案.但这也不起作用.

pyspark作业，如下图所示: 图像

解决方案

我对您的问题的理解-如何从Databricks笔记本中的密钥库访问秘密值?
因此，要访问密钥，您必须将Databricks笔记本链接到密钥库. 必须使用Scopes进行此链接.
在下面的网址中，您必须提供详细信息:

https://westeurope.azuredatabricks.net/#secrets/createScope

如果打开链接，那么您将进入一个秘密范围创建页面: 链接可能会有所不同，具体取决于订阅的区域/区域.在我的情况下，它是"westeurope".

您必须在页面上提供3个值:
1)范围名称
2) DNS名称
3)资源ID

作用域名称必须手动定义，其余两个值必须从密钥库中复制.
现在转到您的Azure密钥库，然后单击属性部分.

复制 DNS名称和资源ID 值，并将其粘贴到作用域创建"页面的必需单元格中.

就是这样.现在您的笔记本将指向您的密钥库.

我为我的示波器命名为 az-kv-scope-test .

让我们假设我的密钥库有一个名为 db_password 的秘密，我想在我的笔记本中访问它.

因此，要从Databricks笔记本中的密钥库访问机密值，我必须使用以下命令.

    // variable declaration and initialization.    
    val my_kv_scope = "az-kv-scope-test"    
    val password = dbutils.secrets.get(scope = my_kv_scope, key = "db_password")

以下链接将指导您获取有关秘密范围的更多信息.
https://docs.databricks.com/security/secrets/index.html

我希望这可以使您对笔记本中的Key-vault值可访问性有所了解.

I want to use dbutils command for accessing secrets in my pyspark job submitting through Spark-Submit inside Jobs on Databricks. While using dbutils command it is giving error dbutils not defined.

Is there is the way to use dbutils in a pyspark job other than a notebook?

Tried the following solutions:

1) import DBUtils, according to this solution. But this is not Databricks dbutils.

2) import pyspark.dbutils import DBUtils , according to this solution. But this also didn't work.

pyspark job as shown in below diagram: image

解决方案

What I understand by your question- How do I access the secret value from the key-vault in Databricks notebook?
So to access the keys you have to link your Databricks notebook to the key-vault. This linking has to be done by using the Scopes.
In the below url you have to provide the details:

https://westeurope.azuredatabricks.net/#secrets/createScope

If you open the link then you will land to a secret scope creation page: Link may be vary depends on the area/region of subscription. In my case it is "westeurope".

You have to provide 3 values on the page:
1) Scope Name
2) DNS Name
3) Resource ID

Scope name have to be define manually and rest of the two values has to copy from the key-vault.
Now go to your Azure key-vault and click on properties section.

Copy the DNS Name and Resource ID values and paste it into required cells of the Scope creation page.

That's it. Now your notebook will point to your key vault.

I have given az-kv-scope-test name to my scope.

Lets assume my key-vault having a secret with the name db_password and I want to access it in my notebook.

So to access the secret value from the key vault inside the Databricks notebook I have to use the below command.

    // variable declaration and initialization.    
    val my_kv_scope = "az-kv-scope-test"    
    val password = dbutils.secrets.get(scope = my_kv_scope, key = "db_password")

Following link will guide you to get more information on secrete scopes.
https://docs.databricks.com/security/secrets/index.html

I hope this will give you some clarity on the Key-vault values accessibility in notebook.

这篇关于如何在笔记本以外的pyspark作业中使用dbutils命令的文章就介绍到这了，希望我们推荐的答案对大家有所帮助，也希望大家多多支持IT屋！