In this lesson, we’ll explore how to securely manage and use sensitive information, such as application IDs, tenant IDs, and secret keys, in your Databricks notebooks. We’ll cover the `DBUtils.secrets.get` method and the use of secret scopes with Azure Key Vault to keep your secrets safe and hidden.

1. Introduction to Secret Scopes

When working with sensitive information in Databricks, it’s crucial to ensure that values like application IDs and secret keys are not directly exposed in your notebooks. Storing these values securely helps prevent unauthorized access and potential security breaches.

2. Using Variables and F-Strings for Secure Access

Let’s start by assigning sensitive information to variables:

application_id = "your_application_id"
tenant_id = "your_tenant_id"
secret = "your_secret_value"
container_name = "your_container_name"
account_name = "your_account_name"
mount_point = "/mnt/your_mount_point"

Instead of hardcoding these values, you can use Python’s F-strings to securely reference variables within strings. For example:

url = f"https://{account_name}.blob.core.windows.net/{container_name}"

#This approach allows you to dynamically construct URLs and other strings without directly displaying sensitive information.

3. Introduction to the `DBUtils.secrets.get` Method

Databricks provides the `DBUtils.secrets.get` method, which retrieves secret values stored in a secure location like Azure Key Vault. This method ensures that your secrets are used securely without exposing them in your code.

4. Creating Secret Scopes in Azure Key Vault

To securely store your secrets, follow these steps:

1. Create an Azure Key Vault:

• In the Azure portal, search for “Key Vault” and create a new Key Vault resource.
• Use the same resource group as your Databricks and storage account instances.
• Give your Key Vault a unique name, such as `databricks-secrets-639`.

• 2. Store Secrets in Key Vault:

  • Once your Key Vault is created, navigate to the “Secrets” section under “Objects.”
  • Add your secrets (e.g., Application ID, Tenant ID, Secret) by clicking the “Generate/Import” button.
  • Provide a name for each secret (e.g., `application-id`, `tenant-id`, `secret`) and paste the corresponding values.

3. Create a Secret Scope in Databricks:

• In Databricks, access the URL of your Databricks instance, then append `/secrets/createScope` to the URL.
• On the secret scope creation page, provide the following information:
• Scope Name: The name of your Azure Key Vault (e.g., `databricks-secrets-639`).
• Managed Principal: Select “All users” so that all users can manage the secret scope.
• DNS Name and Resource ID: These can be found in the Azure Key Vault’s “Properties” section.
• Once created, this secret scope allows you to securely reference your secrets in Databricks.
• 

5. **Using Secret Scopes in Databricks Notebooks

With the secret scope in place, you can now access your secrets securely in your Databricks notebooks:

application_id = dbutils.secrets.get(scope="databricks-secrets-639", key="application-id")
tenant_id = dbutils.secrets.get(scope="databricks-secrets-639", key="tenant-id")
secret = dbutils.secrets.get(scope="databricks-secrets-639", key="secret")

When you run the cell, the values are retrieved securely, and the actual values are not displayed in the notebook, keeping them safe from exposure.

6. Mounting a Storage Container Using Secure Secrets

Now that your secrets are securely stored, you can use them to mount a storage container in Databricks:

mount_point = "/mnt/bronze"
dbutils.fs.mount(
source = f"wasbs://{container_name}@{account_name}.blob.core.windows.net",
mount_point = mount_point,
extra_configs = {
f"fs.azure.account.key.{account_name}.blob.core.windows.net": secret
}
)

After running the command, the storage container is mounted, and the secret values remain secure.

7. Unmounting the Storage Container

If needed, you can unmount the storage container:

dbutils.fs.unmount(mount_point)

This step ensures that the mount point is cleared, and you can confirm it by checking the Databricks File System (DBFS).

Conclusion

By using Azure Key Vault in combination with secret scopes and the `DBUtils.secrets.get` method, you can securely manage and use sensitive information in your Databricks notebooks. This approach helps prevent accidental exposure of credentials and other critical data, enhancing the security of your data workflows.