[go: up one dir, main page]
Scribd
Upload
118 views13 pages

How To Create Secrets in Databricks? - by Ashish Garg - Medium

Uploaded by

walteravelin
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
118 views13 pages

How To Create Secrets in Databricks? - by Ashish Garg - Medium

Uploaded by

walteravelin
Copyright
© © All Rights Reserved
We take content rights seriously. If you suspect this is your content, claim it here.
Available Formats
Download as PDF, TXT or read online on Scribd
You are on page 1/ 13

How to create secrets in

Databricks?
Ashish Garg · Follow
4 min read · Jan 29, 2023

12

Creating secrets is the common norm in the data


engineering world. It is beneficial to store sensitive
information in the Azure portal/GCP console since
these platforms provide secure storage options.
Additionally, these platforms can be integrated
with other authentication and authorization tools.
Alternatively, storing secrets directly in Databricks
simplifies the deployment process and allows more
fine-grained control over their access. This article
is on the secret creation over databricks.

Motivation for this article


I encountered an error while attempting to create a secret in Databricks
using the dbutils.secrets.put command. In the error message
‘SecretHandler’ object has no attribute put’, there is an error that the put
attribute is missing from the dbutils.secrets module. This error usually
occurs when the put attribute has been removed or moved from the
dbutils module in an update.

Fig. 1: data bricks dbutils library does not work for creating new secrets.

Secret key creation

Step 1: Create data bricks personal access token


To access data bricks from the outside environment it is good practice to
generate the personal access token within the data bricks and use it
outside.

Fig. 2: Databricks Token Generation Workflow

To create the token, you need to go to USER SETTINGS, then generate a


new token by specifying the expiry date. Workflow is depicted in the
diagram above.

Step 2: Call the secret API from Postman


You can call secret API from any REST-based API calling tools like cURL,
Python API programmatically, Postman, and so on. In this article, I have
considered the Postman tool as it’s quite an easy and handy tool.

Create Postman collection

This is the critical step and entire magic can be expected here. Follow the
steps mentioning here:
1. Create the collection, then appropriately rename it.

2. Add the REST based request and name it appropriately.

3. Select the request created in step 2 above. Then pass POST URL, select
Authorization type (bearer token in this case). Finally set key and
value inside the databricks key-vault.

4. On tapping SEND button will send the POST request to databricks and
create the new key-value pair.

The same workflow is mentioned here.


Fig. 3: Postman Work Flow

The above workflow is self-explanatory that provide insight into how to


use Postman for creating new REST-based request (GET/POST/etc).

The below JSON raw format script was used to pass the scope name,
secret key, and a corresponding value for the key.

{
scope="dataengineering-team-databricks-key-vault",
key="connstring_legacysql_uatserver_key",
value="jdbc:sqlserver://uat-ukwest-smi.public.66047f747ce7.database.windows.ne
}

Read created secret key


There is a function provided by Databricks called dbutils.secrets.get that
retrieves secrets stored in the Databricks Secrets API. This function is
designed to be used within Databricks notebooks and will not work
outside of Databricks.

To get the value from the defined secret key use the command mentioned
next.

Search
secret_value Write
= dbutils.secrets.get(scope = "dataengineering-team-databricks-key-va
for ch in secret_value:
print(ch)
To check all the keys stored in the particular scope use the below
command.

dbutils.secrets.list("dataengineering-team-databricks-key-vault")

You can read the secret value by using dbutils.secrets get command. You
need to pass the scope name and key name to get the value of the secret.

dbutils.secrets.get(
"dataengineering-team-databricks-key-vault",
"marketing-bucket-root-url"))

Summary
Keeping secrets confidential and secure is important when working with
secrets, as is following best practices for accessing and storing sensitive
information in a secure manner.

References

Secrets
A secret is a key-value pair that stores secret material, with a key name unique within a
secret scope . Each scope is…
docs.gcp.databricks.com
Databricks CLI setup & documentation
This section lists CLI requirements and describes how to install and configure your
environment to run the CLI. Run…
docs.gcp.databricks.com

Databricks Gcp Azure Pyspark Python

Written by Ashish Garg Follow


119 Followers

Data Science | Data Engineering | Machine Learning | Artificial Intelligence

More from Ashish Garg


Ashish Garg Ashish Garg

Big Data—File storage formats Databricks—How to mount the


(Row/CSV vs Column/Parquet) ADLS Gen2 folder to Azure…
This article will introduces you to the row- Databricks?
Storage is provided by multiple cloud
based and column-based data formats.… service providers (like MS, Amazon,…
Further it shows the difference between the Google, etc.) where we store our data or file
3 minand
two · Jul 2, 2023
readat… 2 minor
(big read · AugMS…
small). 11, 2022

3 3

Ashish Garg Ashish Garg

How to create the view and hive Azure Databricks—trouble with


references for data bricks users? “round” function!
A quite common scenario for the data If you get Invalid argument error while
engineers is to create the views and hive… using ROUND function to round off your…
references in different data bricks float value then you’ve landed at the right
2 min read · Mar
workspaces for29, 2023
their… 2 min read · Jul 27, 2022
place…

12 2
See all from Ashish Garg

Recommended from Medium

Rahul Madhani in Towards Data Engineering Guillermo Musumeci

4 ways to execute Databricks API How to Configure Azure


Blog explains 4 ways to execute Databricks Databricks Unity Catalog with…
API: curl, Python, Postman, and databrick… Terraform —will
In this story, we Part 1 how to configure
learn
api package Azure Databricks Unity Catalog with…
Terraform.
6 min read · Sep 27, 2023 · 10 min read · Oct 1, 2023

71 56 1

Lists
Coding & Development Predictive Modeling w/
11 stories · 502 saves Python
20 stories · 1001 saves

Practical Guides to Machine ChatGPT


Learning 21 stories · 518 saves
10 stories · 1198 saves
Prashanth Kumar Ryan Chynoweth in DBSQL SME Engineering

Azure Databricks: Job Using Databricks Notebooks for


Performance Monitoring,… Production Data Pipelines
Troubleshooting
As a part of Databricksand
Job performance Introduction
Optimization
Monitoring, troubleshooting and…
Optimization we will be looking into various
14 min read · Feb 25, 2024
aspects. 10 min read · Mar 5, 2024

5 127 3

Muqtada Hussain Mohammed Matt Weingarten

Efficient Change Data Capture Databricks Cost-Saving Tips


(CDC) on Databricks Delta Tabl…
with Spark
In today’s data-driven applications,
organizations face a critical challenge:…
ensuring near-real-time data aggregation
5 read · Octfor…
minaccuracy
and 20, 2023 4 min read · Jan 9, 2024

75 43 2
See more recommendations

Help Status About Careers Blog Privacy Terms Text to speech Teams

You might also like