---

---

🔹 1. Introduction

In Databricks, you often need to interact with:

• File systems
• Object storage (Azure Data Lake, S3, GCS)
• Secrets (keys, credentials)
• Widgets (to parameterize notebooks)
• Other notebooks

👉 For these tasks, Databricks Utilities (dbutils) provide built-in helpers.

Key points:

• Available only in Python, R, and Scala notebooks.
• Not available in SQL notebooks.
• Used for file management, secrets handling, widgets, and notebook execution.

---

🔹 2. What is dbutils?

dbutils is a Databricks-provided utility library.
You can see what’s available by running:

dbutils.help()

It lists available submodules, such as:

• dbutils.fs → File System utilities
• dbutils.secrets → Secrets handling
• dbutils.widgets → Create interactive inputs
• dbutils.notebook → Run/manage notebooks
• (plus some experimental utilities, avoid in production)

---

🔹 3. File System Utilities (dbutils.fs)

The most widely used part of dbutils.
Run help:

dbutils.fs.help()

It shows common functions:

• ls(path) → list contents
• cp(source, dest, recurse=False) → copy files
• mv(source, dest, recurse=False) → move files
• head(path, maxBytes) → preview file contents
• rm(path, recurse=False) → remove file/folder
• mkdirs(path) → create directory

---

Example 1: List files

# List files in DBFS root
dbutils.fs.ls("/")

# List files in local driver storage
dbutils.fs.ls("file:/")

If you wrap with display(), results show in a neat table:

display(dbutils.fs.ls("/"))

---

Example 2: Explore a Volume

Volumes live under:

/Volumes/<catalog>/<schema>/<volume>/

Example:

dbutils.fs.ls("/Volumes/dev/bronze/managed_volume/")

Output → shows folders like files/emp.csv.

---

Example 3: Preview a file

dbutils.fs.head("/Volumes/dev/bronze/managed_volume/files/emp.csv")

Shows first lines of CSV.

---

Example 4: Create folder inside a Volume

dbutils.fs.mkdirs("/Volumes/dev/bronze/managed_volume/input/csv")

---

Example 5: Copy file

From local driver to Volume:

dbutils.fs.cp("file:/databricks/driver/emp.csv",
              "/Volumes/dev/bronze/managed_volume/input/csv/emp.csv")

From Volume back to local:

dbutils.fs.cp("/Volumes/dev/bronze/managed_volume/input/csv/emp.csv",
              "file:/tmp/emp.csv")

Check in local:

%sh
ls -ltr /tmp/

✅ This way, you can move data between local, DBFS, and cloud storages.

---

🔹 4. Widgets Utilities (dbutils.widgets)

Widgets let you parameterize notebooks — useful for dynamic queries, workflows, and pipelines.

Check available options:

dbutils.widgets.help()

Common types:

• text → free text input
• dropdown → single-select dropdown
• combobox → dropdown + custom entry
• multiselect → select multiple values

---

Example 1: Create a Text Widget

dbutils.widgets.text("input_cust_id", "10000", "Customer ID")

👉 This creates a textbox at the top of the notebook.

• Name: input_cust_id
• Default: 10000
• Label: Customer ID

Get the value:

dbutils.widgets.get("input_cust_id")

If you change the input in the UI, the value updates automatically.

---

Example 2: Use Widget in SQL

For DBR <= 15.1:

%sql
SELECT ${input_cust_id} AS customer_id

For DBR >= 15.1:

%sql
SELECT :input_cust_id AS customer_id

---

🔹 5. Secrets Utilities (dbutils.secrets)

Used to retrieve secrets stored in Databricks-backed scopes or Azure Key Vault-backed scopes.

Check options:

dbutils.secrets.help()

Common functions:

• listScopes() → list available secret scopes
• list(scope) → list secret keys in scope
• get(scope, key) → get a secret value

Example:

db_password = dbutils.secrets.get(scope="my_scope", key="db_password")

⚠️ Note: Cannot print() secrets — they are redacted for security.

---

🔹 6. Notebook Utilities (dbutils.notebook)

Used for running notebooks from other notebooks.

Check:

dbutils.notebook.help()

Example 1: Run another notebook

result = dbutils.notebook.run("/Repos/my_project/process_data", 60, {"input": "emp.csv"})

• Runs /Repos/my_project/process_data
• Timeout: 60 seconds
• Passes input parameter

Example 2: Exit with value

Inside the child notebook:

dbutils.notebook.exit("SUCCESS")

Parent notebook can capture "SUCCESS".

---

🔹 7. Summary

• dbutils = Databricks helper utilities, available in Python/R/Scala notebooks.
• dbutils.fs → Manage files across DBFS, Volumes, cloud storages, and local.
• dbutils.widgets → Create interactive parameters for notebooks.
• dbutils.secrets → Securely retrieve secrets from Databricks or Key Vault.
• dbutils.notebook → Run/manage notebooks programmatically.
• Always use dbutils.help() and dbutils.<submodule>.help() to explore options.

---

✅ With this knowledge, you can now manage files, parameterize workflows, retrieve secrets, and orchestrate notebooks directly inside Databricks.