Rajesh Kumar August 22, 2025 0

🔹 1. Introduction

In Databricks, you often need to interact with:

  • File systems
  • Object storage (Azure Data Lake, S3, GCS)
  • Secrets (keys, credentials)
  • Widgets (to parameterize notebooks)
  • Other notebooks

👉 For these tasks, Databricks Utilities (dbutils) provide built-in helpers.

Key points:

  • Available only in Python, R, and Scala notebooks.
  • Not available in SQL notebooks.
  • Used for file management, secrets handling, widgets, and notebook execution.

🔹 2. What is dbutils?

dbutils is a Databricks-provided utility library.
You can see what’s available by running:

dbutils.help()

It lists available submodules, such as:

  • dbutils.fs → File System utilities
  • dbutils.secrets → Secrets handling
  • dbutils.widgets → Create interactive inputs
  • dbutils.notebook → Run/manage notebooks
  • (plus some experimental utilities, avoid in production)

🔹 3. File System Utilities (dbutils.fs)

The most widely used part of dbutils.
Run help:

dbutils.fs.help()

It shows common functions:

  • ls(path) → list contents
  • cp(source, dest, recurse=False) → copy files
  • mv(source, dest, recurse=False) → move files
  • head(path, maxBytes) → preview file contents
  • rm(path, recurse=False) → remove file/folder
  • mkdirs(path) → create directory

Example 1: List files

# List files in DBFS root
dbutils.fs.ls("/")

# List files in local driver storage
dbutils.fs.ls("file:/")

If you wrap with display(), results show in a neat table:

display(dbutils.fs.ls("/"))

Example 2: Explore a Volume

Volumes live under:

/Volumes/<catalog>/<schema>/<volume>/

Example:

dbutils.fs.ls("/Volumes/dev/bronze/managed_volume/")

Output → shows folders like files/emp.csv.

Example 3: Preview a file

dbutils.fs.head("/Volumes/dev/bronze/managed_volume/files/emp.csv")

Shows first lines of CSV.

Example 4: Create folder inside a Volume

dbutils.fs.mkdirs("/Volumes/dev/bronze/managed_volume/input/csv")

Example 5: Copy file

From local driver to Volume:

dbutils.fs.cp("file:/databricks/driver/emp.csv",
              "/Volumes/dev/bronze/managed_volume/input/csv/emp.csv")

From Volume back to local:

dbutils.fs.cp("/Volumes/dev/bronze/managed_volume/input/csv/emp.csv",
              "file:/tmp/emp.csv")

Check in local:

%sh
ls -ltr /tmp/

✅ This way, you can move data between local, DBFS, and cloud storages.

🔹 4. Widgets Utilities (dbutils.widgets)

Widgets let you parameterize notebooks — useful for dynamic queries, workflows, and pipelines.

Check available options:

dbutils.widgets.help()

Common types:

  • text → free text input
  • dropdown → single-select dropdown
  • combobox → dropdown + custom entry
  • multiselect → select multiple values

Example 1: Create a Text Widget

dbutils.widgets.text("input_cust_id", "10000", "Customer ID")

👉 This creates a textbox at the top of the notebook.

  • Name: input_cust_id
  • Default: 10000
  • Label: Customer ID

Get the value:

dbutils.widgets.get("input_cust_id")

If you change the input in the UI, the value updates automatically.

Example 2: Use Widget in SQL

For DBR <= 15.1:

%sql
SELECT ${input_cust_id} AS customer_id

For DBR >= 15.1:

%sql
SELECT :input_cust_id AS customer_id

🔹 5. Secrets Utilities (dbutils.secrets)

Used to retrieve secrets stored in Databricks-backed scopes or Azure Key Vault-backed scopes.

Check options:

dbutils.secrets.help()

Common functions:

  • listScopes() → list available secret scopes
  • list(scope) → list secret keys in scope
  • get(scope, key) → get a secret value

Example:

db_password = dbutils.secrets.get(scope="my_scope", key="db_password")

⚠️ Note: Cannot print() secrets — they are redacted for security.

🔹 6. Notebook Utilities (dbutils.notebook)

Used for running notebooks from other notebooks.

Check:

dbutils.notebook.help()

Example 1: Run another notebook

result = dbutils.notebook.run("/Repos/my_project/process_data", 60, {"input": "emp.csv"})
  • Runs /Repos/my_project/process_data
  • Timeout: 60 seconds
  • Passes input parameter

Example 2: Exit with value

Inside the child notebook:

dbutils.notebook.exit("SUCCESS")

Parent notebook can capture "SUCCESS".

🔹 7. Summary

  • dbutils = Databricks helper utilities, available in Python/R/Scala notebooks.
  • dbutils.fs → Manage files across DBFS, Volumes, cloud storages, and local.
  • dbutils.widgets → Create interactive parameters for notebooks.
  • dbutils.secrets → Securely retrieve secrets from Databricks or Key Vault.
  • dbutils.notebook → Run/manage notebooks programmatically.
  • Always use dbutils.help() and dbutils.<submodule>.help() to explore options.

✅ With this knowledge, you can now manage files, parameterize workflows, retrieve secrets, and orchestrate notebooks directly inside Databricks.

Category: