{"id":789,"date":"2025-08-22T14:03:18","date_gmt":"2025-08-22T14:03:18","guid":{"rendered":"https:\/\/dataopsschool.com\/blog\/?p=789"},"modified":"2025-08-22T14:03:19","modified_gmt":"2025-08-22T14:03:19","slug":"databricks-databricks-utilities-dbutils-complete-guide","status":"publish","type":"post","link":"https:\/\/dataopsschool.com\/blog\/databricks-databricks-utilities-dbutils-complete-guide\/","title":{"rendered":"Databricks: Databricks Utilities (dbutils) \u2013 Complete Guide"},"content":{"rendered":"\n<p><\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h1 class=\"wp-block-heading\"><\/h1>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">\ud83d\udd39 1. Introduction<\/h2>\n\n\n\n<p>In Databricks, you often need to interact with:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>File systems<\/li>\n\n\n\n<li>Object storage (Azure Data Lake, S3, GCS)<\/li>\n\n\n\n<li>Secrets (keys, credentials)<\/li>\n\n\n\n<li>Widgets (to parameterize notebooks)<\/li>\n\n\n\n<li>Other notebooks<\/li>\n<\/ul>\n\n\n\n<p>\ud83d\udc49 For these tasks, <strong>Databricks Utilities (dbutils)<\/strong> provide built-in helpers.<\/p>\n\n\n\n<p><strong>Key points:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Available only in <strong>Python, R, and Scala notebooks<\/strong>.<\/li>\n\n\n\n<li>Not available in SQL notebooks.<\/li>\n\n\n\n<li>Used for file management, secrets handling, widgets, and notebook execution.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">\ud83d\udd39 2. What is dbutils?<\/h2>\n\n\n\n<p><code>dbutils<\/code> is a Databricks-provided utility library.<br>You can see what\u2019s available by running:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>dbutils.help()\n<\/code><\/pre>\n\n\n\n<p>It lists available submodules, such as:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>dbutils.fs<\/code> \u2192 File System utilities<\/li>\n\n\n\n<li><code>dbutils.secrets<\/code> \u2192 Secrets handling<\/li>\n\n\n\n<li><code>dbutils.widgets<\/code> \u2192 Create interactive inputs<\/li>\n\n\n\n<li><code>dbutils.notebook<\/code> \u2192 Run\/manage notebooks<\/li>\n\n\n\n<li>(plus some experimental utilities, avoid in production)<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">\ud83d\udd39 3. File System Utilities (dbutils.fs)<\/h2>\n\n\n\n<p>The <strong>most widely used<\/strong> part of dbutils.<br>Run help:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>dbutils.fs.help()\n<\/code><\/pre>\n\n\n\n<p>It shows common functions:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>ls(path)<\/code> \u2192 list contents<\/li>\n\n\n\n<li><code>cp(source, dest, recurse=False)<\/code> \u2192 copy files<\/li>\n\n\n\n<li><code>mv(source, dest, recurse=False)<\/code> \u2192 move files<\/li>\n\n\n\n<li><code>head(path, maxBytes)<\/code> \u2192 preview file contents<\/li>\n\n\n\n<li><code>rm(path, recurse=False)<\/code> \u2192 remove file\/folder<\/li>\n\n\n\n<li><code>mkdirs(path)<\/code> \u2192 create directory<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">Example 1: List files<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code># List files in DBFS root\ndbutils.fs.ls(\"\/\")\n\n# List files in local driver storage\ndbutils.fs.ls(\"file:\/\")\n<\/code><\/pre>\n\n\n\n<p>If you wrap with <code>display()<\/code>, results show in a neat table:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>display(dbutils.fs.ls(\"\/\"))\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">Example 2: Explore a Volume<\/h3>\n\n\n\n<p>Volumes live under:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>\/Volumes\/&lt;catalog&gt;\/&lt;schema&gt;\/&lt;volume&gt;\/\n<\/code><\/pre>\n\n\n\n<p>Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>dbutils.fs.ls(\"\/Volumes\/dev\/bronze\/managed_volume\/\")\n<\/code><\/pre>\n\n\n\n<p>Output \u2192 shows folders like <code>files\/emp.csv<\/code>.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">Example 3: Preview a file<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>dbutils.fs.head(\"\/Volumes\/dev\/bronze\/managed_volume\/files\/emp.csv\")\n<\/code><\/pre>\n\n\n\n<p>Shows first lines of CSV.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">Example 4: Create folder inside a Volume<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>dbutils.fs.mkdirs(\"\/Volumes\/dev\/bronze\/managed_volume\/input\/csv\")\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">Example 5: Copy file<\/h3>\n\n\n\n<p>From local driver to Volume:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>dbutils.fs.cp(\"file:\/databricks\/driver\/emp.csv\",\n              \"\/Volumes\/dev\/bronze\/managed_volume\/input\/csv\/emp.csv\")\n<\/code><\/pre>\n\n\n\n<p>From Volume back to local:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>dbutils.fs.cp(\"\/Volumes\/dev\/bronze\/managed_volume\/input\/csv\/emp.csv\",\n              \"file:\/tmp\/emp.csv\")\n<\/code><\/pre>\n\n\n\n<p>Check in local:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>%sh\nls -ltr \/tmp\/\n<\/code><\/pre>\n\n\n\n<p>\u2705 This way, you can move data between <strong>local, DBFS, and cloud storages<\/strong>.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">\ud83d\udd39 4. Widgets Utilities (dbutils.widgets)<\/h2>\n\n\n\n<p>Widgets let you <strong>parameterize notebooks<\/strong> \u2014 useful for dynamic queries, workflows, and pipelines.<\/p>\n\n\n\n<p>Check available options:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>dbutils.widgets.help()\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Common types:<\/h3>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>text<\/code> \u2192 free text input<\/li>\n\n\n\n<li><code>dropdown<\/code> \u2192 single-select dropdown<\/li>\n\n\n\n<li><code>combobox<\/code> \u2192 dropdown + custom entry<\/li>\n\n\n\n<li><code>multiselect<\/code> \u2192 select multiple values<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">Example 1: Create a Text Widget<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>dbutils.widgets.text(\"input_cust_id\", \"10000\", \"Customer ID\")\n<\/code><\/pre>\n\n\n\n<p>\ud83d\udc49 This creates a textbox at the top of the notebook.<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Name: <code>input_cust_id<\/code><\/li>\n\n\n\n<li>Default: <code>10000<\/code><\/li>\n\n\n\n<li>Label: <code>Customer ID<\/code><\/li>\n<\/ul>\n\n\n\n<p>Get the value:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>dbutils.widgets.get(\"input_cust_id\")\n<\/code><\/pre>\n\n\n\n<p>If you change the input in the UI, the value updates automatically.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h3 class=\"wp-block-heading\">Example 2: Use Widget in SQL<\/h3>\n\n\n\n<p>For DBR <strong>&lt;= 15.1<\/strong>:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>%sql\nSELECT ${input_cust_id} AS customer_id\n<\/code><\/pre>\n\n\n\n<p>For DBR <strong>&gt;= 15.1<\/strong>:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>%sql\nSELECT :input_cust_id AS customer_id\n<\/code><\/pre>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">\ud83d\udd39 5. Secrets Utilities (dbutils.secrets)<\/h2>\n\n\n\n<p>Used to retrieve secrets stored in <strong>Databricks-backed scopes<\/strong> or <strong>Azure Key Vault-backed scopes<\/strong>.<\/p>\n\n\n\n<p>Check options:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>dbutils.secrets.help()\n<\/code><\/pre>\n\n\n\n<p>Common functions:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li><code>listScopes()<\/code> \u2192 list available secret scopes<\/li>\n\n\n\n<li><code>list(scope)<\/code> \u2192 list secret keys in scope<\/li>\n\n\n\n<li><code>get(scope, key)<\/code> \u2192 get a secret value<\/li>\n<\/ul>\n\n\n\n<p>Example:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>db_password = dbutils.secrets.get(scope=\"my_scope\", key=\"db_password\")\n<\/code><\/pre>\n\n\n\n<p>\u26a0\ufe0f Note: Cannot <code>print()<\/code> secrets \u2014 they are redacted for security.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">\ud83d\udd39 6. Notebook Utilities (dbutils.notebook)<\/h2>\n\n\n\n<p>Used for running notebooks from other notebooks.<\/p>\n\n\n\n<p>Check:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>dbutils.notebook.help()\n<\/code><\/pre>\n\n\n\n<h3 class=\"wp-block-heading\">Example 1: Run another notebook<\/h3>\n\n\n\n<pre class=\"wp-block-code\"><code>result = dbutils.notebook.run(\"\/Repos\/my_project\/process_data\", 60, {\"input\": \"emp.csv\"})\n<\/code><\/pre>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Runs <code>\/Repos\/my_project\/process_data<\/code><\/li>\n\n\n\n<li>Timeout: 60 seconds<\/li>\n\n\n\n<li>Passes input parameter<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Example 2: Exit with value<\/h3>\n\n\n\n<p>Inside the child notebook:<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>dbutils.notebook.exit(\"SUCCESS\")\n<\/code><\/pre>\n\n\n\n<p>Parent notebook can capture <code>\"SUCCESS\"<\/code>.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\">\ud83d\udd39 7. Summary<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>dbutils<\/strong> = Databricks helper utilities, available in Python\/R\/Scala notebooks.<\/li>\n\n\n\n<li><strong>dbutils.fs<\/strong> \u2192 Manage files across DBFS, Volumes, cloud storages, and local.<\/li>\n\n\n\n<li><strong>dbutils.widgets<\/strong> \u2192 Create interactive parameters for notebooks.<\/li>\n\n\n\n<li><strong>dbutils.secrets<\/strong> \u2192 Securely retrieve secrets from Databricks or Key Vault.<\/li>\n\n\n\n<li><strong>dbutils.notebook<\/strong> \u2192 Run\/manage notebooks programmatically.<\/li>\n\n\n\n<li>Always use <code>dbutils.help()<\/code> and <code>dbutils.&lt;submodule>.help()<\/code> to explore options.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p>\u2705 With this knowledge, you can now <strong>manage files, parameterize workflows, retrieve secrets, and orchestrate notebooks<\/strong> directly inside Databricks.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>\ud83d\udd39 1. Introduction In Databricks, you often need to interact with: \ud83d\udc49 For these tasks, Databricks Utilities (dbutils) provide built-in helpers. Key points: \ud83d\udd39 2. What is&#8230; <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-789","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/789","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=789"}],"version-history":[{"count":1,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/789\/revisions"}],"predecessor-version":[{"id":790,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/789\/revisions\/790"}],"wp:attachment":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=789"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=789"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=789"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}