Databricks: Using Volumes in Databricks with Unity Catalog

by

🔹 1. Introduction

In Databricks, we usually store tabular data in Delta tables (structured data).
But what about:

  • Unstructured (images, logs, videos)
  • Semi-structured (JSON, CSV, XML)
  • Other structured files (Parquet, ORC)

👉 For these, Databricks introduces Volumes, which provide a governed, secure storage layer managed by Unity Catalog.

Key Requirements

  • Unity Catalog enabled in your Databricks workspace.
  • Databricks Runtime 13.3 LTS or above.

🔹 2. What are Volumes?

  • Volumes are part of the Unity Catalog hierarchy: Metastore → Catalog → Schema → Volume
  • Just like tables, Volumes store files but are designed for file-based data.
  • Volumes are governed by Unity Catalog policies (ACLs, permissions).

🔹 3. Types of Volumes

Just like tables, Volumes come in two flavors:

  1. Managed Volume
    • Data location managed by Unity Catalog.
    • Files are stored in the default managed storage.
    • If you drop the volume → both data + metadata are deleted.
  2. External Volume
    • Points to an external location (e.g., Azure Data Lake, S3, GCS).
    • Requires external location + storage credential.
    • If you drop the volume → only metadata is deleted, files remain.

🔹 4. Create External Location (for External Volume)

Before creating an External Volume, you must configure an External Location.

Step 1: Create a folder in Azure Storage

  • Storage Account: adbewithdata01
  • Container: data
  • Folder: adb/ext_volume

Step 2: Create External Location in Databricks (UI or SQL)

Using UI:

  • Go to Catalog Explorer > External Locations > Create
  • Example:
    • Name → ext_volume
    • Credential → sc_catalog_storage
    • Path → abfss://data@adbwithdata01.dfs.core.windows.net/adb/ext_volume
    • Test connection → ✅ Success

Using SQL:

CREATE EXTERNAL LOCATION ext_volume
URL 'abfss://data@adbwithdata01.dfs.core.windows.net/adb/ext_volume'
WITH STORAGE CREDENTIAL sc_catalog_storage
COMMENT 'This is for external volume';

🔹 5. Create a Managed Volume

Let’s create a managed volume in the dev.bronze schema.

CREATE VOLUME dev.bronze.managed_volume
COMMENT 'This is a managed volume';

📌 Key point:

  • No LOCATION specified → Unity Catalog decides storage path.
  • Data stored under metastore-managed location.

Check volume details:

DESCRIBE VOLUME dev.bronze.managed_volume;

Output shows:

  • Location (metastore path)
  • Type = MANAGED

🔹 6. Using Volumes with File Paths

When accessing volumes with dbutils.fs or %sh, you must use a special path format:

/Volumes/<catalog>/<schema>/<volume>/<subfolder>/<file>

Example:

/Volumes/dev/bronze/managed_volume/files/emp.csv

🔹 7. Example: Copy Files into Managed Volume

Step 1: Download a CSV

%sh
wget https://raw.githubusercontent.com/databricks/Spar02Hero-Datasets/main/emp.csv
ls -ltr
pwd

Assume file is saved at /databricks/driver/emp.csv.

Step 2: Create a folder inside Volume

dbutils.fs.mkdirs("/Volumes/dev/bronze/managed_volume/files")

Step 3: Copy file into Volume

dbutils.fs.cp("file:/databricks/driver/emp.csv",
              "/Volumes/dev/bronze/managed_volume/files/emp.csv")

Step 4: Query file directly

SELECT * 
FROM csv.`/Volumes/dev/bronze/managed_volume/files/emp.csv`;

✅ You can now read structured data (CSV, JSON, Parquet) stored in your volume.

🔹 8. Create an External Volume

Now let’s create an external volume that points to the external location we created earlier.

CREATE EXTERNAL VOLUME dev.bronze.external_volume
LOCATION 'abfss://data@adbwithdata01.dfs.core.windows.net/adb/ext_volume'
COMMENT 'External volume for semi/unstructured data';

Check details:

DESCRIBE VOLUME dev.bronze.external_volume;
  • Type = EXTERNAL
  • Location = Azure path provided

Step 1: Create a folder inside external volume

dbutils.fs.mkdirs("/Volumes/dev/bronze/external_volume/files")

Step 2: Copy file into external volume

dbutils.fs.cp("file:/databricks/driver/emp.csv",
              "/Volumes/dev/bronze/external_volume/files/emp.csv")

Step 3: Verify in Azure Portal

  • Navigate to adb/ext_volume/files/emp.csv
  • File is now available outside Databricks too.

🔹 9. Drop a Volume

  • Managed Volume → drops data + metadata.
  • External Volume → drops only metadata; files remain in storage.

Example:

-- Drop external volume
DROP VOLUME dev.bronze.external_volume;

-- Files still exist in Azure

If you recreate the volume pointing to the same location:

CREATE EXTERNAL VOLUME dev.bronze.external_volume
LOCATION 'abfss://data@adbwithdata01.dfs.core.windows.net/adb/ext_volume';

👉 Files reappear inside Databricks.

🔹 10. Summary

  • Volumes allow Databricks to govern files (structured/unstructured) under Unity Catalog.
  • Managed Volume → fully controlled by Databricks, data removed on drop.
  • External Volume → points to external storage, dropping only removes metadata.
  • File access always via /Volumes/<catalog>/<schema>/<volume>/....
  • You can read, write, copy, and query files in volumes with SQL or dbutils.