{"id":427,"date":"2025-08-11T15:16:50","date_gmt":"2025-08-11T15:16:50","guid":{"rendered":"https:\/\/dataopsschool.com\/blog\/?p=427"},"modified":"2025-08-11T15:24:57","modified_gmt":"2025-08-11T15:24:57","slug":"databricks-set-up-metastore-map-azure-storage-account-with-access-connector-enable-unity-catalog","status":"publish","type":"post","link":"https:\/\/dataopsschool.com\/blog\/databricks-set-up-metastore-map-azure-storage-account-with-access-connector-enable-unity-catalog\/","title":{"rendered":"Databricks: Set Up Metastore &amp; Map Azure Storage Account with Access Connector, Enable Unity Catalog"},"content":{"rendered":"\n<p><\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"step-by-step-guide-set-up-metastore--map-azure-sto\"><\/h2>\n\n\n\n<p>This guide walks you through setting up a Unity Catalog metastore in Azure Databricks, connecting it securely to an Azure storage account using the Access Connector, validating the setup, and enabling Unity Catalog for your Databricks workspace.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Step 1: Create a Storage Account and Container for Metastore<\/strong><\/h2>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Navigate to Azure Portal<\/strong> and create an Azure Data Lake Storage Gen2 account (must be in the same region as your Databricks workspace).<\/li>\n\n\n\n<li><strong>Add a container<\/strong> for metastore-level storage. For example: <code>mycontainer<\/code> in <code>mydatalakestorage<\/code>.<\/li>\n\n\n\n<li><strong>Note down the ADLS Gen2 URI:<\/strong> text<code>abfss:\/\/mycontainer@mydatalakestorage.dfs.core.windows.net\/unity-metastore<\/code> This will be the root path for managed tables and volumes.<\/li>\n<\/ol>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Step 2: Create Access Connector (Managed Identity) for Azure Databricks<\/strong><\/h2>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>In Azure Portal:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Click \u201cCreate a resource\u201d \u2192 Search for \u201cAccess Connector for Azure Databricks\u201d.<\/li>\n\n\n\n<li>Click <strong>Create<\/strong>.<\/li>\n\n\n\n<li>Choose <strong>Subscription<\/strong>, <strong>Resource Group<\/strong>, <strong>Region<\/strong>, and enter a connector name (e.g., <code>unity-access-connector<\/code>).<\/li>\n\n\n\n<li>On the Managed Identity tab, select <strong>System-assigned managed identity<\/strong> (recommended).<\/li>\n\n\n\n<li>Click <strong>Review + Create<\/strong>.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Grant Storage Access:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Assign the managed identity <strong>Storage Blob Data Contributor<\/strong> or higher role on the storage account or precise container.<\/li>\n\n\n\n<li>This enables Databricks to write\/read data.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Note the resource ID:<\/strong> text<code>\/subscriptions\/&lt;sub_id&gt;\/resourceGroups\/&lt;rg&gt;\/providers\/Microsoft.Databricks\/accessConnectors\/&lt;name&gt;<\/code><\/li>\n<\/ol>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"431\" src=\"https:\/\/dataopsschool.com\/blog\/wp-content\/uploads\/2025\/08\/image-4-1024x431.png\" alt=\"\" class=\"wp-image-429\" srcset=\"https:\/\/dataopsschool.com\/blog\/wp-content\/uploads\/2025\/08\/image-4-1024x431.png 1024w, https:\/\/dataopsschool.com\/blog\/wp-content\/uploads\/2025\/08\/image-4-300x126.png 300w, https:\/\/dataopsschool.com\/blog\/wp-content\/uploads\/2025\/08\/image-4-768x324.png 768w, https:\/\/dataopsschool.com\/blog\/wp-content\/uploads\/2025\/08\/image-4.png 1206w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"436\" src=\"https:\/\/dataopsschool.com\/blog\/wp-content\/uploads\/2025\/08\/image-5-1024x436.png\" alt=\"\" class=\"wp-image-431\" srcset=\"https:\/\/dataopsschool.com\/blog\/wp-content\/uploads\/2025\/08\/image-5-1024x436.png 1024w, https:\/\/dataopsschool.com\/blog\/wp-content\/uploads\/2025\/08\/image-5-300x128.png 300w, https:\/\/dataopsschool.com\/blog\/wp-content\/uploads\/2025\/08\/image-5-768x327.png 768w, https:\/\/dataopsschool.com\/blog\/wp-content\/uploads\/2025\/08\/image-5.png 1198w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Step 3: Create the Metastore in Databricks and Link Storage<\/strong><\/h2>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Log in to the Databricks Account Console<\/strong> (), as Account Admin.<\/li>\n\n\n\n<li><strong>Go to Catalog<\/strong>.<\/li>\n\n\n\n<li><strong>Click \u201cCreate Metastore.\u201d<\/strong><\/li>\n\n\n\n<li><strong>Fill the form:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Name for the metastore<\/li>\n\n\n\n<li>Region (match storage, workspace)<\/li>\n\n\n\n<li><strong>ADLS Gen2 path:<\/strong> <code>mycontainer@mydatalakestorage.dfs.core.windows.net\/unity-metastore<\/code><\/li>\n\n\n\n<li><strong>Access Connector ID:<\/strong> Paste the resource ID from previous step<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Click \u201cCreate.\u201d<\/strong><\/li>\n<\/ol>\n\n\n\n<figure class=\"wp-block-image size-large\"><img loading=\"lazy\" decoding=\"async\" width=\"1024\" height=\"546\" src=\"https:\/\/dataopsschool.com\/blog\/wp-content\/uploads\/2025\/08\/image-6-1024x546.png\" alt=\"\" class=\"wp-image-433\" srcset=\"https:\/\/dataopsschool.com\/blog\/wp-content\/uploads\/2025\/08\/image-6-1024x546.png 1024w, https:\/\/dataopsschool.com\/blog\/wp-content\/uploads\/2025\/08\/image-6-300x160.png 300w, https:\/\/dataopsschool.com\/blog\/wp-content\/uploads\/2025\/08\/image-6-768x409.png 768w, https:\/\/dataopsschool.com\/blog\/wp-content\/uploads\/2025\/08\/image-6.png 1081w\" sizes=\"auto, (max-width: 1024px) 100vw, 1024px\" \/><\/figure>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Step 4: Link Workspaces to the Metastore &amp; Enable Unity Catalog<\/strong><\/h2>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>After metastore creation<\/strong>, select workspaces to assign to the metastore.\n<ul class=\"wp-block-list\">\n<li>Alternatively, return to Account Console \u2192 Catalog \u2192 Metastore \u2192 Workspaces tab \u2192 \u201cAssign to workspace\u201d.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Confirm assignment:<\/strong> The workspace is Unity Catalog enabled.<\/li>\n<\/ol>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Step 5: Validation Steps<\/strong><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Workspace Validation:<\/strong>\n<ul class=\"wp-block-list\">\n<li>In Databricks Workspace UI, navigate to <strong>Data<\/strong>. Confirm you see Unity Catalog concepts (Catalogs, Schemas).<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Storage Validation:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Access data via Unity Catalog and verify files\/folders are created in the designated ADLS Gen2 storage account\/container.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Security Validation:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Ensure data access controls and audits appear in the Catalog Explorer.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Metastore Validation:<\/strong>\n<ul class=\"wp-block-list\">\n<li>Run a simple create table command in Databricks and check physical storage and permission enforcement.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Step 6: Setup Unity Catalog Objects<\/strong><\/h2>\n\n\n\n<ol class=\"wp-block-list\">\n<li><strong>Create Catalogs, Schemas, and Tables<\/strong> using SQL in the workspace: sql<code>CREATE CATALOG my_catalog; CREATE SCHEMA my_catalog.my_schema; CREATE TABLE my_catalog.my_schema.my_table (id INT, name STRING);<\/code>\n<ul class=\"wp-block-list\">\n<li>Data will be stored in the mapped container, managed by Unity Catalog.<\/li>\n<\/ul>\n<\/li>\n\n\n\n<li><strong>Create External Locations if needed:<\/strong>\n<ul class=\"wp-block-list\">\n<li>For additional storage, register external locations for external tables.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\"><strong>Key Notes &amp; Troubleshooting<\/strong><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Global Admin permissions<\/strong> may be required for setup; SCIM connector recommended for user\/group sync.<\/li>\n\n\n\n<li>You must co-locate storage, connector, metastore, and workspaces in the same region.<\/li>\n\n\n\n<li>Managed identities via Access Connector are preferred over service principals for security and simplicity.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p>By following these steps, you will secure, govern, and validate storage and workspace integration with Unity Catalog in Azure Databricks, ready for enterprise-scale Lakehouse governance.<\/p>\n\n\n\n<p><\/p>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>This guide walks you through setting up a Unity Catalog metastore in Azure Databricks, connecting it securely to an Azure storage account using the Access Connector, validating&#8230; <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-427","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/427","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=427"}],"version-history":[{"count":3,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/427\/revisions"}],"predecessor-version":[{"id":434,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/427\/revisions\/434"}],"wp:attachment":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=427"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=427"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=427"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}