{"id":736,"date":"2025-08-19T12:53:38","date_gmt":"2025-08-19T12:53:38","guid":{"rendered":"https:\/\/dataopsschool.com\/blog\/?p=736"},"modified":"2025-08-19T13:46:51","modified_gmt":"2025-08-19T13:46:51","slug":"databricks-unity-catalog-vs-catalogs-vs-workspace-vs-metastore","status":"publish","type":"post","link":"https:\/\/dataopsschool.com\/blog\/databricks-unity-catalog-vs-catalogs-vs-workspace-vs-metastore\/","title":{"rendered":"Databricks: Unity Catalog vs Catalogs vs Workspace vs Metastore"},"content":{"rendered":"\n
\"\"<\/figure>\n\n\n\n
\n\n\n\n

\ud83d\udd11 Unity Catalog vs Catalogs vs Workspace vs Metastore<\/strong><\/h1>\n\n\n\n
\n\n\n\n

1. Unity Catalog (UC)<\/strong> \u2705<\/h2>\n\n\n\n
    \n
  • Think of it as the master governance system<\/strong>.<\/li>\n\n\n\n
  • It\u2019s account-level<\/strong> (above all workspaces).<\/li>\n\n\n\n
  • Manages:\n
      \n
    • Who can see what (permissions, ACLs).<\/li>\n\n\n\n
    • Metadata (table names, schemas, lineage).<\/li>\n\n\n\n
    • Secure data sharing across workspaces (Delta Sharing).<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n

      \ud83d\udc49 Analogy: National Library System<\/strong> \u2013 it governs all libraries in a country.<\/p>\n\n\n\n

      \n\n\n\n

      2. Catalogs<\/strong> \ud83d\udcda<\/h2>\n\n\n\n
        \n
      • A container for organizing data assets<\/strong> inside Unity Catalog.<\/li>\n\n\n\n
      • A catalog contains Schemas<\/strong> (databases).<\/li>\n\n\n\n
      • Within schemas, you have Tables, Views, Volumes, Functions, Models<\/strong>.<\/li>\n<\/ul>\n\n\n\n

        \ud83d\udc49 Analogy: A library<\/strong> inside the national library system.<\/p>\n\n\n\n

        \n\n\n\n

        3. Schemas (Databases)<\/strong><\/h2>\n\n\n\n
          \n
        • Sub-containers within Catalogs<\/strong>.<\/li>\n\n\n\n
        • Organize Tables<\/strong> and Views<\/strong>.<\/li>\n<\/ul>\n\n\n\n

          \ud83d\udc49 Analogy: Sections in the library<\/strong> (History, Science, Fiction).<\/p>\n\n\n\n

          \n\n\n\n

          4. Tables & Views<\/strong><\/h2>\n\n\n\n
            \n
          • Actual data objects<\/strong> stored in schemas.<\/li>\n\n\n\n
          • Tables<\/strong> \u2192 structured datasets (Delta by default).<\/li>\n\n\n\n
          • Views<\/strong> \u2192 saved queries on tables.<\/li>\n<\/ul>\n\n\n\n

            \ud83d\udc49 Analogy: Books<\/strong> on the library shelves.<\/p>\n\n\n\n

            \n\n\n\n

            5. Workspace<\/strong> \ud83d\udda5\ufe0f<\/h2>\n\n\n\n
              \n
            • A UI and compute environment<\/strong> where users collaborate (notebooks, jobs, clusters).<\/li>\n\n\n\n
            • Workspaces don\u2019t \u201cown\u201d the data; they just connect to Unity Catalog<\/strong> for governed data access.<\/li>\n<\/ul>\n\n\n\n

              \ud83d\udc49 Analogy: The reading room<\/strong> where you sit, study, and work with books.<\/p>\n\n\n\n

              \n\n\n\n

              6. Metastore<\/strong> \ud83d\udcd2<\/h2>\n\n\n\n
                \n
              • The backend metadata database<\/strong> that stores info about catalogs, schemas, tables, permissions.<\/li>\n\n\n\n
              • In Unity Catalog:\n
                  \n
                • Account-level Metastore<\/strong> is shared across workspaces.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                • In legacy mode:\n
                    \n
                  • Each workspace had its own Hive Metastore<\/strong> (separate, siloed).<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n

                    \ud83d\udc49 Analogy: The card catalog \/ index system<\/strong> telling you where each book is and who can borrow it.<\/p>\n\n\n\n

                    \n\n\n\n

                    \u2705 Hierarchy<\/strong><\/h1>\n\n\n\n
                    Unity Catalog (Account-level Governance)\n   \u2514\u2500\u2500 Metastore (Metadata storage)\n         \u2514\u2500\u2500 Catalogs (Top-level containers)\n               \u2514\u2500\u2500 Schemas (Databases)\n                     \u2514\u2500\u2500 Tables \/ Views \/ Volumes \/ Models\n<\/code><\/pre>\n\n\n\n
                    And Workspaces<\/strong> are where you interact with all of this (via notebooks, jobs, queries).<\/p>\n\n\n\n
                    \n\n\n\n
                    Key Distinction<\/strong><\/h1>\n\n\n\n
                      \n
                    • Unity Catalog<\/strong> = The system of rules + governance layer.<\/li>\n\n\n\n
                    • Catalog<\/strong> = Logical container for data inside UC.<\/li>\n\n\n\n
                    • Metastore<\/strong> = Metadata database that keeps track of it all.<\/li>\n\n\n\n
                    • Workspace<\/strong> = Your working environment (UI + compute) that connects to the above.<\/li>\n<\/ul>\n\n\n\n
                      \n\n\n\n
                      \"\"<\/figure>\n\n\n\n
                       <\/p>\n\n\n\n
                      Let\u2019s wrap up everything we\u2019ve discussed about Metastore \u2192 Catalog \u2192 Schema \u2192 Table<\/strong> and external locations<\/strong> into a single, step-by-step tutorial<\/strong> that you can follow on Databricks.<\/p>\n\n\n\n
                      \n\n\n\n
                      \ud83d\udcd8 Tutorial: Understanding Databricks Data Hierarchy & External Locations<\/h1>\n\n\n\n
                      \n\n\n\n
                      1. The Hierarchy<\/h2>\n\n\n\n
                      In Unity Catalog<\/strong>, Databricks enforces this hierarchy:<\/p>\n\n\n\n
                      Metastore \n   \u2514\u2500\u2500 Catalog \n         \u2514\u2500\u2500 Schema \n               \u2514\u2500\u2500 Table \/ View \/ Volume \/ Model\n<\/code><\/pre>\n\n\n\n
                        \n
                      • Metastore<\/strong> \u2192 The root metadata container<\/strong>. Every account gets one Unity Catalog metastore.<\/li>\n\n\n\n
                      • Catalog<\/strong> \u2192 Top-level logical container for schemas and data assets.<\/li>\n\n\n\n
                      • Schema (Database)<\/strong> \u2192 Organizes objects within a catalog.<\/li>\n\n\n\n
                      • Table<\/strong> \u2192 Stores data (managed or external).<\/li>\n<\/ul>\n\n\n\n
                        \n\n\n\n
                        2. Managed vs External Tables<\/h2>\n\n\n\n
                          \n
                        • Managed Table<\/strong>: Databricks manages both metadata + storage<\/strong>. Dropping the table deletes files.<\/li>\n\n\n\n
                        • External Table<\/strong>: Databricks manages only metadata<\/strong>. The data stays in your cloud storage when dropped.<\/li>\n<\/ul>\n\n\n\n
                          \n\n\n\n
                          3. External Location (Key Concept)<\/h2>\n\n\n\n
                            \n
                          • An External Location<\/strong> is a Unity Catalog object that maps a cloud storage path<\/strong> (S3, ADLS, GCS) + a storage credential<\/strong>.<\/li>\n\n\n\n
                          • Defined at the Metastore level<\/strong> \u2192 not at Catalog or Schema.<\/li>\n\n\n\n
                          • Used when creating external tables.<\/li>\n<\/ul>\n\n\n\n
                            Example:<\/p>\n\n\n\n
                            -- Step 1: Create storage credential (cloud-specific)\nCREATE STORAGE CREDENTIAL my_cred\nWITH AZURE_MANAGED_IDENTITY 'my-managed-identity'\nCOMMENT 'Credential for ADLS';\n\n-- Step 2: Register external location\nCREATE EXTERNAL LOCATION my_ext_loc\nURL 'abfss:\/\/external-container@mydatalake.dfs.core.windows.net\/data\/'\nWITH (STORAGE CREDENTIAL my_cred)\nCOMMENT 'External data location';\n<\/code><\/pre>\n\n\n\n
                            \n\n\n\n
                            4. Creating Catalog, Schema, and Tables<\/h2>\n\n\n\n
                            -- Create a catalog\nCREATE CATALOG sales_catalog;\n\n-- Create a schema (inside catalog)\nCREATE SCHEMA sales_catalog.sales_schema;\n\n-- Managed table (Databricks manages storage)\nCREATE TABLE sales_catalog.sales_schema.customers_managed (\n  id INT, name STRING\n);\n\n-- External table (you provide LOCATION)\nCREATE TABLE sales_catalog.sales_schema.customers_external\nUSING DELTA\nLOCATION 'abfss:\/\/external-container@mydatalake.dfs.core.windows.net\/data\/customers\/';\n<\/code><\/pre>\n\n\n\n
                            \n\n\n\n
                            5. Insert, Query, and Drop Data<\/h2>\n\n\n\n
                            -- Insert data (only works on managed tables)\nINSERT INTO sales_catalog.sales_schema.customers_managed VALUES (1, 'Alice'), (2, 'Bob');\n\n-- Query data\nSELECT * FROM sales_catalog.sales_schema.customers_managed;\n\n-- Drop table\nDROP TABLE sales_catalog.sales_schema.customers_managed;\n\n-- For external table \u2192 only metadata removed, data files remain\nDROP TABLE sales_catalog.sales_schema.customers_external;\n<\/code><\/pre>\n\n\n\n
                            \n\n\n\n
                            6. Quick Rules Recap \u2705<\/h2>\n\n\n\n
                              \n
                            • Metastore<\/strong> \u2192 Mandatory, stores metadata & external location definitions.<\/li>\n\n\n\n
                            • Catalog<\/strong> \u2192 Mandatory, logical top-level container.<\/li>\n\n\n\n
                            • Schema<\/strong> \u2192 Mandatory, organizes tables within catalogs.<\/li>\n\n\n\n
                            • Table<\/strong> \u2192 Optional, where the actual data lives.<\/li>\n\n\n\n
                            • External Location<\/strong> \u2192 Defined at Metastore<\/strong>, used at Table<\/strong> level.<\/li>\n<\/ul>\n\n\n\n
                              \n\n\n\n
                              \ud83c\udfaf Conclusion<\/h2>\n\n\n\n
                                \n
                              • You cannot<\/strong> define external locations at catalog or schema level.<\/li>\n\n\n\n
                              • You must<\/strong> define them at the metastore<\/strong> and use them when creating external tables<\/strong>.<\/li>\n\n\n\n
                              • Always think:\n
                                  \n
                                • Metastore = registry<\/strong><\/li>\n\n\n\n
                                • Catalog = library section<\/strong><\/li>\n\n\n\n
                                • Schema = shelf<\/strong><\/li>\n\n\n\n
                                • Table = book (data itself)<\/strong><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
                                  \n\n\n\n
                                  Catalogs themselves don\u2019t directly \u201cuse\u201d external locations<\/strong>, but you can associate them<\/strong> with external storage in Unity Catalog<\/strong>. Let me explain:<\/p>\n\n\n\n
                                  \n\n\n\n
                                  Catalog Storage in Databricks using External Location<\/h2>\n\n\n\n
                                  1. Default Behavior<\/strong><\/h3>\n\n\n\n
                                    \n
                                  • When you create a catalog in Unity Catalog: CREATE CATALOG sales_catalog;<\/code> \u2192 Databricks automatically assigns it a default storage location<\/strong> (in the metastore\u2019s root storage).\n
                                      \n
                                    • All managed tables created inside sales_catalog<\/code> go there by default.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
                                      \n\n\n\n
                                      2. Catalog With External Location<\/strong><\/h3>\n\n\n\n
                                        \n
                                      • You can<\/em> override this by explicitly binding a catalog to an external location<\/strong>: CREATE CATALOG sales_catalog MANAGED LOCATION 'abfss:\/\/container@storageacct.dfs.core.windows.net\/sales_data\/';<\/code>\n
                                          \n
                                        • Here, the catalog\u2019s default managed tables<\/strong> will live in that external location.<\/li>\n\n\n\n
                                        • This is sometimes called a \u201ccatalog-level managed location.\u201d<\/strong><\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
                                          \n\n\n\n
                                          3. Schema Level<\/strong><\/h3>\n\n\n\n
                                            \n
                                          • Similarly, you can also define a managed location for a schema<\/strong>: CREATE SCHEMA sales_catalog.retail MANAGED LOCATION 'abfss:\/\/container@storageacct.dfs.core.windows.net\/retail_data\/';<\/code>\n
                                              \n
                                            • Now tables created in this schema (without explicit LOCATION<\/code>) go here.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
                                              \n\n\n\n
                                              4. Table Level<\/strong><\/h3>\n\n\n\n
                                                \n
                                              • For external tables<\/strong>, you still provide the LOCATION<\/code> explicitly: CREATE TABLE sales_catalog.retail.customers USING DELTA LOCATION 'abfss:\/\/container@storageacct.dfs.core.windows.net\/customers\/';<\/code><\/li>\n<\/ul>\n\n\n\n
                                                \n\n\n\n
                                                \u2705 So to answer your question<\/h2>\n\n\n\n
                                                  \n
                                                • Yes, you can create a catalog in Databricks that uses an external location<\/strong> as its default managed storage<\/strong>.<\/li>\n\n\n\n
                                                • But \u26a0\ufe0f this doesn\u2019t mean all tables are external tables<\/em> \u2014 it just changes the default storage path<\/strong> for managed tables created in that catalog (or schema).<\/li>\n\n\n\n
                                                • External tables still need their own explicit LOCATION<\/code> or use a registered external location.<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"
                                                  \ud83d\udd11 Unity Catalog vs Catalogs vs Workspace vs Metastore 1. Unity Catalog (UC) \u2705 \ud83d\udc49 Analogy: National Library System \u2013 it governs all libraries in a country…. <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-736","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/736","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=736"}],"version-history":[{"count":6,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/736\/revisions"}],"predecessor-version":[{"id":747,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/736\/revisions\/747"}],"wp:attachment":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=736"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=736"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=736"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}