{"id":734,"date":"2025-08-19T12:49:35","date_gmt":"2025-08-19T12:49:35","guid":{"rendered":"https:\/\/dataopsschool.com\/blog\/?p=734"},"modified":"2025-08-19T12:49:36","modified_gmt":"2025-08-19T12:49:36","slug":"databricks-components","status":"publish","type":"post","link":"https:\/\/dataopsschool.com\/blog\/databricks-components\/","title":{"rendered":"Databricks Components"},"content":{"rendered":"\n

<\/p>\n\n\n\n

\n\n\n\n

Databricks Components Hierarchy<\/strong><\/h1>\n\n\n\n

1. Account Level (Top Layer)<\/strong><\/h3>\n\n\n\n
    \n
  • Account Console<\/strong> \u2013 central place to manage everything across workspaces.<\/li>\n\n\n\n
  • Workspaces<\/strong> \u2013 logical environments where teams work.<\/li>\n\n\n\n
  • Unity Catalog (Metastore)<\/strong> \u2013 unified governance across all workspaces.<\/li>\n<\/ul>\n\n\n\n
    \n\n\n\n

    2. Governance & Data Management<\/strong><\/h3>\n\n\n\n
      \n
    • Unity Catalog<\/strong>\n
        \n
      • Catalogs<\/strong> \u2192 top container of data assets.<\/li>\n\n\n\n
      • Schemas (Databases)<\/strong> \u2192 inside catalogs.<\/li>\n\n\n\n
      • Tables<\/strong> \u2192 structured data (Managed \/ External).<\/li>\n\n\n\n
      • Views<\/strong> \u2192 logical queries on tables.<\/li>\n\n\n\n
      • Volumes<\/strong> \u2192 for non-tabular data (images, PDFs, etc.).<\/li>\n\n\n\n
      • Models<\/strong> \u2192 ML models registered.<\/li>\n\n\n\n
      • Functions<\/strong> \u2192 SQL or Python-defined functions.<\/li>\n\n\n\n
      • Lineage<\/strong> \u2192 track where data comes from and how it\u2019s used.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
      • Access Control<\/strong>\n
          \n
        • Users<\/strong> \u2192 individual identities.<\/li>\n\n\n\n
        • Groups<\/strong> \u2192 manage permissions collectively.<\/li>\n\n\n\n
        • Service Principals<\/strong> \u2192 for apps\/automation.<\/li>\n\n\n\n
        • ACLs (Access Control Lists)<\/strong> \u2192 fine-grained permissions.<\/li>\n\n\n\n
        • Personal Access Tokens (PATs)<\/strong> \u2192 authentication for APIs.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
          \n\n\n\n

          3. Computation & Execution<\/strong><\/h3>\n\n\n\n
            \n
          • Clusters<\/strong>\n
              \n
            • All-purpose clusters<\/strong> \u2192 interactive, shared by users.<\/li>\n\n\n\n
            • Job clusters<\/strong> \u2192 spin up just for a job, then shut down.<\/li>\n\n\n\n
            • Pools<\/strong> \u2192 pre-warmed instances to reduce cluster spin-up time.<\/li>\n\n\n\n
            • Databricks Runtime (DBR)<\/strong> \u2192 core software stack (Spark + optimizations).\n
                \n
              • DBR for Machine Learning (ML\/DL libraries pre-installed).<\/li>\n\n\n\n
              • DBR for Genomics, SQL, etc.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n<\/li>\n\n\n\n
              • Jobs & Pipelines<\/strong>\n
                  \n
                • Jobs UI<\/strong> \u2192 scheduling & automation of notebooks, SQL, scripts.<\/li>\n\n\n\n
                • Lakeflow Declarative Pipelines<\/strong> \u2192 manage Delta tables with orchestration.<\/li>\n\n\n\n
                • Workflows<\/strong> \u2192 CI\/CD style orchestration.<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                • Workloads<\/strong>\n
                    \n
                  • Data Engineering<\/strong> \u2192 ETL, batch jobs.<\/li>\n\n\n\n
                  • Data Analytics<\/strong> \u2192 interactive queries, dashboards.<\/li>\n\n\n\n
                  • Machine Learning<\/strong> \u2192 model training\/inference.<\/li>\n\n\n\n
                  • Streaming<\/strong> \u2192 real-time with Structured Streaming.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
                    \n\n\n\n

                    4. Developer Interfaces<\/strong><\/h3>\n\n\n\n
                      \n
                    • Workspace UI<\/strong> \u2192 notebooks, data, clusters, jobs, dashboards.<\/li>\n\n\n\n
                    • Notebooks<\/strong> \u2192 code in Python, SQL, R, Scala.<\/li>\n\n\n\n
                    • Dashboards<\/strong> \u2192 visual insights.<\/li>\n\n\n\n
                    • Git Folders (Repos)<\/strong> \u2192 version control integration.<\/li>\n\n\n\n
                    • Libraries<\/strong> \u2192 attach external or custom libraries.<\/li>\n\n\n\n
                    • Catalog Explorer<\/strong> \u2192 browse data assets.<\/li>\n\n\n\n
                    • APIs & Tools<\/strong>\n
                        \n
                      • REST API<\/strong> \u2192 programmatic access.<\/li>\n\n\n\n
                      • SQL REST API<\/strong> \u2192 SQL automation.<\/li>\n\n\n\n
                      • CLI<\/strong> \u2192 Databricks command line tool.<\/li>\n\n\n\n
                      • dbutils<\/strong> \u2192 utility commands inside notebooks.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
                        \n\n\n\n

                        5. Data & AI Layers<\/strong><\/h3>\n\n\n\n
                          \n
                        • Delta Lake (Default Table Format)<\/strong>\n
                            \n
                          • Delta Tables<\/li>\n\n\n\n
                          • Delta Transaction Logs (ACID)<\/li>\n\n\n\n
                          • Time Travel, Schema Evolution<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                          • Lakehouse Storage Pattern<\/strong>\n
                              \n
                            • Bronze \u2192 Raw data<\/li>\n\n\n\n
                            • Silver \u2192 Clean\/curated data<\/li>\n\n\n\n
                            • Gold \u2192 Business-ready data<\/li>\n<\/ul>\n<\/li>\n\n\n\n
                            • AI & ML (Mosaic AI)<\/strong>\n
                                \n
                              • MLflow<\/strong> \u2192 experiment tracking, model registry.<\/li>\n\n\n\n
                              • Feature Store<\/strong> \u2192 reusable features for ML.<\/li>\n\n\n\n
                              • Generative AI (LLMs)<\/strong> \u2192 foundation models, fine-tuning.<\/li>\n\n\n\n
                              • AI Playground<\/strong> \u2192 test LLMs interactively.<\/li>\n\n\n\n
                              • Model Serving<\/strong> \u2192 REST API for deploying models.<\/li>\n<\/ul>\n<\/li>\n<\/ul>\n\n\n\n
                                \n\n\n\n

                                \u2705 In one line:<\/strong><\/p>\n\n\n\n

                                  \n
                                • Account Console<\/strong> (top) \u2192 Workspaces<\/strong> \u2192 Unity Catalog (Governance)<\/strong> \u2192 Data Assets<\/strong> (Tables, Schemas, Models, Volumes) \u2192 Compute (Clusters, Jobs, Pipelines)<\/strong> \u2192 Developer Interfaces (Notebooks, APIs, CLI)<\/strong> \u2192 AI\/ML & Analytics Tools<\/strong>.<\/li>\n<\/ul>\n\n\n\n
                                  \n\n\n\n

                                  <\/p>\n","protected":false},"excerpt":{"rendered":"

                                  Databricks Components Hierarchy 1. Account Level (Top Layer) 2. Governance & Data Management 3. Computation & Execution 4. Developer Interfaces 5. Data & AI Layers \u2705 In… <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-734","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/734","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=734"}],"version-history":[{"count":1,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/734\/revisions"}],"predecessor-version":[{"id":735,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/734\/revisions\/735"}],"wp:attachment":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=734"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=734"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=734"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}