{"id":437,"date":"2025-08-11T17:17:07","date_gmt":"2025-08-11T17:17:07","guid":{"rendered":"https:\/\/dataopsschool.com\/blog\/?p=437"},"modified":"2025-08-11T17:17:08","modified_gmt":"2025-08-11T17:17:08","slug":"databricks-service-principal-in-databricks-using-azure","status":"publish","type":"post","link":"https:\/\/dataopsschool.com\/blog\/databricks-service-principal-in-databricks-using-azure\/","title":{"rendered":"Databricks: Service Principal in Databricks using Azure?"},"content":{"rendered":"\n<h2 class=\"wp-block-heading\" id=\"what-is-a-service-principal-in-databricks\">What Is a Service Principal in Databricks?<\/h2>\n\n\n\n<p>A <strong>service principal<\/strong> is a specialized, non-human identity within Azure Databricks, designed exclusively for automation, integrations, and programmatic access. Service principals are intended for use by tools, scripts, CI\/CD pipelines, or external systems\u2014never by individual users. They provide <strong>API-only access<\/strong> to Databricks resources, which increases security and stability by decoupling permissions from user accounts.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Key Features<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Security:<\/strong> No risk of workflow interruptions when users change roles or leave the organization.<\/li>\n\n\n\n<li><strong>Fine-grained Access:<\/strong> Can be granted specific entitlements (e.g., workspace access, SQL access) or admin roles.<\/li>\n\n\n\n<li><strong>API-Only:<\/strong> Cannot log into the Databricks UI directly.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"use-cases\">Use Cases<\/h2>\n\n\n\n<h2 class=\"wp-block-heading\">At the Databricks Account Console Level<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Global automation across multiple workspaces (e.g., create workspaces, assign users\/groups, manage Unity Catalog, auditing, and workspace configurations).<\/li>\n\n\n\n<li>Central identity for CI\/CD pipelines, Terraform\/Pulumi scripts, or admin task automations that span all organizational Databricks resources.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">At the Databricks Workspace Level<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Manage and automate workspace resources (clusters, jobs, notebooks).<\/li>\n\n\n\n<li>Programmatic data access and ingest, including API access to tables, Delta Lake resources, and job runs.<\/li>\n\n\n\n<li>Secure credential for data engineering pipelines or scheduled jobs that need persistent, stable permissions.<\/li>\n\n\n\n<li>Running jobs &#8220;as service principal&#8221; so workflows don&#8217;t fail if a user account changes or is removed.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"how-to-use-service-principal-step-by-step-with-cur\">How to Use Service Principal: Step-by-Step with cURL<\/h2>\n\n\n\n<p><strong>Prerequisites:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>You must be an account or workspace admin.<\/li>\n\n\n\n<li>You need a registered service principal with appropriate roles\/entitlements.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">1. Create\/Assign Service Principal<\/h2>\n\n\n\n<h2 class=\"wp-block-heading\">Account Console<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Log into the Databricks Account Console.<\/li>\n\n\n\n<li>Go to \u201cUser management\u201d > \u201cService principals\u201d > \u201cAdd service principal\u201d, enter details, and add.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">Workspace<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Go to Workspace UI > Settings > Identity and Access > Manage > Add Service Principal.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">2. Grant Permissions and Generate Token\/Secret<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Assign roles (User\/Manager) and required entitlements.<\/li>\n\n\n\n<li>Generate OAuth secret or Personal Access Token (PAT) for API usage.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\">3. Authenticate with cURL for Databricks REST APIs<\/h2>\n\n\n\n<h2 class=\"wp-block-heading\">Example: Create a Personal Access Token for Service Principal<\/h2>\n\n\n\n<pre class=\"wp-block-preformatted\">bash<code>curl -X POST \\\n  https:\/\/&lt;databricks-instance&gt;\/api\/2.0\/token-management\/on-behalf-of\/tokens \\\n  --header \"Content-Type: application\/json\" \\\n  --header \"Authorization: Bearer &lt;ADMIN_PERSONAL_ACCESS_TOKEN&gt;\" \\\n  --data '{\n     \"principal\": \"&lt;service-principal-id&gt;\",\n     \"comment\": \"Token for service principal automation\"\n   }'\n<\/code><\/pre>\n\n\n\n<p><em>You need an admin token or OAuth for initial API access. The returned token is your service principal\u2019s API credential.<\/em><\/p>\n\n\n\n<h2 class=\"wp-block-heading\">Example: Use Service Principal to List Databricks Jobs<\/h2>\n\n\n\n<p>(Assume <code>&lt;SP_PAT&gt;<\/code> is the token generated for the service principal)<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">bash<code>curl -X GET \\\n  https:\/\/&lt;databricks-instance&gt;\/api\/2.1\/jobs\/list \\\n  --header \"Authorization: Bearer &lt;SP_PAT&gt;\"\n<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\">4. Create and Use Storage Credential (Advanced Example)<\/h2>\n\n\n\n<p>For Unity Catalog or storage integration, you may need to create a storage credential with service principal for secure access.<\/p>\n\n\n\n<pre class=\"wp-block-preformatted\">bash<code>curl -X POST \\\n  https:\/\/&lt;databricks-instance&gt;\/api\/2.1\/unity-catalog\/storage-credentials \\\n  -d '{\n    \"name\": \"sp-credential\",\n    \"azure_service_principal\": {\n      \"directory_id\": \"&lt;tenant-id&gt;\",\n      \"application_id\": \"&lt;sp-client-id&gt;\",\n      \"client_secret\": \"&lt;sp-client-secret&gt;\"\n    },\n    \"skip_validation\": false\n  }'\n<\/code><\/pre>\n\n\n\n<p>This sets up data access using the service principal identity.<\/p>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"summary-table-service-principal-use-cases\">Summary Table: Service Principal Use Cases<\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table class=\"has-fixed-layout\"><thead><tr><th>Level<\/th><th>Use Case Examples<\/th><\/tr><\/thead><tbody><tr><td>Account Console<\/td><td>Workspace automation, global governance, CI\/CD<\/td><\/tr><tr><td>Workspace<\/td><td>Data access, job automation, scheduled pipelines<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n<hr class=\"wp-block-separator has-alpha-channel-opacity\"\/>\n\n\n\n<p><strong>To use service principals in Databricks:<\/strong><\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Register and assign them at account or workspace level.<\/li>\n\n\n\n<li>Grant relevant permissions\/entitlements.<\/li>\n\n\n\n<li>Generate a token for API authentication.<\/li>\n\n\n\n<li>Execute REST API calls securely with cURL\u2014ideal for automation, integration, and stable orchestration of Databricks resources.<\/li>\n<\/ol>\n\n\n\n<p><\/p>\n","protected":false},"excerpt":{"rendered":"<p>What Is a Service Principal in Databricks? A service principal is a specialized, non-human identity within Azure Databricks, designed exclusively for automation, integrations, and programmatic access. Service&#8230; <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-437","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/437","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=437"}],"version-history":[{"count":1,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/437\/revisions"}],"predecessor-version":[{"id":438,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/437\/revisions\/438"}],"wp:attachment":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=437"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=437"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=437"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}