Databricks: User Management in Databricks

by

Introduction

In Databricks, identities (users, groups, service principals) live at the account level and can be assigned to one or more workspaces. For Unity Catalog (UC), principals must exist at the account level to be granted data privileges, and Databricks recommends provisioning via SCIM from your IdP (Microsoft Entra ID). (Microsoft Learn)

Roles you typically need

  • Cloud admin (Azure) to create users/service principals in Entra ID.
  • Account admin to add principals to the Databricks account and assign to workspaces.
  • Workspace admin to manage workspace entitlements (personas) and workspace-local settings. (Databricks Documentation)

How to add/create new users in Microsoft Entra ID (Azure AD)

  1. In the Azure portal, open Microsoft Entra ID → Users → New user.
  2. Set User principal name (UPN) and Display name, create an initial password, and Create.
  3. (Optional) Add the user to Entra groups you’ll sync to Databricks later via SCIM.

Entra “users” become principals you can provision into Databricks with SCIM. (Microsoft Learn)

How to add/create new users in the Databricks Account console

✅ Best practice: Use SCIM to sync identities automatically (next section).
If you must add manually:

  1. Go to Account console → User management → Users → Add user.
  2. Enter the user’s email/UPN and Add.
  3. (Optional) Assign account admin or other account-level roles if needed (use sparingly). (Microsoft Learn)

Account admins can also assign users to workspaces from the account console (assuming identity federation). (Microsoft Learn)

What is SCIM in Databricks?

SCIM (System for Cross-domain Identity Management) is the standard used to auto-provision users and groups from your IdP (Microsoft Entra ID) into the Databricks account. Once configured, new Entra users/groups sync automatically into Databricks (no manual adds), and removals/updates propagate. (Microsoft Learn)

High-level setup (Entra → Databricks account)

  1. In Entra, create/select the Azure Databricks (SCIM) enterprise app (account-level).
  2. Under Provisioning, set Provisioning Status = On, configure the Databricks SCIM endpoint + token, and Save.
  3. Add Users/Groups to the enterprise app (assignments). Entra will sync them on its schedule.
  4. (Recommended) In Properties, set Assignment required = No if you want all IdP users to sign in to the Databricks account. (Microsoft Learn, Databricks Documentation)

If you previously synced identities to individual workspaces, disable those connectors when enabling account-level SCIM. (Databricks Documentation)

Create and Use Groups in Databricks

Why groups? Grant permissions to a group (e.g., data-analysts, data-engineers) instead of individual users. This simplifies workspace access, persona/entitlements, and data permissions in Unity Catalog. (Microsoft Learn)

Create groups

  • If you use SCIM, create and manage groups in Entra and let SCIM sync them to Databricks.
  • If manual: Account console → User management → Groups → Add group; add members. (Microsoft Learn)

Use cases

  • data-analysts: Databricks SQL persona only.
  • data-engineers: Workspace persona (DE/ML), plus Cluster create ability.
  • platform-admins: Workspace admins.

Assign users (or groups) to a Workspace

From the Account console:

  1. Workspaces → choose a workspace → Permissions.
  2. Add permissions → search your group (preferred) or individual user → assign User or Workspace admin → Save. (Microsoft Learn)

Assign groups instead of individual users so new team members inherit access automatically.

Workspace-level access or Persona in Databricks (Entitlements)

Within each workspace, a principal needs the right entitlements (“personas”):

  • Workspace access: access to Data Engineering / ML UI (core workspace).
  • Databricks SQL access: access to SQL persona (warehouses, dashboards).
  • Allow cluster creation: permission to create clusters (DE/ML persona).
    Entitlements are available in Premium or above and managed by a workspace admin. (Microsoft Learn, Databricks Documentation)

Where to set (workspace)

  • Workspace Settings → Identity and access → Manage (Users/Groups/Service principals) → Entitlements → toggle Workspace access, Databricks SQL access, Allow cluster creation. (Microsoft Learn)

Examples

  • Data Analyst group: enable Databricks SQL access, disable Workspace access and Allow cluster creation → gives just SQL persona (warehouses), no clusters.
  • Data Engineer group: enable Workspace access and Allow cluster creation; optionally disable Databricks SQL access if they don’t need warehouses. (Microsoft Learn)

How to create a Service Principal in Microsoft Azure

A service principal (SP) is a non-human identity you use for automation (Jobs, pipelines, CI/CD, API/CLI). You can create:

  • A Databricks-managed SP in the Account console, or
  • A Microsoft Entra ID app registration (recommended for Azure automation outside Databricks).

Create Entra service principal (app registration)

  1. Entra ID → App registrations → New registration → Name (e.g., etl-automation-sp) → Register.
  2. Copy Application (client) ID and Directory (tenant) ID.
  3. Create a client secret under Certificates & secrets; copy the secret value. (Microsoft Learn)

You’ll use these when granting the SP access to Databricks and other Azure resources.

Add Service Principal to the Databricks Workspace

At the account level

  • Account console → User management → Service principals → Add.
    • Choose Microsoft Entra ID managed and paste the Application (client) ID, or
    • Create a Databricks-managed service principal. (Databricks Documentation)

Assign the SP to workspaces

  • Account console → Workspaces → [Workspace] → Permissions → Add permissions → select the SP → choose User (or Workspace admin, rarely) → Save.
  • In the workspace, a workspace admin can set SP entitlements and who can manage/use the SP (Permissions tab on the SP). (Databricks Documentation)

Use cases

  • Run as a service principal for Jobs/Workflows to avoid dependency on human accounts.
  • Use OAuth (M2M) or PATs scoped to the SP for API/CLI automations. (Databricks Documentation)

Handy snippets & checklists

Databricks CLI (optional) quick checks

# List account/workspace scopes or principals (run in the right context/profile)
databricks secrets list-scopes
databricks groups list        # list workspace groups (if using workspace API)
databricks service-principals list

(Use the Account vs Workspace CLI context appropriately.)

Who can manage what?

  • Account admin: add users/groups/SPs to the account, assign to workspaces, and configure identity federation/SCIM. (Microsoft Learn)
  • Workspace admin: grant entitlements (personas), manage local settings, optionally make a principal a workspace admin. (Microsoft Learn)

Best practices

  • Provision via SCIM from Entra ID; manage group membership in the IdP. (Microsoft Learn)
  • Grant access to groups, not individuals. (Microsoft Learn)
  • Keep Workspace admin role minimal; use personas (entitlements) tailored to job function. (Microsoft Learn)
  • For UC permissions, always ensure principals exist at the account level. (Microsoft Learn)
  • Use service principals for jobs/automation (“Run as” SP), not human users. (Databricks Documentation)