<\/p>\n\n\n\n
Hard-coding credentials (DB passwords, API tokens, SAS keys, hosts) in notebooks or jobs is risky. In Databricks you store them as secrets<\/strong> inside a secret scope<\/strong>, then read them safely at runtime (not printed in plain text). Databricks supports two scope types: Azure Key Vault-backed<\/strong> and Databricks-backed<\/strong>. (Microsoft Learn<\/a>)<\/p>\n\n\n\n Secret scope<\/strong> = a named container for secrets. Tip: AKV-backed scopes grant Databricks Get\/List<\/strong> permissions to the workspace\u2019s service app so it can read secrets; you still manage the secrets in AKV<\/strong>. (Microsoft Learn<\/a>)<\/p>\n<\/blockquote>\n\n\n\n Azure Portal \u2192 your Key Vault \u2192 Objects > Secrets<\/strong> \u2192 Generate\/Import<\/strong><\/p>\n\n\n\n (You can also use Azure\u2019s In a Databricks notebook:<\/p>\n\n\n\n Databricks redacts<\/strong> secret values printed to outputs as Use these when you don\u2019t need AKV, want quick setup, or are multi-cloud. Scopes & secrets are managed in the workspace via CLI \/ API<\/strong>, and you can control access via secret scope ACLs<\/strong>. (Microsoft Learn<\/a>)<\/p>\n\n\n\n Install the current CLI (0.205+). Verify:<\/p>\n\n\n\n Docs: CLI commands overview. (Microsoft Learn<\/a>)<\/p>\n\n\n\n (You can also auth with a service principal for automation.) (Microsoft Learn<\/a>)<\/p>\n\n\n\n List\/verify:<\/p>\n\n\n\n (Microsoft Learn<\/a>)<\/p>\n\n\n\n CLI 0.205+ (\u2018put-secret\u2019):<\/strong><\/p>\n\n\n\n List secrets (metadata only):<\/p>\n\n\n\n Read (CLI returns base64; decode if you must read via CLI):<\/p>\n\n\n\n (Usually you read secrets in notebooks with Compatibility: older CLI used End-to-end JDBC + secrets example: official tutorial. (Microsoft Learn<\/a>)<\/p>\n\n\n\n Cluster config supports Redaction: Databricks replaces printed secret literals with Secret permissions are at scope<\/strong> level (READ \/ WRITE \/ MANAGE). Use the CLI to manage ACLs:<\/p>\n\n\n\n Notes:<\/p>\n\n\n\n AKV-backed scope (UI):<\/strong><\/p>\n\n\n\n Databricks-backed scope (CLI):<\/strong><\/p>\n\n\n\n (Microsoft Learn<\/a>)<\/p>\n\n\n\n Use in code:<\/strong><\/p>\n\n\n\n (Databricks Documentation<\/a>)<\/p>\n\n\n\n <\/p>\n\n\n\n <\/p>\n","protected":false},"excerpt":{"rendered":" Introduction Hard-coding credentials (DB passwords, API tokens, SAS keys, hosts) in notebooks or jobs is risky. In Databricks you store them as secrets inside a secret scope,… <\/p>\n","protected":false},"author":1,"featured_media":0,"comment_status":"closed","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[1],"tags":[],"class_list":["post-838","post","type-post","status-publish","format-standard","hentry","category-uncategorized"],"_links":{"self":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/838","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=838"}],"version-history":[{"count":1,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/838\/revisions"}],"predecessor-version":[{"id":839,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/838\/revisions\/839"}],"wp:attachment":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=838"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=838"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=838"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}
\n\n\n\nWhat is a Secret Scope in Databricks?<\/h1>\n\n\n\n
Types:<\/p>\n\n\n\n\n
You create scopes, set permissions (ACLs), add secrets, and reference them in notebooks\/jobs. (Microsoft Learn<\/a>)<\/li>\n<\/ul>\n\n\n\n
\n\n\n\n01:04 \u2014 Azure Key Vault\u2013backed Secret Scope (AKV)<\/h1>\n\n\n\n
A) Prepare Azure Key Vault (one-time)<\/h2>\n\n\n\n
\n
\n
B) Add a secret to AKV (example)<\/h2>\n\n\n\n
\n
db-password<\/code><\/li>\n\n\n\nS0m3Str0ngP@ss<\/code><\/li>\n\n\n\nSetSecret<\/code> REST\/CLI.) (Microsoft Learn<\/a>)<\/p>\n\n\n\n
\n\n\n\nCreate the AKV-backed scope (Databricks UI)<\/h1>\n\n\n\n
\n
https:\/\/<your-workspace-url>#secrets\/createScope<\/code> (case-sensitive<\/strong>: createScope<\/code> with capital S<\/strong>). (Microsoft Learn<\/a>)<\/li>\n\n\n\nakv<\/code> (any name; case-insensitive).<\/li>\n\n\n\nhttps:\/\/mykv.vault.azure.net\/<\/code>) and Resource ID<\/strong> (from KV Properties<\/strong>). (Microsoft Learn<\/a>)<\/li>\n\n\n\ndatabricks secrets list-scopes # expect to see: akv<\/code> (Microsoft Learn<\/a>)<\/li>\n<\/ol>\n\n\n\n
\n\n\n\nUse
dbutils.secrets<\/code> (help, list, get)<\/h1>\n\n\n\n# What can I do?\ndbutils.secrets.help()\n\n# List all scopes\ndbutils.secrets.listScopes()\n\n# List secrets (metadata only) in a scope\ndbutils.secrets.list(\"akv\")\n\n# Read a secret (value is redacted in outputs)\npassword = dbutils.secrets.get(scope=\"akv\", key=\"db-password\")\n<\/code><\/pre>\n\n\n\n[REDACTED]<\/code>. Use them directly in code instead of printing. (Databricks Documentation<\/a>, Microsoft Learn<\/a>)<\/p>\n\n\n\n
\n\n\n\nDatabricks-backed Secret Scopes<\/h1>\n\n\n\n
Set up the Databricks CLI<\/h2>\n\n\n\n
databricks --version\n<\/code><\/pre>\n\n\n\nAuthenticate the CLI with your workspace<\/h2>\n\n\n\n
# Creates\/updates a profile interactively (opens browser):\ndatabricks auth login\n\n# Or set env creds \/ profiles as needed.\n<\/code><\/pre>\n\n\n\nManage Secret Scopes using the CLI<\/h2>\n\n\n\n
1) Create a Databricks-backed<\/strong> scope<\/h3>\n\n\n\n
databricks secrets create-scope my-db-scope\n<\/code><\/pre>\n\n\n\ndatabricks secrets list-scopes\n<\/code><\/pre>\n\n\n\n2) Put a secret into the scope<\/h3>\n\n\n\n
# Option A: JSON (string_value)\ndatabricks secrets put-secret --json '{\n \"scope\": \"my-db-scope\",\n \"key\": \"db-host\",\n \"string_value\": \"xyz.example.com\"\n}'\n\n# Option B: via stdin (multi-line)\n( cat << 'EOF'\nvery\nsecret\nvalue\nEOF\n) | databricks secrets put-secret my-db-scope db-password\n<\/code><\/pre>\n\n\n\ndatabricks secrets list-secrets my-db-scope\n<\/code><\/pre>\n\n\n\ndatabricks secrets get-secret my-db-scope db-password | jq -r .value | base64 --decode\n<\/code><\/pre>\n\n\n\ndbutils.secrets.get<\/code>.) (Microsoft Learn<\/a>)<\/p>\n\n\n\n\n
databricks secrets put --scope ... --key ... --string-value ...<\/code>. Prefer put-secret<\/code> on new CLI. (Databricks Documentation<\/a>)<\/p>\n<\/blockquote>\n\n\n\n
\n\n\n\nUse secrets in notebooks & jobs (examples)<\/h1>\n\n\n\n
A) JDBC with Python<\/h2>\n\n\n\n
user = dbutils.secrets.get(\"my-db-scope\", \"db-user\")\npwd = dbutils.secrets.get(\"my-db-scope\", \"db-password\")\njdbc_url = dbutils.secrets.get(\"my-db-scope\", \"jdbc-url\")\n\ndf = (spark.read.format(\"jdbc\")\n .option(\"url\", jdbc_url)\n .option(\"dbtable\", \"public.orders\")\n .option(\"user\", user)\n .option(\"password\", pwd)\n .load())\n<\/code><\/pre>\n\n\n\nB) Reference secrets in Spark config<\/strong> (preview) or environment variables<\/strong><\/h2>\n\n\n\n
{{secrets\/<scope>\/<key>}}<\/code> placeholders:<\/p>\n\n\n\n\n
spark.password {{secrets\/my-db-scope\/db-password}}<\/code> Use later: spark.conf.get(\"spark.password\")<\/code><\/li>\n\n\n\nMY_DB_PWD={{secrets\/my-db-scope\/db-password}}<\/code> Then use $MY_DB_PWD<\/code> in an init script or code.
(Mind the security considerations in that doc.) (Microsoft Learn<\/a>)<\/li>\n<\/ul>\n\n\n\n\n
[REDACTED]<\/code> in notebook output (and attempts to redact in SQL too), but follow permission best practices\u2014redaction isn\u2019t a substitute for access control. (Microsoft Learn<\/a>)<\/p>\n<\/blockquote>\n\n\n\n
\n\n\n\nPermissions (ACLs) for secret scopes<\/h1>\n\n\n\n
# Grant\ndatabricks secrets put-acl my-db-scope alice@example.com READ\ndatabricks secrets put-acl my-db-scope data-engineers MANAGE\n\n# View\ndatabricks secrets list-acls my-db-scope\ndatabricks secrets get-acl my-db-scope alice@example.com\n\n# Revoke\ndatabricks secrets delete-acl my-db-scope alice@example.com\n<\/code><\/pre>\n\n\n\n\n
\n\n\n\nUse-cases<\/h2>\n\n\n\n
\n
\n\n\n\nBest practices & gotchas<\/h2>\n\n\n\n
\n
etl-prod<\/code>, bi-prod<\/code>) rather than individuals; rotate secrets regularly.<\/li>\n<\/ul>\n\n\n\n
\n\n\n\nQuick \u201ccheat sheet\u201d<\/h2>\n\n\n\n
\n
https:\/\/<workspace>#secrets\/createScope<\/code> \u2192 name scope \u2192 DNS Name<\/strong> + Resource ID<\/strong> \u2192 Create. (Microsoft Learn<\/a>)<\/li>\n<\/ol>\n\n\n\ndatabricks auth login\ndatabricks secrets create-scope my-db-scope\ndatabricks secrets put-secret --json '{\"scope\":\"my-db-scope\",\"key\":\"db-user\",\"string_value\":\"demo\"}'\ndatabricks secrets list-scopes\ndatabricks secrets list-secrets my-db-scope\n<\/code><\/pre>\n\n\n\npwd = dbutils.secrets.get(\"my-db-scope\",\"db-password\")\n<\/code><\/pre>\n\n\n\n
\n\n\n\nReferences (key docs)<\/h3>\n\n\n\n
\n