Databricks: Custom Cluster Policies & Instance Pools in Databricks

Perf

---

1. 🔹 Why Policies and Pools?

• Policies → Standardize and enforce cluster configurations across your organization.
• Instance Pools → Pre-create and reuse VMs to reduce startup time for clusters.

These features are critical in enterprise Databricks deployments to enforce compliance, control costs, and improve performance.

---

2. Custom Cluster Policies in Databricks

📌 What is a Cluster Policy?

• A JSON template that defines allowed, fixed, or forbidden cluster settings.
• Ensures users follow org standards (e.g., fixed runtime, mandatory auto-termination).

---

🛠 How to Create a Custom Policy

1. Go to Admin Console → Cluster Policies.
2. Instead of creating from scratch, clone an existing policy (e.g., Shared Compute).
3. Edit the JSON to override settings. Example:

{
  "autotermination_minutes": {
    "type": "fixed",
    "value": 10
  },
  "num_workers": {
    "type": "fixed",
    "value": 1
  },
  "autoscale.min_workers": {
    "type": "forbidden"
  },
  "autoscale.max_workers": {
    "type": "forbidden"
  },
  "spark_version": {
    "type": "fixed",
    "value": "15.4.x-scala2.12"
  },
  "node_type_id": {
    "type": "enum",
    "values": ["Standard_DS3_v2", "Standard_DS4_v2"],
    "defaultValue": "Standard_DS4_v2"
  }
}

🔑 Explanation:

• Auto termination → Always 10 minutes.
• Fixed workers → No autoscaling allowed.
• Fixed runtime → Spark 15.4 only.
• Restricted VM types → Only Standard_DS3_v2 or DS4_v2 allowed.

---

📌 How to Apply Policy to New Clusters

• When creating a cluster, select Policy → Custom Policy Name.
• UI will grey out forbidden fields (e.g., Spark version, node type).

---

📌 Enforcing Policy on Existing Clusters

• If policy changes (e.g., Spark version updated), old clusters show “Non-compliant”.
• Click Fix All → Databricks auto-updates them to comply.
• Example: Changing Spark version from 14.3 → 15.4 updates all linked clusters.

✅ This ensures org-wide compliance instantly.

---

3. Instance Pools in Databricks

📌 What is an Instance Pool?

• A predefined set of VMs ready to be attached to clusters.
• Benefit → Reduce startup time (clusters don’t need to wait for VM provisioning).
• Clusters draw workers from the pool instead of requesting fresh VMs.

---

🛠 How to Create an Instance Pool

1. Go to Compute → Instance Pools → Create Pool.
2. Configure:
   • Min Idle Instances → Always running. Keeps pool “warm.”
     • Example: 2 = always 2 ready VMs.
   • Max Capacity → Upper limit of VMs in the pool.
     • Example: 10 = pool can scale up to 10 nodes.
   • Idle Auto Termination → Time (mins) after which unused VMs shut down.
   • Node Type → VM family (e.g., DS4_v2).
   • Databricks Runtime (DBR) → Pre-load runtime for faster attach.

---

📌 Warm vs Cold Pools

• Warm Pool → Min Idle > 0 (e.g., 2 VMs always running).
  • ✅ Fast startup (sub-second).
  • ❌ Higher cost (pay for idle VMs).
• Cold Pool → Min Idle = 0.
  • ✅ Cost-efficient (no idle VMs).
  • ❌ Slower startup (still need to spin up VMs).

---

Example: Warm Instance Pool

Name: demo-pool
Min Idle Instances: 1
Max Capacity: 10
Idle Auto Termination: 10 mins
Node Type: Standard_DS4_v2
Runtime: 15.4 LTS

• At least 1 VM always running.
• Jobs launch instantly by borrowing warm node.
• Released nodes wait 10 mins before termination → reused if another job comes.

---

4. Best Practices

✅ Use Custom Policies to:

• Enforce auto-termination (prevent zombie clusters).
• Fix runtime versions (e.g., always LTS).
• Restrict node types to control cost.
• Disable autoscaling if not needed.

✅ Use Warm Pools for:

• Low-latency SLA jobs (e.g., real-time ETL, streaming, dashboards).

✅ Use Cold Pools for:

• Batch jobs that can tolerate 2–5 min startup delay.

---

5. Key Differences: Policy vs Pool

Feature	Cluster Policy 🚦	Instance Pool 🏊
Purpose	Enforce rules	Reduce startup time
Controls	Runtime, nodes, auto-termination	VM availability
Cost Impact	Avoids misuse	May add idle VM costs
Governance	Compliance tool	Performance tool

---

✅ Conclusion:

• Use Policies for governance and cost control.
• Use Pools to optimize SLA and startup latency.
• Combine both: Policy + Pool-backed clusters = controlled + fast compute.