---

1. What is Compute in Databricks?

• Compute = processing power in Databricks.
• In practice, compute means clusters (a group of virtual machines).
• A cluster always has:
  • Driver node → coordinates the job.
  • Worker nodes → perform actual data processing.

---

2. Types of Compute in Databricks

🔹 All-Purpose Compute

• Interactive clusters used for notebooks, SQL queries, or ad-hoc jobs.
• Stay running until manually terminated or auto-terminated.
• Good for:
  • Exploratory data analysis
  • Development
  • Testing

🔹 Job Compute

• Ephemeral clusters created automatically when you run a scheduled job/workflow.
• Start when the job runs → terminate immediately after.
• Good for:
  • Production workloads
  • Automated pipelines
• Saves cost since cluster exists only while the job runs.

🔹 Serverless Compute (coming in preview/GA by region)

• Fully managed, no need to configure cluster size/type.
• Databricks decides resources behind the scenes.

---

3. Access Modes in Compute

Access modes determine how users and Unity Catalog interact with clusters:

• Single User → Cluster tied to one user; good for personal work.
• Shared → Multiple users can attach notebooks; Unity Catalog enabled.
• No Isolation Shared → Legacy option for Hive metastore, not supported by Unity Catalog.

💡 Best practice:

• Use Shared clusters with Unity Catalog for team projects.
• Use Single User clusters for development.

---

4. Cluster Permissions

You can assign access at the cluster level:

• Can Manage → Full rights (edit, delete, restart).
• Can Restart → Start/stop cluster only.
• Can Attach To → Attach notebooks or SQL queries but cannot stop/start or modify.

---

5. Cluster Policies

• A policy = template + restrictions for cluster creation.
• Unrestricted = full freedom (default).
• Predefined Policies:
  • Personal Compute → single node, single user.
  • Shared Compute → multi-node, shared mode.
  • Power User Compute → allows scaling.
  • Legacy Shared → for non-Unity Catalog workloads.
• You can also create custom policies to enforce:
  • Allowed VM types
  • Auto-termination rules
  • Worker/driver size

---

6. Important Cluster Settings

• Databricks Runtime (DBR) → Pre-packaged Spark + Scala + Python + libraries.
  • Always pick the latest LTS (Long-Term Support) version.
• Photon → C++ engine, speeds up Spark SQL jobs, but slightly higher cost.
• Autoscaling → Define min/max workers; cluster grows/shrinks automatically.
• Auto-Termination → Saves cost by shutting cluster after X mins of inactivity.
• VM Types → Choose compute optimized vs memory optimized based on workload.

---

7. Monitoring & Debugging

Clusters provide:

• Event Logs → track autoscaling up/down.
• Spark UI → debug jobs and see DAG execution.
• Metrics tab → monitor CPU/memory usage.
• Driver Logs → check stdout, stderr for errors.

---

8. Key Differences: All Purpose vs Job Compute

Feature	All Purpose Compute	Job Compute
Usage	Interactive (notebooks, SQL)	Scheduled Jobs
Lifecycle	Manual start/stop	Auto-create, auto-kill
Cost Efficiency	Less efficient if left running	More efficient
Best for	Dev & exploration	Production workloads

---

✅ Conclusion:

• Use All Purpose Compute for dev/test.
• Use Job Compute for scheduled production pipelines.
• Always enable auto-termination and policies to save cost.
• Prefer Unity Catalog enabled clusters (Single User / Shared) for governance.

---