Uncategorized

The DataOps Certified Professional (DOCP) is a globally recognized certification designed to validate practical expertise in DataOps principles, tools, and automated data pipeline management. Overview DOCP covers essential capabilities for modern data engineers, analysts, and DevOps practitioners by emphasizing real-world applications of DataOps such as orchestration, integration, data quality, automation, monitoring, and governance. Successful candidates … Read more

What is DataOps? DataOps is an organizational practice (people + process + platforms) that applies DevOps and agile principles to the end-to-end data lifecycle—from ingestion and transformation to testing, observability, governance, and delivery. The goal is reliable, fast, and compliant data/AI delivery through collaboration, automation, and continuous improvement. Independent industry research shows DataOps adoption is … Read more

Databricks Account Console Databricks Lab – Create an Azure Databricks workspace Databricks: Set Up Metastore & Map Azure Storage Account with Access Connector, Enable Unity Catalog Databricks Lab & Excercise – Notebook Databricks Lab & Excercise – Notebook – Unity Catalog → schema → table Databricks Lab – Catalog with External Location, & Storage Credentials … Read more

Introduction In Databricks, identities (users, groups, service principals) live at the account level and can be assigned to one or more workspaces. For Unity Catalog (UC), principals must exist at the account level to be granted data privileges, and Databricks recommends provisioning via SCIM from your IdP (Microsoft Entra ID). (Microsoft Learn) Roles you typically … Read more

Introduction Hard-coding credentials (DB passwords, API tokens, SAS keys, hosts) in notebooks or jobs is risky. In Databricks you store them as secrets inside a secret scope, then read them safely at runtime (not printed in plain text). Databricks supports two scope types: Azure Key Vault-backed and Databricks-backed. (Microsoft Learn) What is a Secret Scope … Read more

Introduction Today we’ll cover four production patterns for Delta Live Tables (DLT): Truncate-Load table as Source for Streaming Tables (with skipChangeCommits) Problem: Your upstream system truncates a Delta table and then inserts new data. A “pure” streaming read sees this as non-append changes and will fail. Fix: When reading a Delta table as a streaming … Read more

Here’s a complete, hands-on tutorial for DLT Data Quality & Expectations — including how to define rules, use warning / fail / drop actions, and monitor a DLT pipeline with SQL for observability. I’ve aligned the flow (and examples) to your timestamped outline so it reads like a video transcript you can paste into docs, … Read more

Introduction Goal: Build a CDC-ready dimension pipeline in Delta Live Tables (DLT) that supports: Core ideas you’ll use What we’ll model How to build SCD1 or SCD2 tables in DLT Pipelines? You’ll create: ✅ DLT Edition: set your pipeline Product edition to Pro (or Advanced) to use apply_changes. Slowly Changing Dimension Type 1 table (SCD1) … Read more

Pass parameters in a DLT pipeline | Generate tables dynamically This hands-on guide shows how to: We’ll build on your earlier DLT pipeline (Orders + Customers → Silver → Gold). If you’re starting fresh, you can still follow along—each step is self-contained. Prereqs (one-time) Introduction (What we’ll build) Use Auto Loader inside DLT 1) Create … Read more

Delta Live Tables (DLT) Internals & Incremental Load Part 2: Add/Modify Columns | Rename Tables | Data Lineage This tutorial walks step by step through advanced Delta Live Tables (DLT) features in Databricks. It is based on the transcript you provided but rewritten into a structured learning guide with examples you can try. Introduction In … Read more