Data pipelines fail in predictable ways. A source file arrives late, a schema changes without notice, a join suddenly produces duplicates. What differs between teams is not whether these things happen but how long it takes to respond. This guide shows how to write if-then plans for data pipelines in six steps, so the response to a known failure is decided before the failure appears.
An if-then plan has two halves: a trigger that anyone can observe, and a single action that follows it. “If the nightly extract has not landed by the agreed time, then the dependent jobs are held and the owner is notified.” Short, testable, written down. What follows is deliberately procedural, meant to be read with your own pipelines open next to it.
Step 1: List the failures you have already seen
Open your incident notes, chat history or ticket queue and list the last handful of pipeline problems. Do not brainstorm hypothetical disasters yet. Real history is the best guide to which plans will actually fire.
Group the entries by type: late data, malformed data, wrong volume, failed transformation, failed load. Most teams find that three or four categories cover nearly everything. Those categories are your first candidates.
Step 2: Turn each failure into an observable trigger
A trigger must be something a check can detect without interpretation. Compare these two:
- Weak: “If the data looks wrong, then investigate.”
- Strong: “If the row count of the loaded table differs from the source extract, then the load is marked failed.”
To make triggers observable, tie them to a check you already run or can add cheaply. Validation frameworks help here. A practical walkthrough of validating data with Great Expectations shows how expectations turn vague worries into pass-or-fail results, and the same pass-or-fail quality is exactly what an if-then trigger needs.
Step 3: Choose exactly one response per trigger
This is where precision matters. If the plan says “retry or escalate”, you have written a menu, not a plan. Pick the first action and make it unambiguous.
Reasonable single responses include:
- Hold downstream jobs until the upstream job succeeds.
- Retry once automatically, then stop and notify the owner.
- Load the previous day’s snapshot and tag the output as stale.
- Quarantine the offending records into a review table and continue with the rest.
Notice that each one is cheap, reversible and safe to run unattended. Prefer responses with those three properties. A plan that deletes data or rewrites history on its own is a plan that will eventually cause an incident of its own.
Step 4: Name an owner and an exception rule
Add one line to each plan: who is responsible, and who may override. Without this, plans stall at the moment someone hesitates to apply them. The owner can be a role rather than a person, such as “the on-call data engineer”.
Define the exception in the same breath. “If the owner judges the delay acceptable for a month-end run, then the owner may release the held jobs manually and records the reason.” That sentence keeps the plan from being brittle and keeps the override visible.
Step 5: Write it where the work happens
Store each plan beside the thing it protects. If your orchestrator allows descriptions on jobs, put the plan there. If your alerts support links, point each alert at its plan. A plan that lives in a different tool from the failure is a plan that nobody consults.
Keep the format consistent. A simple template works well:
- Trigger: the observable condition.
- Action: the one response.
- Owner: the role responsible.
- Exception: who may override and what they record.
- Last reviewed: a date field you actually update.
When transformations are involved, linking to the model’s own notes helps. For plans that lean on tests inside the transformation layer, a guide to building transformations with dbt makes a useful companion, since your plans reference tests defined inside the transformation layer, it shows where such tests live and what they are meant to catch.
Step 6: Run it, log it, and prune it
For two weeks, treat every firing as data. Record when a plan triggered, whether the action ran as written, and whether it resolved the situation. Then sort plans into three groups: never fired, fired and worked, fired and failed.
Plans that never fired may have triggers that cannot be observed; test them with a deliberate, harmless fault. Plans that fired and worked are keepers. Plans that fired and failed need a rewrite of the action or the trigger, not a longer explanation.
A short worked example
Suppose a daily sales table is loaded from a partner file. The team writes three plans.
- If the file is absent at the agreed time, then the dependent reports display the previous day’s figures with a stale label and the owner is notified.
- If the file arrives with an unexpected column set, then the load stops and the file is moved to a review folder.
- If the loaded row count falls outside the agreed tolerance compared with the file, then the load is rolled back and the owner is paged.
Each is three lines long. Each can be tested by a deliberate bad file in a staging area. Together they remove the three most common arguments the team used to have during incidents.
Keeping the plans useful over time
Systems evolve, so schedule a short review each month. Remove plans for retired jobs, tighten tolerances that proved too loose, and update owners after any rota change. A stale plan is more dangerous than no plan, because it creates confidence the system has not earned.
Avoid the opposite error as well. If your list grows past what a new engineer can read in ten minutes, merge or retire entries. The aim is a compact reference that people trust.
Staying connected while you are away
Data engineers on call sometimes travel, and connectivity becomes part of the plan. Settle beforehand how you would cope if your usual connection failed abroad. One resource some travellers use is a site that lays out SIM and eSIM choices for travel, with country, provider, plan and guide sections to browse before the trip, so the choice does not have to be made under time pressure.
Summary of the six steps
List real failures, convert each into an observable trigger, choose one cheap and reversible response, name an owner with an exception rule, store the plan where the work happens, and then run and prune it. Run the sequence once on your most troublesome pipeline, then copy the resulting template across the others. The reward is not elegance; it is shorter, calmer incidents.