New data engineers tend to learn pipelines by building them, and they tend to learn pipeline failures by living through them. There is a gentler route. Before a data pipeline goes live, you can run a pre-mortem: you picture the pipeline already broken in production, with unhappy analysts and a late-night page, and you write down how it got there. This primer is written for beginners, and a short glossary is included so that unfamiliar jargon is not a barrier.

Why a data pipeline pre-mortem is worth the time

A new pipeline usually arrives with a confident story: sources feed a transformation, the transformation feeds a table, the table feeds a report. Every arrow in that diagram is an assumption. The source will keep its format. The job will finish before people open their dashboards. The numbers will mean the same thing next quarter. A pre-mortem takes each arrow and asks what happens if the assumption fails.

The word “pre” is the whole point. After an outage, everyone becomes wise, and the lessons arrive at maximum cost. Before launch, the same lessons are available for the price of an hour and some sticky notes.

Mini-glossary for the session

Keep these terms handy. They come up in nearly every data failure story.

• Pre-mortem: a planning session where the group announces that the project went wrong and works out the causes in reverse.
• Schema drift: a source changing its columns or types without telling the people who consume it.
• Freshness: how recent the newest data in a table is compared with what consumers expect.
• Backfill: re-running a pipeline over past periods to repair or fill in history.
• Lineage: the trail showing which upstream sources and steps produced a given table.
• Idempotent job: a job that gives the same result if it runs twice, so retries do no harm.
• Data owner: the named person who answers questions about what a dataset means and whether it is trustworthy.

Running the session in four steps

Step 1: Fix the scene

State the launch in one sentence, for example: “From Monday, the sales summary table refreshes nightly from the order system.” Everyone in the room should agree on that sentence before moving on.

Step 2: Declare the failure

Say aloud: “It is three months later. The table is wrong, and the finance team has stopped trusting it. Write the reasons.” Give people five minutes in silence. Beginners often hold back; point out that being specific and slightly gloomy is the assignment.

Step 3: Sort the stories

Group similar notes. For a typical pipeline the clusters look familiar:

• The source changed shape (schema drift) and the job kept running on bad assumptions.
• The job ran late, so the dashboard showed yesterday’s numbers as if they were today’s.
• A retry doubled a day of orders because the job was not idempotent.
• A backfill overwrote corrected history.
• Nobody knew who the data owner was, so questions went unanswered for days.

Step 4: Assign one action each

Choose the three likeliest and most damaging stories. Give each a small action and a name. “Add a check that the newest record is under a day old” is an action. “Be careful with freshness” is not.

Turning stories into checks

Many failure stories convert directly into automated checks. The post on writing validation rules for incoming datasets with an open-source tool shows how rules about columns, ranges and null counts are written down so that a pipeline can stop itself when the data does not fit. Another companion read, why a DataOps approach makes delivery of data more dependable, sets the wider context for these habits.

Not every story becomes a check, though. “Nobody knew who owned the table” is solved by a name in a catalogue entry, and “the backfill overwrote corrected history” is solved by a rule about who may run backfills and how they are reviewed. Choosing the right kind of fix is part of the skill you are learning.

A glance at how security teams think

The mindset behind a pre-mortem is familiar to people who protect software delivery. DevSecOps Now approaches the same topic from the security side, with a blog about locking down CI/CD pipelines, and it is a handy reminder that every automated path from source to production deserves a skeptical look. Data pipelines are automated paths too, so the same skepticism applies to them, whether or not security is your day job.

Common beginner mistakes

• Treating it as a blame session. The exercise is about the plan, not the planner.
• Listing only technical risks. People, handoffs and unclear meanings of columns fail more often than servers.
• Writing vague notes. “Data quality issues” cannot be acted on; “null customer IDs after a source upgrade” can.
• Doing it once and filing it away. Revisit the page when the pipeline changes materially.

A worked miniature

Imagine a small team moving a spreadsheet-based weekly report onto a scheduled job. In the pre-mortem, one participant writes: “The report was published with last week’s figures because the export step silently failed.” That note leads to a freshness check. Another writes: “Two people disagreed about what counted as an active customer, and the table matched neither definition.” That note leads to a short written definition stored beside the table, with a named owner.

Neither fix required new tooling, and both would have been obvious in hindsight. That is the quiet promise of the method.

Where to go next

If you are new to the field, try a pre-mortem on the next small pipeline you build, even a toy one. Keep the page, then compare it with what actually happens in the first month. Over time you will notice which of your guesses were right, and your sense of where data work breaks will sharpen much faster than it would from reading alone.