Memory, Automation or a Card? Guarding Risky Data Changes

Some data changes can be reversed with a quick rerun. Others, such as overwriting a production table, dropping a column or backfilling months of history, leave you restoring from backups at the worst possible hour. A pre-flight checklist for data pipeline changes is the cheapest protection against the second kind. This piece compares three ways teams guard those moments, says where each one fails, and shows why a short written list still earns its place beside the automated tooling.

The scenario throughout is deliberately ordinary: you are about to run a backfill that rewrites a reporting table the finance team reads every morning. The code has been reviewed. The tests pass. You are still one command away from a very unpleasant Tuesday.

Option one: rely on experience and memory

This is the default. A competent engineer has done this ten times, knows the gotchas and runs the job. It works until the day it does not, and the failure has a recognisable shape. The engineer is interrupted halfway through their mental list, or is covering for a colleague who knew the quirks, or runs the job against the wrong environment because two terminal windows looked alike.

Strengths: zero overhead, flexible, fine for low-stakes changes.

Weaknesses: invisible to everyone else. Nobody can tell afterwards which checks were done. The list lives in one head, and heads get tired, distracted and promoted. When the job goes wrong, the post-incident discussion starts with “I thought I had checked that”, which is a sentence worth treating as a signal.

Option two: automated gates in the pipeline

The second approach pushes the checks into code. Row-count assertions, schema comparisons, freshness tests and uniqueness checks run on every build. Tools in this space are mature; the dbt tests and expectation suites that many teams use are good examples, and our own reading list covers a hands-on guide to validating data with expectations if you want the mechanics.

Strengths: consistent, tireless, auditable. A failing test blocks the run whether or not anyone is paying attention.

Weaknesses: automated checks only guard what someone thought to encode. They are excellent at “is this column still unique?” and hopeless at “did I point this at production when I meant staging?” or “is the downstream owner aware a rewrite is happening during their reporting window?” They also tend to be tuned for the routine path. A one-off backfill or a risky migration is exactly the kind of event that sits outside the standard test suite.

Option three: a short written checklist, read before the launch

The third approach is old and unglamorous. Someone writes a handful of questions on a card or in the runbook, and whoever is about to run the risky step reads them in order and answers each one. No tooling needed, though it happily sits beside the tooling.

Strengths: catches human and context errors that tests cannot see: wrong target, wrong window, missing owner, no rollback. Cheap to create and cheap to change. It makes the launch visible to the team.

Weaknesses: it depends on people actually reading it. A list that grows past a page becomes decoration. And it can create false comfort if the items are vague, such as “check data quality”, which nobody can honestly answer in five seconds.

Side-by-side summary

  • What it catches: memory catches whatever you happen to recall; automation catches what was encoded in advance; a checklist catches context mistakes around the launch.
  • Cost to start: memory is free; automation takes engineering time; a checklist takes an afternoon.
  • Visibility: memory leaves no trace; automation leaves logs; a checklist leaves a record of who confirmed what.
  • Failure mode: memory fails under interruption; automation fails on the unexpected; a checklist fails when ignored or bloated.

The verdict: combine the second and third

The honest answer is not “pick one”. Automated gates should carry everything that can be expressed as a machine check, because machines do not get bored. The written checklist should carry what machines cannot see. In practice the two overlap in a useful way: the card includes a line such as “pipeline checks green on this exact commit”, which turns a tool’s result into a human confirmation.

Here is a list that fits the backfill scenario and rarely needs more than a few minutes to run.

  1. Right target. Read the connection string or environment name out loud and match it to the plan. Not the alias; the actual destination.
  2. Snapshot taken. Confirm a copy of the table, or a point you can restore to, exists and was verified to open. A backup you have never touched is an assumption.
  3. Row-count sanity. Compare the number of rows the job expects to write with a quick estimate from the source. A tenfold difference in either direction means stop.
  4. Consumers told. The people or jobs that read this table know a rewrite is coming and roughly when it will finish. Silence here leads to questions about “wrong numbers” at the worst time.
  5. Window clear. No scheduled load, report refresh or month-end task overlaps the run.
  6. Rollback path rehearsed in the head. You can say, in one sentence, how you would undo this and how long it would take.
  7. Owner on hand. Someone who can approve a rollback is reachable for the next while.

Seven items, one page, no jargon. Note what is absent: the list does not repeat the unit tests, and it does not try to review the code again. It guards the launch, not the whole job.

What the card looks like on a real Tuesday

Picture the engineer with the finance backfill ready to go. She reads line one and notices the shell is still pointed at the staging warehouse from yesterday’s test; the job would have run harmlessly in the wrong place and left production untouched, with everyone assuming the numbers had been fixed. Line two sends her to confirm that last night’s snapshot opens. It does. Line three produces a small surprise: the source holds noticeably fewer rows than the job intends to write, because a filter was left over from debugging. She removes it, and the estimate now matches.

Lines four to seven take two minutes. The analytics lead gets a short message, the reporting refresh is nudged an hour later, the undo path is stated aloud and the on-call engineer confirms availability. Then, and only then, the command runs. Nothing about this was clever. Each catch was something a rushed person would plausibly miss, which is the whole argument for writing the list down.

Compare that with the memory-only version of the same afternoon, where the wrong-environment slip and the leftover filter both survive into the run. Neither would have tripped an automated test, because the data was well-formed in both cases. It was simply the wrong data in the wrong place.

How to roll the card out without a revolt

Engineers resist process that feels like bureaucracy. The way around it is to make the first version tiny and to write it from real incidents rather than from imagination. Ask the team for the last couple of data mishaps and turn each one into a line. People tend to adopt a list they recognise as their own history.

Put it where the risky command is run. A link in the runbook, a pinned message in the team channel or a comment at the top of the migration script all work. If it lives in a wiki nobody opens, it does not exist.

Review it every few weeks. Delete lines that never fire. Add the one that just saved someone. If you want a broader view of the transformation layer these changes usually touch, this walkthrough of structuring transformations with dbt is a good companion.

A side note on routines from other fields

People in very different communities converge on the same habit when a step cannot be undone. A blog about cryptocurrency and blockchain, CryptoBlockCoins, publishes coin pages, tutorials and daily logs, and the broader theme across that world is that routine beats improvisation when mistakes are permanent. Data engineers can borrow the instinct without borrowing anything else.

Bottom line

If your team runs risky data changes on memory alone, the cheapest improvement available is a seven-line card. If you already have strong automated tests, the card does not compete with them; it covers the gaps between them. Use tools for what tools are good at and a human list for what only humans notice, and run both before the change that is hard to take back.

Start this week: write the card for your next backfill, run it once, and note which line made someone pause. That single pause is the return on a few minutes of effort.

Related Posts

Priority-First vs Inbox-First: Two Ways to Run a Data Day

A friendly comparison of two ways to start a data engineering day, reacting to the inbox or naming one must-ship outcome first, with a verdict on when each style fits.

Read More

Data Pipeline Failed Again? A Blameless Review FAQ for Data Teams

Questions data engineers ask after a broken pipeline: how to run a blameless review, what to write down, and how to finish with one concrete fix.

Read More

The Weekly DataOps Review: A Beginner’s Primer and Glossary

New to data engineering? This primer explains the weekly DataOps review, defines the key terms and walks through a four-step routine you can start this week.

Read More

Pipeline Fixes: One by One or in a Weekly Batch?

A human comparison of two DataOps habits, fixing each pipeline problem as it lands versus grouping similar fixes into scheduled review sessions.

Read More

Six DataOps Myths Corrected for Working Data Engineers

Six persistent DataOps myths, from the DevOps-with-SQL shortcut to the green-job fallacy, corrected against the Ops Atlas entry.

Read More

From DataOps to MLOps: Skills, Tools and a 90-Day Career Plan

A data engineer’s map of the move into MLOps: which skills carry over, which gaps to close first, and a ninety-day project plan you can finish while employed.

Read More