Let’s be honest: most of us have pushed a change to a data pipeline on a Friday afternoon, crossed our fingers, and then spent Saturday morning explaining to a finance analyst why yesterday’s numbers look odd. A pipeline dry run is the habit that stops that story. In this checklist you will find the steps to rehearse a change on a throwaway copy of your data, so that the broken join, the surprise null or the slow query shows up when nobody is waiting on a report.
First, decide what the rehearsal is for
A dry run is not a full second production. It is a cheap, disposable run that answers a few specific questions: does the job finish, does the output look right, and can we undo it. Write those three questions at the top of a page before you start. If a check on your list does not help answer one of them, leave it out and keep the rehearsal light enough that people will actually do it.
The checklist
Prepare the sandbox
- Create a scratch schema or workspace that cannot write to production tables. Name it clearly so nobody mistakes it for the real thing.
- Copy a representative slice of source data, not just the cleanest slice. Include the odd records: late arrivals, duplicate keys, empty strings, and the oldest format you still receive.
- Use the same credentials pattern and the same scheduler settings as production, minus the permission to publish.
Run the change end to end
- Execute the full chain from ingestion to the final table or dashboard feed, not only the step you edited. Many failures come from the step downstream of the change.
- Time every stage and note it. A transform that runs in seconds on a slice may take an hour on the full set; record the ratio so you can predict the real duration.
- Run it twice. A second run tells you whether the job is safe to repeat, which matters the first time anything restarts in the middle of the night.
Read the output like a skeptic
- Compare row counts before and after at each stage, and explain every difference in a sentence.
- Check a handful of records by hand, from raw to final, using a business example someone in the company would recognise.
- Look at the distributions of a few key columns, not just totals. A total can stay the same while the shape underneath it changes.
Rehearse the awkward moments
- Stop the job halfway and start it again. Does it resume cleanly or double count?
- Feed it a malformed file and see which alert, if any, reaches a human.
- Practise the rollback on the scratch copy: restore the previous version of the table or model and confirm that downstream consumers see the old shape.
Where tests and rehearsals meet
Automated checks make a dry run much stronger, because they turn your skeptical reading into something repeatable. If you are building that habit, the walkthrough on writing validation rules for incoming data with an open-source testing library shows how to encode expectations so they run on every rehearsal. And if your change lives in the modelling layer, the article about building reliable transformations with a SQL-first modelling tool is a good companion for deciding what to run in the scratch schema first.
Keep the rehearsal honest
Teams sometimes turn a dry run into theatre. They run it on perfect data, skip the restart test, and tick the box. Guard against that by assigning someone to try to break it. Their job is to bring the ugliest file they can find, to ask what happens if the source is an hour late, and to look for numbers that moved without an explanation. Rotate that role, and treat a rehearsal that found nothing as slightly suspicious rather than reassuring.
Size matters too. If the slice you copied is tiny, a query plan that looks fine will behave very differently at full scale, so note which results you trust and which need a bigger test. A good compromise is to rehearse correctness on a small slice and timing on a larger sample taken from a recent busy day. Be explicit about that split in your notes so nobody assumes a quick sandbox run guarantees a quick production run.
A little inspiration from the kitchen
It helps to remember that careful cooks do something similar. Before a big family dinner, they make a trial batch of the tricky recipe to learn how long it really takes. The same instinct drives Blendz, an international archive of traditional dishes and ingredients that lets readers contribute their own family recipes, where documented methods are valuable precisely because someone once tested them. Data teams can borrow the attitude: write the method down, then test it before it matters.
What to record afterwards
Once the rehearsal wraps up, jot four items in the team notes: the durations you measured, the surprises you hit, the repairs you made, and any step you left out together with the reason. The note is worth more than it looks. Whoever inherits the pipeline next month can read it and see why the jobs behave the way they do, and the dry run stops being a private ritual and becomes something the whole group owns.
Boiled down, the order is: rehearse on a scratch copy, read the result with suspicion, practise the undo, record the lessons, and only then touch production. That costs perhaps an hour of effort and takes a weekend of worry off the table. It also gives you a story to tell in the next retrospective that is about a problem caught on a Tuesday, not one discovered by a customer.
A quick self-check for this week
Look back at your last three production changes. For each, ask: was there a rehearsal, did anyone time it, and had the undo ever been exercised? Write yes or no beside every answer. A column full of noes is not a failure. It is a to-do list, and its first line is the sandbox you should build before the next release goes out. Pick the change that scares you most, rehearse that one first, and let the result decide how much of this checklist your team really needs.
