Data teams love a rewrite. A slow nightly job becomes a streaming rebuild, a tangle of scripts becomes a tidy orchestrated graph, a warehouse gets migrated because the new one promises speed. Six months later, someone asks the awkward question: is it actually better? Too often nobody can answer, because nobody wrote down how the old pipeline behaved. Here is where that habit sits in DataOps practice today, and why recording a pipeline baseline before any refactor is the cheapest insurance a team can buy.
The state of play: rewrites outrun measurement
Tooling for building pipelines has moved quickly. Transformation frameworks, orchestrators and validation libraries have all matured, and teams adopt them for good reasons. Record-keeping about the starting point is the part that lagged behind. Listen to enough migration post-mortems and one story repeats: the migration plan has a timeline, a budget line and a cutover date, but the section titled “how we will know it worked” is a single vague sentence.
The consequence is predictable. After a big change, opinions replace evidence. Engineers who built the new system remember it as faster. Analysts who lost a familiar table remember it as worse. Both are working from impressions, because the numbers from before were never captured.
What a pipeline baseline actually contains
Forget heavyweight frameworks. A useful baseline is a small, dated record of how the current system behaves in ordinary weeks. It needs four ingredients.
- Run duration. How long the key jobs take from trigger to finish, including the slow days, not only the typical one.
- Data lateness. How old the data in the important tables is when people start using it each morning.
- Failure and rerun history. How often jobs fail, who notices, and how long a manual rerun takes.
- Consumer complaints. The themes in the questions analysts raise: missing rows, stale numbers, confusing columns.
Notice that none of these require new tooling. Most orchestrators already log durations, and complaint themes live in your chat history or ticket queue. The work is gathering them in one place and dating them.
Trend one: validation now comes first, and it supplies baseline data
Many teams have started adding automated data checks early, using libraries such as the one covered in this practical walkthrough of expectation-based validation. A side benefit is easy to miss. Once checks run on a schedule, they produce a history of how often your data breaks its own rules. That history is a ready-made baseline for data quality: run the checks for a few weeks before you change the pipeline, keep the pass and fail record, and compare after.
Trend two: transformation layers are being standardised
Another shift is the move toward version-controlled, tested transformation code. If you are weighing it, the article on building transformations with dbt in a DataOps setting covers the mechanics. From a measurement standpoint, the point is simple: when you move logic from ad hoc scripts into a new layer, time the old scripts first. The temptation is to assume the new layer is faster because it is cleaner. Cleaner and faster are different claims, and only a before number can support the second one.
Trend three: orchestration is getting smarter, and heavier
Modern orchestrators give you dependency graphs, retries and observability out of the box, and that explains the appeal of write-ups like bringing order to scattered data jobs with Dagster, which speak to teams drowning in cron entries. But heavier machinery can also add overhead. A baseline of how long your cron-driven jobs take end to end, and how often they quietly fail, lets you weigh that overhead against the benefit with real figures.
How to capture it in one working week
You do not need a project. A pragmatic sequence looks like this:
- Pick the three to five pipelines that matter most to people outside the data team.
- Export their run logs for the last few ordinary weeks. Mark any week with an incident so it is not mistaken for normal.
- Record duration and finish time for each, then note the typical and the worst case.
- Ask two or three consumers what they complain about most. Write the answers verbatim.
- Put everything in a single dated document and share it with the team before the work begins.
Sharing is the underrated part. Once the snapshot is visible, nobody can quietly redefine “the old system was slow” to suit the story later.
A parallel from outside software
People face the same measurement problem when they plan services across borders. A resource like BestEyeHospitals, a directory built for patients travelling abroad for eye care, sets out country listings and an enquiry process, and it states in its own disclaimer that its content is informational and that nobody should delay professional care because of it. The practical lesson for anyone planning something this complex is the same one data teams keep relearning: write down what you already know before you compare anything new against it.
Pitfalls teams keep repeating
- Baselining the best week. A clean run during a quiet period flatters the old system and makes the new one look worse.
- Changing the measurement along with the system. If the new orchestrator reports durations differently, compare using the same stopwatch, not each tool’s built-in numbers.
- Forgetting the people. Pipeline speed matters less than whether analysts trust the morning data. Keep the complaint themes.
- Not saving raw logs. Summaries are fine for slides; raw logs are what you need when a number looks odd.
Where this is heading
Expect measurement habits to become part of how DataOps work is defined, much as automated tests became part of software delivery. Teams that already keep a dated snapshot before major changes will find it routine; the rest will keep arguing from memory. The effort is small: an afternoon of log exports and a few conversations.
If a rewrite is on your roadmap, add one line to the plan today: “Capture current behaviour before changing anything.” It is the cheapest line item you will write this quarter, and the only one that lets you prove the rest of the plan was worth it.
