Same Fires Every Month? A Quarterly Look-Back for Data Teams

Here is a situation most data engineers will recognise. A nightly load fails on a Tuesday, someone patches it, and everyone moves on. Three weeks later a very similar load fails again, on a different table, for a reason that feels oddly familiar. By quarter end you have fixed the same family of problem four times and learned nothing durable. If that sounds like your team, the issue may not be the pipelines. It may be that nobody ever looks back at them as a set.

Think of what follows as a short consultation: we will name the symptoms, find the probable cause and prescribe a remedy that fits into an ordinary working week. The treatment is a quarterly engineering look-back for data engineering work: one calm session to review incidents, releases and tooling changes, and to choose exactly one change for the next three months.

The symptoms: how repeat fires show up

Repeat problems rarely announce themselves as repeats. They look like fresh surprises because each one has a slightly different surface. A schema change upstream, a late-arriving file, a quietly changed definition of a column, a job that now runs longer than its window. Different tickets, different people on call, different fixes.

The giveaway is a feeling rather than a metric: “didn’t we just deal with this?” If that sentence shows up in your team chat more than once a month, you have the main symptom. Others include fixes that live only in one engineer’s memory, runbook pages nobody trusts and a backlog full of items that begin with “make more robust”.

The likely cause: no zoomed-out view

Most data teams are good at the close-up view. They read logs, trace lineage and compare row counts with real skill. What is usually missing is the wide shot. Incidents are closed individually, releases are announced individually and tool changes are made individually, so nobody sees how they relate.

That gap is not laziness. It is structural. Daily work pays you for the next fix, not for the pattern across fixes. Unless someone schedules the wide shot, it does not happen, and the same fires keep returning under new names.

One clarification about the session itself. Do not mistake it for a post-mortem for one dramatic outage, which deserves its own careful write-up soon after the event. Nor is it a performance review, and the moment people suspect it is, the honest answers dry up. Think of it as a cartographer’s afternoon: you are drawing the terrain of the past three months so the next route can avoid the same swamp.

The treatment: a ninety-minute seasonal review

Block ninety minutes at the end of each quarter and invite everyone who touched the pipelines. Before the meeting, one person gathers three lists from your ticket system and deployment history: the incidents of the quarter, the releases that changed pipeline code or models, and the tooling changes, such as a new orchestrator setting or a swapped validation library.

In the room, read the lists aloud and tag every incident with a plain-language cause. Use ordinary words: “upstream renamed a field”, “job outgrew its window”, “test did not cover nulls”. Do not agonise over taxonomy. After tagging, count the tags. You will almost always find that two or three causes account for most of the pain.

A worked example from an imagined quarter

Suppose a team of four ran a reporting warehouse for a retail client. Their look-back showed eleven incidents. Five came down to upstream changes nobody had announced, three were jobs that had grown past their scheduled window, and the rest were one-offs. Looking at tooling, they had adopted a new data-quality tool mid-quarter, but it only covered half of their tables.

The pattern was impossible to miss once it was on one page. The dominant problem was not bugs in their code. It was surprise from upstream. Their chosen change for next quarter was modest and concrete: set up a short, written agreement with the two upstream owners to share planned schema changes a week ahead, and add a contract check on the most fragile tables. If you are weighing validation approaches, a practical guide to expectation-based data validation gives useful background on what those checks can look like.

A useful trick for the tagging step is to ask “what would have had to be true for this not to happen?” For the late-file incidents, the answer might be a clearer delivery promise from the supplier. For the null-handling bug, it might be a test that existed. Each answer points at a different kind of fix, and grouping the answers is often more revealing than grouping the symptoms. It keeps the conversation on conditions and systems rather than on who happened to be holding the pager.

Common snags and how to dodge them

  • Too many fixes chosen. Pick one. If everything is a priority, nothing is.
  • Vague wording. “Improve monitoring” cannot be assigned. “Add a null-rate check to the orders table” can.
  • Skipping the tooling column. New tools often explain incidents better than code does.
  • Letting it drift. If the session slips a quarter, the next one has twice as much to review.

Notice, too, how much the tooling column can quietly explain. A new scheduler default, a changed retry policy or an upgraded library can shift behaviour in ways that no individual release note mentions. When a cluster of incidents starts shortly after a tooling change, give that change the benefit of your suspicion first, and check it before blaming the code that happened to be running at the time.

Borrowing a habit from a different business

Teams in other industries run this rhythm with their own operations. As one example, the site VehicleRentalSystem publishes material about software for companies that rent vehicles, a world where a busy stretch is reviewed before the next begins. The parallel is loose, yet it makes the point: reviewing on a schedule is ordinary practice everywhere that work repeats.

Keeping the transformation layer honest

One more place the look-back pays off is the transformation layer. Models built with tools like dbt accumulate small, sensible changes that nobody revisits, and old logic quietly outlives the reason it was written. During the review, ask which transformations were touched most often and which were never touched at all. Frequent edits suggest a definition that is still being argued over. Never-edited models deserve a gentle audit in case they have gone stale. For a refresher on structuring that layer, see this walkthrough of data transformations with dbt.

Whatever you find, write it as a sentence a newcomer could understand. The aim is to leave behind a record that explains not only what broke but what the team now believes about why it broke.

Checking whether the treatment worked

The follow-up is simple. At the next look-back, read the previous page first and ask whether the chosen change shipped and whether the tagged cause shrank. If the upstream-surprise count fell, the change did its job and you can pick a new target. If it did not, ask why: was the agreement ignored, was the check too narrow, or did a different cause take its place?

Keep the pages. After a year you will have four short documents that show how your team’s problems evolved, which is a remarkably honest picture of data engineering maturity and far more convincing than any slide.

If your team keeps fighting the same fires, resist the urge to buy a new tool first. Take ninety minutes, gather three lists, tag the causes and pick one fix. It is a small, friendly habit, and it is often the difference between a team that reacts and a team that learns.

Related Posts

A DataOps Pipeline Teardown Checklist for a Finished Project

A fifteen-point checklist for reading a finished data transformation project room by room: layout, one traced column, tests, schedule and a written summary.

Read More

Seven Plain Ways to Rewrite Your Data Team’s Docs and Alerts

Cryptic pipeline errors and READMEs slow everyone down. Here are seven friendly rewrites that keep the detail and lose the fog.

Read More

Three Self-Imposed Limits for a Data Pipeline, Compared

A side-by-side look at three deliberate limits for a DataOps pipeline: a column budget, a SQL-only rule and a fixed sample size, and what each one reveals.

Read More

Five Myths About Leaving Data Pipeline Clean-Up for Later

Five common beliefs that keep data teams postponing pipeline housekeeping, and the plain-spoken reasons why small, immediate clean-up wins over the big sweep.

Read More

Why Your Data Pipeline Is Late, Costly or Shaky

A late, pricey data pipeline is often a ranking problem, not a technical one. A diagnosis and a treatment plan built around speed, cost and correctness.

Read More

Why the Same Data Pipeline Keeps Breaking: A Plain Explainer

A plain-language guide to logging pipeline failures, counting them by category and fixing the shared cause once instead of patching each break.

Read More