Symptom: a data pipeline breaks on a Monday morning, someone opens the transformation code, and finds a filter that drops every row with a null region. Nobody on the current team knows why it is there. Removing it might fix the report. It might also corrupt a downstream table that finance reads. So the engineer does what engineers do when they lack context: they leave it alone, add a workaround on top, and the pile of mystery grows. If this sounds familiar, the diagnosis below is for you, and the fix is a small, boring decision journal for your data stack.
The symptoms worth recognising
Before prescribing anything, confirm you actually have the problem. In data engineering the pattern shows up in a handful of ways:
- Odd thresholds in data quality checks that nobody can justify, such as a row-count tolerance that looks arbitrary.
- Tables with duplicate-looking siblings, where nobody knows which one is the source for dashboards.
- Orchestration schedules that run at strange hours for reasons lost to time.
- Repeated debates about choices that were already settled last year, just under different names.
- New hires asking “why do we do it this way?” and receiving shrugs.
If you tick two or more boxes, the culprit is the same: the reasoning behind decisions was never recorded, so it left with the person who made them.
Root cause: decisions without a trail
Code captures what a system does. Commit messages sometimes capture what changed. Almost nothing captures why one option beat another, and what the author expected to happen. In data work this hurts more than usual, because data decisions are entangled with business meaning. A cutoff on late-arriving records is partly a technical choice and partly a statement about how long the business is willing to wait for a number to settle. When the statement is lost, the code looks arbitrary, and arbitrary code is dangerous to touch.
The fix: a short decision journal
A decision journal is one page per meaningful choice, written at the time of the choice, with a fixed set of fields. No tooling is required beyond a folder of text files next to your pipeline repository. Keep the fields tight:
- Context: the problem and any constraint, such as a deadline or a source system limitation.
- Options: the realistic alternatives, including doing nothing.
- Decision and reason: what you picked and the main argument for it.
- Expected outcome: what you think will happen, written so it could be checked, ideally with a date.
- Review: filled in later with what actually happened.
The “expected outcome” is what turns a note into a learning tool. Without it you can only describe history. With it you can calibrate how well your team predicts the consequences of its own choices.
Worked example: where to put the validation checks
A team has two places they could enforce data quality. One is inside the transformation layer, using the testing features of a tool such as dbt. The other is a separate validation step run against tables after they land, using a framework like Great Expectations. Both are reasonable. The journal entry settles the argument for future readers.
The context: a stakeholder report broke twice in a quarter because an upstream feed changed a column type. The options: test inside the transformation project only; add a standalone validation suite; do both. The decision: begin with transformation tests for the key tables, and add a standalone suite only for the three tables that cross team boundaries. The reason: fewer moving parts to maintain, and the cross-team tables are where surprises originate. The expectation: the same class of failure will be caught before reaching the report, within one quarter.
At the review, the team finds that the type-change failure was caught, but a different problem slipped through: a feed that arrived complete in row count but with stale values. The journal shows that stale values were never named as a risk. The next entry adds freshness checks and records why. Over time the log becomes a map of how the team’s understanding of its own system grew. Readers wanting the mechanics of the tools involved can look at the hands-on article about validating datasets with Great Expectations and the walkthrough on building transformations in dbt-based pipelines; the journal itself stays tool-agnostic.
Diagnosing why your first attempt may fail
Most teams who try a decision log stall within a month. The common causes, and the repair for each:
- Entries are too long. Cap them at a screen. If you need more, link to a design document.
- Nobody reads them. Add a link to the journal in the pull request template, so reviewers see the reasoning next to the change.
- The review step never happens. Put review dates in the team calendar. A recurring monthly slot works.
- Entries are rewritten after the fact. Add outcomes beneath the original text, never over it. The original expectation must survive.
- Everything gets logged. Only record choices that are hard to reverse or that others depend on.
A brief aside from outside the data world
The habit is useful far beyond pipelines. Consumer sites like MotoShareGo, a marketplace where people can rent or share motorbikes and scooters, rely on account sign-in and stored choices in much the same way our systems rely on stored configuration; in both cases, knowing why a setting exists beats guessing.
Checking that the journal pays off
Skip the dashboard. A few plain signals show whether the journal earns its keep. Do onboarding questions about “why” now get answered by pointing at a page? Are the same debates reopening less often? Does anyone cite an old entry when proposing a change? Is the proportion of entries with a completed review rising? Mostly yes after a quarter? Keep going. If not, shrink the template further rather than abandoning it.
Rolling it out in one week
Day one: create the folder and a one-page template. Day two: write entries for the three most puzzling existing choices, marking them as reconstructed so nobody mistakes them for contemporaneous records. Day three: add the link to your review template. Day four: agree a monthly slot. Day five: write a real entry for a decision you are about to make. After that, the habit sustains itself as long as each entry stays small.
A note on tone
Write entries the way you would brief a sensible colleague over coffee: direct, a little informal, free of blame. The aim is to capture what was known on the day, including the uncertainty. Phrases such as “we are guessing that” and “we could not verify” are valuable, because they tell a future reader which parts of the reasoning were solid and which were hopeful. Teams that write confident-sounding entries lose that signal and end up trusting old decisions more than they deserve.
Also resist the urge to defend choices in the text. If the decision was made under time pressure, say so. A candid note about a rushed call is more helpful at review time than a polished justification, and it makes it easier to revisit the choice once the pressure has gone.
Bottom line
The problem is lost reasoning, not lost code. The remedy is a brief written record of choices, expectations and outcomes, kept next to the pipelines it describes. It costs minutes per decision and saves hours of archaeology later. Start with the next choice you make, write five honest lines, and schedule the day you will check how it went.