Let’s talk about the data check you meant to run. You know the one: the quick look at row counts and null rates that was going to happen “after the next load.” Then a dashboard broke, a stakeholder pinged, and the check slid to tomorrow, where it met the same fate. If your data checks keep getting skipped, the cause is probably not discipline. It is that nothing in your day tells you when to do them.
What follows works through the problem in stages and finishes with a remedy that costs almost nothing: hook one tiny check onto a pipeline event that already happens every day.
Symptom: the check exists, but only on paper
The pattern is familiar to many data engineers. There is a wiki page titled something like “Pre-release validation,” a ticket template with a checkbox, maybe a long-promised notebook. Everyone agrees it matters. Few people run it consistently.
Ask what triggers it and you hear answers like “whoever is on call should” or “before big releases.” Those are intentions, not triggers. An intention lives in someone’s head and competes with whatever is loudest today. A trigger is an event, and events do not forget.
Diagnosis: three common causes
- The check is too big to start. A full validation suite takes twenty minutes and a clear head. On a rushed morning the sensible-looking choice is to skip it.
- The check has no fixed place in the day. It lives in the vague territory of “when there is time,” and there is never time.
- The check depends on one person remembering. When that person is out, it quietly stops.
All three come from the same root. The action has not been welded to anything. Notice that none of them says “the team does not care about data quality.” Caring is not the missing ingredient. Placement is.
The fix: after X, I do Y
Here is the whole technique in a sentence. Pick something you already do reliably, and decide that immediately after it, you will do one small new thing. The existing event becomes the reminder. Your brain no longer needs to hold the plan.
For data work, you have plenty of reliable events to choose from:
- the nightly load finishing and the success message arriving
- the first coffee and the opening of the orchestrator’s run list
- merging a change to a transformation model
- closing the morning stand-up
Choose the one that happens daily without exceptions. If your loads occasionally fail to run, do not anchor to them. If your stand-up is regularly cancelled, skip that one. You want a cue that behaves like a clock but announces itself like an alarm.
Make the check tiny on purpose
This part surprises people. The check you attach should be almost laughably small, at least at first. Here is a concrete version:
After the nightly load finishes, I open the main fact table’s summary and set today’s row total beside the previous day’s. That is all. If the difference looks strange, I post a line in the team channel and stop there.
It takes under a minute, and it catches a surprising amount: a partition that silently failed to arrive, a join that doubled rows, a source that went quiet. The reason it stays alive is that it is small enough to survive your worst day. Once the fusion between cue and action is solid, you can extend it. Add a null-rate glance on one critical column. Later, add a freshness check. Do not add all three at once.
If you want to turn the glance into repeatable rules, tools built for expressing expectations about data can take over the mechanical part. A walkthrough such as this practical piece on validating data with expectation suites shows how a manual look can grow into an automated test later. Start by hand, then automate what proves its worth.
Choosing which check goes first
If you are unsure which glance to start with, look backwards. Think of the last two data incidents that reached a stakeholder. What would a person have needed to see, a few minutes after the load, to notice them early? For many teams the answer is blunt: a count that dropped, a date column that stopped advancing, or a key that suddenly had blanks. Pick the one that would have caught the most painful incident and give that single check the cue. The others can wait their turn, each with its own anchor once the first one is automatic.
What a week of it looks like
On Monday, the load finishes, you open the summary, compare counts, and close the tab. Tuesday, same. On Wednesday the row count is down by a fifth. You have not been hunting for trouble; the habit simply put your eyes in the right place at the right moment. Ten minutes of digging finds an upstream export that ended early.
Without the habit, that gap would have been found by a business user on Friday, looking at a report that “seems low.” With it, the story is a short message and a re-run before anyone downstream notices. That is the entire payoff: earlier information, bought with a minute a day.
Common troubleshooting
“I forgot anyway.” Then the cue is not firm enough. Change the anchor to something you cannot avoid, like opening the orchestrator or merging a change.
“It feels pointless when nothing is wrong.” That is normal. A check that usually passes is doing its job. The value is in the rare morning when it does not.
“I am the only one doing it.” Share the sentence rather than the task. Write “after the load, glance at the row count” in the team’s README and let each person attach it to their own cue, or rotate who looks.
“It grew into a monster.” Cut it back. A check that has become a thirty-minute ritual will be skipped. Move the heavy parts into scheduled jobs and keep the human glance short.
One more thing worth saying: a skipped day is not a failure of the method. If the load runs late or the cue misfires, simply do the glance at the next opportunity and carry on. Missing once costs very little, whereas treating the miss as proof that the habit does not work tends to end it for good.
Where the heavier tooling fits
None of this replaces proper testing in your transformation layer. Model-level tests, schema contracts and alerting belong in the pipeline itself. The daily glance is a complement: it catches the things nobody thought to write a test for. If you are working on those layers, a guide on building dependable transformations with dbt covers the automated side, and the two approaches fit together nicely.
A tiny aside about everyday decisions
The same logic is useful outside the pipeline. Plenty of people hang a quick pre-purchase look onto the moment they add something to a basket. One place that people in India might use for that look is iReviewed, a site that says in its own words that it covers company details and reviews of everyday products and services for Indian households. The destination is incidental; the point is that a fixed trigger beats a promise to remember.
Your two-week experiment
Here is a simple plan. First, write down the one event in your data day that never fails. Second, define a check small enough to finish in sixty seconds. Third, put both in one sentence and keep it visible near your desk: after the nightly load, I compare yesterday’s and today’s row counts for the main fact table. Run it for two weeks and note how many days it actually happened.
If the answer is most days, you have a habit. If it is fewer, change the cue and try again. Either way you will know much more than you did when the check lived on a wiki page, waiting for someone to remember it.