{"id":4757,"date":"2026-11-21T01:35:00","date_gmt":"2026-11-21T01:35:00","guid":{"rendered":"https:\/\/dataopsschool.com\/blog\/?p=4757"},"modified":"2026-11-21T01:35:00","modified_gmt":"2026-11-21T01:35:00","slug":"pipeline-assumptions-log-how-to","status":"publish","type":"post","link":"https:\/\/dataopsschool.com\/blog\/pipeline-assumptions-log-how-to\/","title":{"rendered":"How to Start Writing Down Assumptions for a Data Pipeline"},"content":{"rendered":"<p>A data pipeline that breaks loudly is a nuisance. One that keeps running on a false premise is a liability, because dashboards keep refreshing and nobody looks twice. The fix is dull and effective: writing down assumptions for each pipeline before you build it, then checking the shaky ones. This guide gives you the steps in order, with a worked example you can copy.<\/p>\n<h2>Step 1: Pick one pipeline and one sentence of purpose<\/h2>\n<p>Do not try to document the whole platform. Choose a single job, for example &#8220;load yesterday&#8217;s orders from the app database into the warehouse table the finance team reads.&#8221; Write that sentence at the top of a plain text file, a wiki page or the ticket for the work. Everything else on the page must serve it.<\/p>\n<h2>Step 2: Walk the data from source to consumer and note every hand-off<\/h2>\n<p>Draw the route as a short list: source system, extraction method, landing area, transformation, final table, consumer. At each arrow, ask what the next stage expects. Those expectations are your raw assumptions. Typical ones in data work include:<\/p>\n<ul>\n<li>The source table keeps a column with a reliable &#8220;last updated&#8221; stamp.<\/li>\n<li>Primary keys are unique and never reused.<\/li>\n<li>Timestamps arrive in one time zone, and it is the one you think.<\/li>\n<li>Volume stays within the range the warehouse job was sized for.<\/li>\n<li>Upstream teams announce schema changes before they ship.<\/li>\n<li>Late-arriving records are rare enough to ignore for the daily report.<\/li>\n<\/ul>\n<p>Write them as full sentences. &#8220;Timestamps are UTC&#8221; beats &#8220;UTC.&#8221; If a line makes you wince when you imagine saying it to the finance lead, mark it for the next step.<\/p>\n<h2>Step 3: Rate your confidence honestly<\/h2>\n<p>Next to each line put one of three labels: confirmed, believed or unknown. Confirmed means someone looked at the actual data or a written contract in the last month. Believed means you heard it, or it used to be true. Unknown means you hold no view at all, which is a useful thing to record.<\/p>\n<p>Resist the urge to give everything a comfortable label. A list where every line is &#8220;confirmed&#8221; invites suspicion; a realistic one has a mix.<\/p>\n<h2>Step 4: Attach a cheap test to every believed or unknown line<\/h2>\n<p>For each weak line, write the smallest action that would settle it. A few examples from daily practice:<\/p>\n<ol>\n<li>&#8220;Timestamps are UTC&#8221;: query the minimum and maximum values for one busy day and compare them with a known event.<\/li>\n<li>&#8220;Keys are unique&#8221;: run a count of rows against a count of distinct keys.<\/li>\n<li>&#8220;Upstream announces changes&#8221;: ask the owning team how they currently notify consumers, and note the answer verbatim.<\/li>\n<li>&#8220;Volume stays in range&#8221;: plot the last month of row counts and see where the biggest day sits.<\/li>\n<\/ol>\n<p>Many of these become automated checks later. A validation framework can enforce the unique-key rule every night; for a practical walk through one such approach, see this <a href=\"https:\/\/dataopsschool.com\/blog\/great-expectations-for-data-validation-in-dataops-a-practical-guide\/\">guide to declaring data expectations in a pipeline<\/a>. The point of the written list is that you know which rules deserve to be automated first.<\/p>\n<h2>Step 5: Decide what happens if a line turns out false<\/h2>\n<p>This is the step people skip. For each assumption, add a short &#8220;if wrong&#8221; note: what breaks, who would notice and how soon. &#8220;If keys are reused, the join doubles revenue for that customer and finance would see it at month end.&#8221; Writing it exposes which beliefs deserve an alert and which can wait.<\/p>\n<h2>Step 6: Store the list next to the code<\/h2>\n<p>Put the file in the same repository folder as the pipeline, or attach it to the ticket that created it. Reviewers should see it in the same pull request as the logic. If you build transformations in a modelling tool, the descriptions and tests you attach to each model are a natural home for some of these lines; an overview such as <a href=\"https:\/\/dataopsschool.com\/blog\/master-data-transformations-with-dbt-in-modern-dataops-pipelines\/\">this article on building transformations with dbt<\/a> shows where that metadata lives.<\/p>\n<h2>Step 7: Review on a schedule and after every incident<\/h2>\n<p>Set a recurring reminder, monthly for busy pipelines and quarterly for quiet ones. After any incident, reread the list and ask which line was involved. Either it was already there and marked weak, which tells you to invest in the test, or it was missing, which tells you to add it. Both outcomes improve the list.<\/p>\n<h2>A compact example you can copy<\/h2>\n<p>The order-loading job, reduced to one screen, could read as follows.<\/p>\n<ul>\n<li><strong>Purpose<\/strong>: load yesterday&#8217;s orders for the finance table.<\/li>\n<li><strong>A1, believed<\/strong>: the updated_at column changes on every edit. Test: edit a test order, rerun extraction, see if it moves. If wrong: edits missed for a day or more.<\/li>\n<li><strong>A2, unknown<\/strong>: refunds arrive as negative rows rather than a separate table. Test: ask the application team. If wrong: totals overstate revenue.<\/li>\n<li><strong>A3, confirmed<\/strong>: nightly volume stays below the warehouse job&#8217;s limit. Checked last month.<\/li>\n<\/ul>\n<p>That is three lines. It took ten minutes to write, and it already points at two concrete errands.<\/p>\n<h2>Handling the awkward assumptions about people<\/h2>\n<p>Not every line is technical. &#8220;The analytics team will tell us before they change a report&#8217;s definition of an active customer&#8221; is a belief about habits. Treat it like any other entry: rate it, then test it by asking. A good test here is a short message to the team asking how they handle such changes today, with their reply pasted under the line. Sometimes the answer is a pleasant surprise, and sometimes it reveals that nothing formal exists, which is itself a finding worth recording.<\/p>\n<h2>How many lines is enough?<\/h2>\n<p>For a small, single-purpose job, five to ten lines is plenty. For a pipeline with several sources, group the lines under each source and keep three or four per group. If a section of the list never changes across many reviews and has always been confirmed, move it to a footnote and spend your attention on the lines that keep shifting. The shifting ones are where the surprises hide.<\/p>\n<h2>Where else to see this practised<\/h2>\n<p>Training providers in the operations space often teach teams to keep premises in writing. One such provider is NoOpsSchool, an India-based training website whose own pages describe <a href=\"https:\/\/noopsschool.com\/\">certification offerings, consulting work and services for IT practitioners<\/a>. Treat any outside material as inspiration only; your team&#8217;s own template, kept close to the code, matters more.<\/p>\n<h2>Mistakes to avoid<\/h2>\n<p>Do not confuse assumptions with requirements. A requirement says what the pipeline must do; an assumption says what must already be true for that to be possible. Do not bury the file in a folder nobody opens. And do not let the list grow into an essay. When it passes a page, split it by pipeline stage.<\/p>\n<h2>Your next action<\/h2>\n<p>Choose the pipeline that wakes you up most often, write its purpose sentence, and list six beliefs about its inputs. Mark each as confirmed, believed or unknown, and run the cheapest test on the first unknown. Before you log off, you will either have retired a doubt or caught a genuine defect, and either result justifies the time.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A seven-step walk-through for listing what a data pipeline relies on, rating your confidence, adding cheap tests and keeping the list beside the code.<\/p>\n","protected":false},"author":6,"featured_media":4758,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[814],"tags":[],"class_list":["post-4757","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-dataops"],"_links":{"self":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/4757","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=4757"}],"version-history":[{"count":1,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/4757\/revisions"}],"predecessor-version":[{"id":4859,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/4757\/revisions\/4859"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/media\/4758"}],"wp:attachment":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=4757"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=4757"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=4757"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}