The following is a composite story, a fictional team assembled from situations that data engineering groups regularly describe. It follows a small analytics platform group that was certain it needed a new data catalog, and it shows what a plain inventory of existing DataOps assets did to that certainty. No real company is depicted.
The request that started it
The platform group served around a dozen analysts and a handful of product teams. In a planning meeting, the head of analytics said what many leaders say: nobody can find anything. Tables had duplicated names, dashboards pointed at unclear sources, and every new hire spent a week asking colleagues which orders table was the real one. The proposed remedy was a dedicated data catalog, with lineage and search, and the group’s lead engineer was asked to shortlist products.
The lead engineer, referred to below only by that title, felt uneasy. The pain was genuine, but the shopping list had appeared before anyone had listed what the team already had. She asked for two weeks before any vendor call and proposed a different first task: take stock of every place where information about data already lived.
Week one: the walk through the estate
She split the work between two engineers and an analyst. They used a shared sheet with five columns: asset, where it lives, who maintains it, what it describes, and when it was last touched. The rule was to record, not to judge.
The first discoveries were modest. The transformation project already contained descriptions on roughly a third of its models, written by whoever built them. A set of schema tests existed for the core tables, with column-level notes that nobody had ever rendered anywhere. The workflow scheduler stored owner tags on jobs, and the tags were mostly accurate. A wiki space titled “data dictionary” held a long page last edited eighteen months earlier.
Then came the surprises. A forgotten script, written for an audit, exported table and column lists nightly into a storage bucket, building a quiet history of every schema change. A BI workspace contained a folder of certified datasets curated by one analyst, with a README that explained which tables fed them. And the team’s cloud bill listed a metadata subscription from a pilot two years earlier, still paid for, still holding a partial scan of the warehouse.
Week two: reading what they owned
The second week was spent reading rather than listing. The analyst opened the old pilot subscription and found lineage for about forty tables, accurate enough to be useful. The nightly schema export, once charted, answered a long-standing argument about when a key column had changed meaning. The model descriptions, when rendered as browsable documentation from the transformation project, turned out to be better than expected. If you work with that tooling, the overview in this walkthrough of dbt transformations in DataOps pipelines shows why generated documentation is such a natural by-product of well-kept models.
The team also noticed what was missing, and this part was just as valuable. Many staging tables had no description at all. No one owned the orphaned dashboards. Retention on the schema export was undefined. These were precise gaps, not a vague feeling that the whole thing was lost.
The conversation that followed
At the next planning meeting the lead did not present a vendor shortlist. She presented the sheet, with three colour codes: assets that were good as they stood, assets that were present but neglected, and holes. The head of analytics had expected a proposal with a price. Instead the room looked at a picture of the group’s own information, and the discussion shifted from “which product” to “who writes the missing descriptions and who reads them”.
They agreed on a small set of actions, none of them involving a purchase:
- Publish the generated documentation internally and make it the first link in the onboarding guide.
- Require a one-line description on any new model before review sign-off.
- Assign an owner to each of the orphaned dashboards or retire them.
- Cancel the forgotten pilot subscription after exporting its lineage.
- Keep the schema export and add a retention rule.
The monitoring angle
One finding deserves its own section. The inventory showed that alerting on pipeline freshness lived in three places: scheduler notifications, a handful of custom scripts, and a dashboard someone had built for one team. They were not wasteful duplicates so much as partial answers to the same question. Before touching any of them, the group read through guidance such as a beginner-oriented guide to monitoring DataOps pipelines to compare their set-up against a sensible outline, then wrote down which alerts they truly relied on. That reading changed no tools; it only clarified what to keep checking.
Looking sideways for a framing
The lead later mentioned that the habit had a precedent outside data work. Operations-minded writers in other industries describe the same reflex; the RobotsOps site, which writes about running robots day to day, shows the same operations mindset aimed at a setting far from analytics tables. She found it a useful reminder that the instinct to list what is running before adding anything is not unique to one discipline.
What the group measured afterwards
Three months later the group did not run a formal study, but a few simple observations stood out. New hires reached their first independent query faster, because the onboarding guide now pointed to rendered documentation rather than to a person. Questions in the team chat about “which table is correct” dropped noticeably. And the platform budget no longer included the dormant subscription. The group still revisits the catalog question once a year; the difference is that the next conversation starts from the sheet instead of from a hunch.
One more detail from the sheet is worth repeating. The “last touched” column did more work than expected. Anything untouched for a year and a half was flagged for a quick read-through, and those items split cleanly into still valid, easily refreshed and safe to retire. Without that single column the old wiki page would have sat there looking authoritative while quietly steering newcomers wrong.
Lessons worth carrying to your own team
- Time-box the inventory. Two weeks of part-time effort was enough for a dozen-person group.
- Record, then judge. Judgement during listing makes people defensive about old work.
- Read the oldest assets too. A forgotten export or pilot can hold more value than a new purchase.
- Classify the gaps. Missing descriptions, missing owners and missing retention are different problems with different fixes.
- Let the sheet lead the decision. The proposal that follows is smaller, cheaper and easier to defend.
The cast is invented; the pattern is not. Data teams that start with an honest list of what they own tend to discover that the missing catalog was partly an unread one. Whether they later buy something or not, they do so knowing exactly which gap it has to fill.