What follows is a made-up team, assembled from habits common to many data groups, not an account of one real firm. A mid-sized analytics team had a habit that many data groups share: pipelines were built privately, tested privately, and announced only once the numbers looked right. Then a single quarter of surprises pushed them to try the opposite. What follows is the sequence of events, the evidence the team gathered, and the lessons that seem to carry over to other data engineering groups.
The starting point: tidy results, untidy process
The team served a handful of internal consumers: finance, marketing and an operations group. Each consumer received a weekly set of tables. The analysts and engineers behind those tables worked in separate branches, and the only shared artefact was the finished dataset. Questions arrived by message, and answers took a day or two because each reply required someone to reconstruct what had been done.
Three recurring problems were visible in hindsight.
- Two engineers had built near-duplicate cleaning steps for the same source, with slightly different rules for null handling.
- A downstream consumer built a report on a column that was about to be renamed, and learned of the change only when the report broke.
- A quality problem sat undetected for several days because the person who noticed it assumed someone else already knew.
The incident that changed the conversation
A monthly revenue summary went out with a figure that finance immediately questioned. Tracing it took most of two days. The cause turned out to be unglamorous: a join key had been changed in a branch that nobody else had seen, and the change merged on a Friday afternoon. Nothing about the fix was hard. What hurt was that the information existed, but only in one person’s head and one private branch.
In the retrospective, the analyst who led the investigation made an observation that the team wrote on the wall: “The problem was not that we changed something. The problem was that nobody could see us changing it.”
The experiment: make the work visible
The team agreed to a four-week trial with three rules, deliberately small.
- A shared status board. Every pipeline change gets a card when work starts, with a one-line purpose and a state: planned, in progress, in review, live.
- Draft change requests from day one. The first push opens a draft request describing what is intended, even if the code is rough.
- A plain-language change note for consumers. Anything touching a column, key or refresh time is posted to a page the consumers can read before it ships.
Nobody was asked to polish anything. The instruction was only to show the work earlier than felt comfortable, and to say honestly how finished it was. A card marked “rough idea” was perfectly acceptable.
What the team measured
Because the group prided itself on being analytical, they tracked a few simple counts before and after, using their own ticket history. They were careful not to over-read a four-week window, so the figures below are described in rough terms only.
Duplicate effort fell noticeably: in the first month, two cards were spotted as overlapping within a day of being created, and one engineer redirected to a different task. Surprise breakages reported by consumers dropped to none during the trial, compared with a few in the previous quarter. Time to answer a consumer’s “why did this number change?” shrank, because the answer was already on the board.
The team also noted what did not improve. Review turnaround stayed about the same, and one person admitted that seeing every half-finished idea on the board made it harder to concentrate. A caveat worth stating: a four-week sample from one team is a signal, not proof.
How validation fit into the picture
Once changes were visible, quality checks became easier to discuss in the open as well. The team had been experimenting with declarative expectations on their tables, along the lines described in a practical walk-through of writing validation rules with Great Expectations. With the board in place, a new rule could be proposed as a card, discussed by anyone affected, and added with everyone aware of what it would flag.
Transformations got the same treatment. The group kept its modelling layer in a shared repository, and a teammate who was new to the tooling learned the conventions by reading open change requests, a path that pairs well with the approach in building clean transformations in dbt.
Where outside ideas came from
An unexpected benefit appeared in the fourth week. A finance consumer, reading the change notes, pointed out that a proposed renaming would collide with a spreadsheet macro they maintained. The engineers had no way of knowing that. A one-line comment on a draft saved a broken workflow and a round of apologies.
The team discussed this at length. Their conclusion was that visible work invites help from people who hold context you lack, and that those people only speak up when they can see what is happening. The same logic is easy to spot in everyday life. Take a site built for finding local products and services, where nearby shops put their availability on a board for shoppers to read: people act on what they can look at.
The objections they heard, and what happened
“It exposes unfinished thinking.” It did, and a few people disliked that at first. The group added a simple convention: a draft label means “not ready for judgement”. After two weeks the discomfort faded.
“The board will become a second job.” It nearly did. They trimmed the card to a title, a purpose and a state, and agreed that moving a card takes less effort than a status message.
“Consumers will complain about everything they see.” In practice, consumers asked sharper questions, which was the goal, and complaints about surprises disappeared.
Lessons for other data teams
- Treat visibility as a property of the process, not a report written afterwards.
- Keep the artefacts tiny: a card, a purpose, a state.
- Let consumers read change notes before release; they hold context you do not.
- Keep the trial short, record plainly what improved and what did not, and decide with evidence.
- Expect some friction about unfinished work being seen, and agree on a label for it.
Once the four weeks were over, the team simply carried on with it. The retrospective note from the next quarter records only one change: the board moved to the front page of their internal wiki, because people kept asking where to find it.