Data engineering teams tend to say yes to everything. A new source to ingest, a metric definition to revisit, a backfill someone asked for last week, a model refactor that is nearly done. Weeks later the team has a long list of items that are all “in progress” and a stakeholder wondering why nothing has landed. Limiting work in progress is the standard answer in flow-based delivery, yet data teams often push back on it with objections that seem reasonable but do not survive scrutiny. Below are six of those myths, each corrected in order.
Myth 1: “More parallel streams means more output for the data team”
It feels logical that three engineers each carrying four tasks will produce twelve things. In practice, each person spends part of every day reloading context: which table was I joining, which stakeholder asked for that cut, which test was failing. The more items that sit open, the longer each one takes to reach production, and the first deliverable arrives later than it would have under a lower limit.
The correction is simple. Output is measured in finished, deployed and used changes, not in started ones. A team that completes one pipeline change before pulling the next delivers its first result sooner and tends to deliver the same total or more over a month.
Myth 2: “Data work is too unpredictable to cap”
Data tasks do carry uncertainty: a source schema changes, a vendor file arrives late, a quality check reveals a surprise. Teams conclude that a cap would only produce idle time. The reverse is more common. A cap on work in progress does not require predicting duration; it only counts how many items are open.
When an item is blocked, you mark it blocked, name what it is waiting for, and use the freed slot to finish something that is not blocked. The limit becomes a prompt to resolve blockers rather than to hide behind them.
Myth 3: “Half-built pipelines are fine because they are almost done”
Almost done is the most expensive state in data engineering. A half-built pipeline holds a branch that drifts away from the main code, a staging table nobody remembers to clean up, and tests that were written against assumptions that later changed. Each of these has to be rediscovered when work resumes.
Compare that with the discipline of finishing a transformation layer before opening the next. Articles such as the one on building reliable data transformations with dbt in DataOps pipelines describe modular, testable models; that modularity is what makes a clean hand-off from “in progress” to “done” possible. A model that is merged, tested and documented is a finished unit. A model that is nearly there is a liability.
Myth 4: “Quality checks can wait until all the pipelines are built”
Deferring validation looks efficient because it delays a slow step. The result is a pile of unvalidated work, any piece of which might contain a defect, and a late discovery phase in which fixes are tangled together. Validation belongs inside the definition of finished for every single item.
The practical guide to using Great Expectations to validate data in DataOps work is a good illustration of treating checks as a normal part of building. When a limit on open items exists, each item has to pass its checks before it leaves the active column, so defects are found while the author still remembers the details.
Myth 5: “A work-in-progress limit is a management tool for controlling people”
The limit is sometimes read as a way for managers to restrict engineers. In a healthy team it works the other way. It protects engineers from an inbox of competing requests by giving them a visible, agreed reason to say “that is next, once the current items are done”. The team sets the number together and adjusts it using evidence from how work actually flows.
If an engineer is regularly pulled into five things at once, the cap provides cover for an honest conversation about priorities. It moves the argument from personal willingness to a shared rule that everyone can see on the board.
Myth 6: “Finishing first only matters for software teams”
Finishing before starting is a general habit, not a software one. Even outside engineering, people exchange ideas about managing their time and energy in online communities. AskDoctorLive, for instance, is described as a wellness-themed website with discussion forums, blog posts and hospital and doctor directory pages, which is one of many places where everyday routines are talked through; the underlying advice to close one loop before opening the next transfers well to a data backlog.
What a sensible limit looks like in practice
Start with a number close to the size of the team, perhaps one fewer, and place it above the “in progress” column. Define the columns precisely: ready, in progress, in review, validated, deployed. An item counts against the limit until it is validated, not merely until the code is written. Then review the evidence after two or three weeks.
- Time from start to deployment for each item.
- Number of items blocked and for how long.
- Number of times the limit was exceeded and why.
- Rework found after an item was called done.
These measurements are enough to tell you whether to tighten, loosen or keep the cap. They also give you something concrete to show stakeholders who want to know why the team is, for the moment, saying “not yet” to new requests.
A worked example
Consider a team of four with fourteen open items: three backfills, two new sources, four metric changes, three refactors and two dashboards fixes. Nothing has been released in three weeks. The team chooses a limit of four, moves ten items to a ready list and finishes the four closest to deployment first. Within a week two backfills and two metric changes are live, the team has a clearer view of the real blockers, and the next four items are chosen with that knowledge. Nothing about the team’s skill changed; only the number of simultaneous obligations did.
The six myths share a root: confusing activity with progress. Limiting work in progress replaces that confusion with a plain measure, which is how many things reached the end this week.
