Data pipelines rarely fail because of one dramatic mistake. They get harder to run through accumulated small irritations: a column renamed without a heads-up, a backfill that needs three manual steps, an alert that fires at the wrong hour. People adapt around these irritations, which is exactly why they stay invisible. A simple question brings them back into view: what would make this easier?
Below are eight places in a data engineering team where that question pays off. For each one you will find who to ask, what evidence to check so the answer is grounded in more than opinion, and an example of a small change that tends to follow. The list is meant as a menu, not a sequence. Pick two or three that match your current pain.
1. Ask whoever carries the pager
On-call engineers know which failures repeat. Ask them what would make the next incident easier to handle, then check the evidence: skim the last month of incident notes and tally how many began with “could not find where the job logs live” or “unsure who owns this table”. The tally turns a vague complaint into a ranking.
Typical small fix: a one-line owner field in the job definition, so the alert message itself names the team to contact.
2. Ask the newest person on the team
Someone in their first fortnight sees what everyone else has forgotten. Ask what would have made day three easier. For evidence, record the order in which they struggled: environment setup, access requests, understanding naming conventions, finding sample data.
Typical small fix: a short checklist stored beside the repository’s readme, updated by each newcomer before they finish their first month. The person who needed it last writes the next edit.
3. Ask the people who request data
Analysts and product colleagues are your users. Ask what would make requesting or locating a dataset easier. Evidence lives in your request channel: count how often the same question about definitions repeats, and how long answers take on average. You do not need exact numbers to see a pattern; a rough weekly count works.
Typical small fix: a request form with three required fields, namely the purpose, the deadline and who will use it, which cuts the clarifying back-and-forth.
4. Ask whoever reviews schema changes
Reviewers of migrations and model changes often carry hidden effort. Ask what would make a review quicker to complete well. Look at past review threads for repeated comments such as “which downstream tables does this touch?” That repeated line is a request for information the author could attach up front.
Typical small fix: a pull request section listing affected downstream models, generated automatically from the project’s lineage where possible. A transformation framework such as the one covered in this walkthrough of data transformations with dbt makes that listing far easier to produce.
5. Ask about data quality checks
Checks that nobody trusts get ignored; checks that fire too often get muted. Ask the team what would make the validation step easier to live with. Evidence: look at which checks failed most in recent weeks and how many of those failures led to action. A check that fails often with no action is a candidate for tuning or removal.
Typical small fix: moving one noisy rule from “block the run” to “warn and record”, while keeping a visible log. If you want a primer on validation rules, a hands-on article about data validation using Great Expectations is a useful companion.
6. Ask who runs backfills
Backfills tend to be remembered as ordeals. Ask the person who last ran one what would have made it easier, and write down each manual step in order. Evidence is the step list itself: if it contains many hand-typed commands, there is room to simplify.
Typical small fix: a parameterised wrapper that accepts a start and end date, plus a dry-run mode that prints what it would do before anything is touched.
7. Ask the people who read the documentation
Documentation exists for readers, so ask them. What would make finding the right page easier? For evidence, watch where people ask in chat even though a page exists; those are the places where the page is hard to find or hard to trust.
Typical small fix: a single index page grouped by task, such as “load a new source” or “rerun a failed job”, rather than by system. Retire any page nobody will vouch for.
8. Ask yourself, at the end of the week
The simplest place is a personal one. On Friday, jot down the single task that felt harder than it should have, and ask what would have made it easier. Evidence accumulates in your own notes; after a month, the same entry may appear three times, and that is your signal.
Typical small fix: whatever small change you can make in under an hour, such as a shell alias or a saved query, because those get done while larger ideas wait for a quarter that never arrives.
How to weigh the answers
Collecting answers is the easy half. Weighing them takes a little discipline. Here is a straightforward way to do it without it growing into a side project.
- Write each answer as one sentence on a shared page.
- Next to each, note the evidence you found: incident notes, repeated chat questions, review comments, your own log.
- Mark how many different people raised the same thing.
- Choose the item with the strongest evidence and the smallest fix.
- Decide in advance how you will tell whether it helped, such as fewer repeated questions in the channel next month.
This keeps the exercise analytical. You are not polling for popularity; you are looking for friction that shows up in more than one source.
A short illustration
Picture a team that loads partner files every night. The on-call rotation tally showed that nearly every page began with the same confusion: which of the many staging tables had been refreshed. Nobody had complained, because the tables were all listed in a wiki. The question “what would make this easier?” drew one reply from the newest hire: “I would like the job to tell me what it just loaded.”
The fix took an afternoon. Each run now writes a single summary line, source name, row count and finish time, to a small status table that the alert links to. Pages that used to take twenty minutes of searching now start with the answer. The team did not buy anything or redesign anything. They simply listened to the person who had not yet learned to ignore the problem.
Two lessons stand out. First, the evidence mattered: the tally gave the team confidence that this was the most common snag rather than the most recent one. Second, the change was small enough to ship before anyone lost interest. Large ideas need champions and budgets; small ones only need an owner and an afternoon.
A note on scope
The question works because it is narrow and kind. It does not ask people to criticise colleagues or to propose a grand redesign. It asks about ordinary, local obstacles. That framing makes honest answers more likely, and it keeps the resulting changes small enough to ship and, if needed, reverse.
It also travels beyond engineering. Plenty of informational websites use a similar mindset when shaping pages for first-time readers. Urology Hospitals, for instance, describes itself as a general information site on urology with a non-ranked hospital and doctor directory, and it says it is not a medical service. Whether you maintain a public site or a private pipeline, the people who use it daily can usually tell you what slows them down.
What to do next
Choose two items from the list, ask the question this week, and gather the matching evidence. Share the findings with the team in a few lines, pick one fix, and set a date to look again. Data work rewards teams that remove small obstacles steadily, because every obstacle you remove frees time for the problems that actually need deep thought.