Your nightly pipeline stalls, the sales dashboard shows yesterday’s numbers at nine in the morning, and the chat fills up with “is it just me?” That moment is where IncidentOps lives for data teams. These are the questions we hear most about it, with answers kept chatty and jargon-light. Grab a drink and read in any order.

What does IncidentOps mean for a data team?

It is the habit of handling unplanned disruptions in an organised way: noticing, declaring, assigning roles, telling people what is going on, getting things working again and learning afterwards. The Ops Atlas write-up on IncidentOps openly says the word is only an informal synonym for incident management rather than a standalone field. For data folks the incident might be a broken load job, a table that quietly stopped refreshing, or a report that suddenly disagrees with finance.

Is every failed job an incident?

No, and the Atlas is explicit about that. Not every alert deserves the full treatment, and planned maintenance windows are not incidents at all. A failing job that retries and recovers by itself is just noise you may want to tune. It becomes an incident when it crosses a threshold you agreed on in advance, for instance when downstream consumers are affected, when another team has to be pulled in, or when nobody has cracked it after an hour or so.

Who should be in charge when a pipeline goes sideways?

Not necessarily the most experienced engineer, and definitely not the person who knows the pipeline best, because that person is usually needed for hands-on diagnosis. The model described in the Atlas separates the coordinator from the people changing things. The coordinator keeps the picture, asks what has been tried and decides what happens next. Hands-on specialists, often just two or three, are the only ones touching production data and code. A third person posts updates so that analysts, managers and partner teams are not pinging engineers directly.

The roles are hats rather than job titles. A newer analytics engineer can wear the coordinator hat for a small incident, and that is a sensible way to practise.

Do we have to find the cause before we fix anything?

Thankfully not. A very data-flavoured example: a transformation shipped last night, the numbers look strange, and you suspect it. You do not have to prove that theory before rolling back to the previous version or pausing the downstream refresh. The Atlas lists quick, generic mitigations such as reverting a release, draining traffic, turning off a new feature, failing over to another system or shedding load. They need you to know where the problem sits, not why it exists. Once the data is trustworthy again you can run the post-mortem at leisure.

What should we tell stakeholders, and how often?

Short and regular beats long and rare. The suggested rhythm is an opening note inside five minutes of kickoff, a follow-up shortly after saying how wide the problem is, and fresh updates no less than every twenty minutes for the first two hours. At the end, post a closing message that states clearly whether any data was lost or needs reprocessing. For data teams that last sentence is the one executives care about most.

If you use a hosted status page, the typical labels are investigating, identified, monitoring and resolved. Internal consumers love a page like that because they can check it instead of asking.

How do we keep alerts from burying us?

The Atlas cites a 2026 community survey of about thirteen hundred self-selected respondents, in which roughly three in ten named alert fatigue as a pain point and about one in six mentioned painful coordination across teams. It is a self-selected sample, so treat the figures as a flavour rather than a universal truth. The practical advice that follows is to measure pager load over a trailing window of about three weeks and to review which alerts led to action. If you want a deeper look at designing checks for the data itself, our walkthrough of validating tables with an expectations library is a good companion.

What if the paging path breaks along with the system?

This one trips up even experienced teams. The Atlas states two architectural rules. First, the route that wakes people and the page that informs users must not fail together with the systems they describe. Second, the incident record should be written once and reused, rather than retyped for chat, tickets and summaries. As a cautionary tale it points to a big 2017 object-storage failure where the vendor’s health console leaned on the broken service itself, so honest public reporting lagged.

For a data team, translate that as: do not host your incident notes in the same warehouse that is down, and do not rely on a dashboard to announce that the dashboards are broken.

How do we run a review people actually want to attend?

Name the owner before the incident call ends. Schedule the review soon after. One paging vendor’s public process sets a deadline of three calendar days for its top severity and five working days for the tier below. Ask someone who was not in the thick of it to facilitate. Keep the list of follow-ups short, with one owner and one due date for each, and avoid actions that are “mostly mitigative”, which means they patch the symptom without removing what caused it.

Another gentle rule is to stop digging at system conditions rather than individual blame. If a column rename took down a model, the useful question is why nothing checked the contract between the two teams. For a practical look at spotting such problems early, try our explainer on catching faulty data before it reaches production.

What numbers should we report?

Resist the single average. The Atlas notes that “mean time to repair” can mean at least four different things, and that incident lengths are skewed so much that an average can swing without real change. In the lab example it quotes, eleven incidents produced a mean time-to-mitigate of roughly 78 minutes against a median of 32, and one severe event alone shifted the mean by about 45 minutes. Better numbers include the spread of detection, acknowledgement, mitigation and resolution times, reliability budget burned, how heavy the pager has been, and what fraction of follow-ups got finished by their deadline.

Where do we start if we have nothing today?

The Atlas offers a gentle ramp. In the first month, agree a severity scale, set up an on-call rotation (ideally defined in code), choose a status page and write short role cards. Hold one tabletop rehearsal. In the second month, train a handful of coordinators, set deadlines for follow-up actions and run a first paging drill. In the third, keep reviews on time and start reporting distributions instead of averages.

• Week one: jot down which failures count as incidents for your data products.
• Week two: make a one-page role card for coordinator, responders and updater.
• Week three: pick a quiet afternoon and rehearse with a made-up failure.
• Week four: hold a short review of the rehearsal and note two changes.

One last reassurance

Starting does not require new software. A shared document plus a standing agreement among colleagues covers a lot of ground. The data may be messy, but the way you respond to the mess can be calm, clear and kind, and that is what the whole idea is about.