{"id":4581,"date":"2026-11-17T04:35:00","date_gmt":"2026-11-17T04:35:00","guid":{"rendered":"https:\/\/dataopsschool.com\/blog\/?p=4581"},"modified":"2026-11-20T09:35:00","modified_gmt":"2026-11-20T09:35:00","slug":"aiops-questions-data-teams","status":"publish","type":"post","link":"https:\/\/dataopsschool.com\/blog\/aiops-questions-data-teams\/","title":{"rendered":"Ten AIOps Questions a Data Team Asks, and Plain Answers"},"content":{"rendered":"<p>Last spring a data engineer I will call Mira told her team something that stuck with me: &#8220;We built clean pipelines and then drowned in pings about them.&#8221; Her warehouse jobs were well tested, yet the on-call channel buzzed all night with failed-task notices, late-arrival warnings and freshness alerts that all described the same stalled upstream feed. Her question was the one many data teams ask: is AIOps the answer, and where does a person even begin? What follows are the questions Mira and her colleagues actually asked, answered in plain terms.<\/p>\n<h2>So what is AIOps when your daily work is pipelines?<\/h2>\n<p>Think of it as letting software absorb the repetitive part of alert handling: spotting duplicates, bundling alerts that share a cause, flagging oddities that fixed rules cannot catch, and suggesting where to look. Analyst glossaries phrase it as combining big data and machine learning for event correlation, anomaly detection and causality. The Ops Atlas page on the topic, <a href=\"https:\/\/www.xopsschool.com\/aiops\/\">an explanation of AIOps scope and what falls outside it<\/a>, adds a refreshing caveat: it is artificial intelligence for operations, not operations for artificial intelligence, so building and serving models belongs to a different discipline.<\/p>\n<h2>Do we need machine learning to get started?<\/h2>\n<p>Mira expected to say yes. The Atlas says no for the first stretch. Its ordering puts four plain practices ahead of any learning: a contract for alert labels, removing duplicates, grouping with careful suppression, and correlating alerts with the dependency map and the change history. Those are the &#8220;unglamorous&#8221; steps, in the Atlas&#8217;s own word, and they deliver most of the noise reduction.<\/p>\n<p>For a data team the parallel is obvious. Before reaching for anomaly models on table row counts, you first make sure every pipeline alert names its dataset, owner and severity, and that the same failure does not page six different people.<\/p>\n<h2>Why did our &#8220;identical&#8221; alerts not collapse into one?<\/h2>\n<p>This was the first real bug Mira found. Deduplication works by fingerprinting each alert on its labels; two alerts with identical labels are one thing. Her scheduler stamped every task attempt with a run identifier, and that volatile value sat inside the fingerprint, so every retry looked brand new. The Atlas describes the same trap with generated container names. The cure is dull and effective: keep anything that changes per attempt out of the fingerprint and carry it as plain detail instead.<\/p>\n<h2>How do we group alerts without hiding something important?<\/h2>\n<p>Grouping bundles alerts into one notification, and inhibition suppresses a downstream alert while its upstream cause is firing. Both are powerful and both can backfire. One worked runbook in the Atlas shows a suppression rule muting a genuine outage, thanks to a sloppy label: the same field meant &#8220;the system that is broken&#8221; in some alerts and &#8220;the system doing the checking&#8221; in others. Splitting that field in two fixed it.<\/p>\n<p>Mira&#8217;s team took away a rule of thumb: every suppression rule must name which labels have to match, and nobody merges a rule until a colleague has tried to break it with an awkward example. The Atlas also suggests starting the incident runbook with a command that lists whatever is currently being suppressed, so responders see the hidden alerts first.<\/p>\n<h2>Can we trust a time window to group related failures?<\/h2>\n<p>Mira&#8217;s first shortcut was to treat anything failing inside a five-minute span as one event. Funnily enough, the Atlas had run that very experiment in its lab. Alert volume shrank 5.5-fold, and then the authors looked closer and found that unrelated incidents had been fused. A cheaper-looking inbox was hiding mistakes. For data pipelines this would be like merging a late vendor file and an unrelated schema change into one ticket, then assigning it to the wrong squad.<\/p>\n<p>The proper test is to score the grouping against incidents people have already labelled, using precision (how many of the proposed groups were right) and recall (how many of the true groups were found).<\/p>\n<h2>How do we map dependencies when our lineage is messy?<\/h2>\n<p>Start with what you have. The Atlas calls for a dependency list and a change log before anything else, and says that correlation quality tracks the quality of that map. In one of its experiments, removing a single edge from a test topology pulled recall down sharply. Mira&#8217;s team began with the lineage tool&#8217;s export, hand-corrected the five most important feeds and recorded every deployment of transformation code. That change record paid off straight away, since the Google SRE book, cited by the Atlas, attributes most outages to changes in running systems.<\/p>\n<p>If your own pipeline checks are still being designed, the walkthrough of <a href=\"https:\/\/dataopsschool.com\/blog\/great-expectations-for-data-validation-in-dataops-a-practical-guide\/\">writing validation checks that catch bad data early<\/a> pairs well with this step, because good checks produce alerts that already carry the right labels.<\/p>\n<h2>Which anomaly detection should we trust on seasonal data?<\/h2>\n<p>Data teams live with weekly and monthly rhythm, so this question matters. The Atlas lab ran a seasonal baseline that caught every injected fault yet fired a pile of false alarms, almost all of them on a public holiday that was not on any calendar. Entering that one date removed the lot. The lesson Mira wrote on the whiteboard: tell the detector about your known events before judging it.<\/p>\n<p>The Atlas also cites research that tempers expectations. A 2022 conference paper reported that neural log-anomaly detectors often disappoint, and a 2024 paper found that simple detectors based on how often each log template appears were competitive on common benchmarks. In other words, try the simple thing first and measure.<\/p>\n<h2>When is it safe to let automation act?<\/h2>\n<p>Only after it has been earning trust quietly. The Atlas describes a lifecycle in which a new rule is proposed, runs in shadow (scored but invisible to responders), moves to a canary on one or two services, becomes active, and is later reviewed and eventually retired, each step with a defined way back. It also suggests having about a hundred labelled incidents before trying any learned grouping in shadow mode.<\/p>\n<p>Actions themselves are fenced: allow-listed, reversible and rate-limited. Mira&#8217;s first automation was modest, a job that restarts a stuck ingestion worker no more than a couple of times per hour and pages a person if the second restart does not help.<\/p>\n<h2>What about language models reading our alerts?<\/h2>\n<p>Useful for drafting summaries, risky as unsupervised operators. The Atlas notes that agent benchmark scores of around 56 to 57 percent for diagnosis measure narrower work than live resolution, and that alert text can be crafted by an attacker, so it should be treated as untrusted input. Mira lets a model suggest a first paragraph for the incident note, and a human edits and sends it.<\/p>\n<h2>How do we know it is working?<\/h2>\n<p>The Atlas advocates a calm cadence of reviews: weekly for the noisiest alerts, monthly for a scorecard, quarterly for the platform as a whole. Mira&#8217;s team added one line to their Monday stand-up, which was to name the three noisiest alerts and decide whether each should be fixed, grouped or deleted. Within a quarter the overnight channel had gone quiet enough that the team trusted it again. Another post here, on <a href=\"https:\/\/dataopsschool.com\/blog\/master-data-transformations-with-dbt-in-modern-dataops-pipelines\/\">building reliable transformations with dbt<\/a>, shows how the upstream half of that story is handled.<\/p>\n<p>If there is a single takeaway from Mira&#8217;s spring, it is that the way out of alert fatigue is a sequence of small, boring, measurable fixes, and the glamorous parts can wait their turn.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A data engineer&#8217;s alert-fatigue story told through the questions her team asked about AIOps, from deduplication to when automation may act.<\/p>\n","protected":false},"author":6,"featured_media":4582,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[814],"tags":[],"class_list":["post-4581","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-dataops"],"_links":{"self":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/4581","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=4581"}],"version-history":[{"count":1,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/4581\/revisions"}],"predecessor-version":[{"id":4719,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/4581\/revisions\/4719"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/media\/4582"}],"wp:attachment":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=4581"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=4581"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=4581"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}