Every data team has that one scheduled job nobody wants to touch. Maybe it is the nightly vacuum, the quarterly backfill, the failover script that saved you once in 2023 and has been a mystery ever since. It works, until the night it does not, and then you learn it had no owner, no rehearsal mode and no off switch. The discipline built to prevent that night is AutomationOps — the craft of running your operational automation like production software rather than a shoebox of shell scripts. As the Ops Atlas defines it, that means choosing what deserves automating, making every repeat safe, fencing the blast radius of a bad run, and keeping an owner attached until retirement day. Here are nine rules, translated for people whose robots live in the warehouse.
One boundary note before the list. This discipline concerns run-time actions — the restarts, the cleanups, the failovers, the credential rotations, the fleet-wide configuration sweeps. It does not claim the territory of infrastructure templates, workflow engines, chat-triggered commands or toil accounting; those are neighbouring fields with their own playbooks. What it takes from them is candidates: every fix your incident reviews keep repeating is a candidate for governed automation.
Nine AutomationOps Rules, Warehouse Edition
Describe the destination, not the errand
A job that says “run vacuum” is a coin flip wearing a cron schedule. A job that says “reach the state where dead tuples sit below threshold” can run ten times and be right ten times. Write the ending you want — ensure the partition is loaded, ensure the replica has caught up — and repeats become boring in the best way. Tooling exists to test this instinct: Molecule, the testing framework for infrastructure roles, ships an idempotence check inside its default sequence, because “run it twice, compare” is the cheapest honesty test going.
Refuse the empty list
This one has a body count. Google ran a housekeeping tool called Diskerase that received a list of disks to wipe; one day the list arrived empty and the tool treated blankness as blank-check permission, pointing itself at everything in sight. The lesson hardened into a control: an automation handed an empty target list must halt, never improvise. For data folks the translation is direct — a backfill handed an empty partition list should raise an alarm, not help itself to the whole table. Empty never means all.
Rehearse with the safety on
Before any run touches real tables, do a pass that computes but does not commit. Caveat worth tattooing somewhere visible: Ansible’s check mode quietly skips command and shell steps unless they are explicitly flagged for checking, so a rehearsal that looked spotless may simply have skipped the scary passages.
Ship in cohorts, canary first
Nobody pushes a model change to every downstream mart at once, and the same manners apply to operational jobs. Run one node, check its pulse, then march onward in small groups; Ansible bakes the habit in with its serial batching and a max-fail tolerance knob. A bad run that touches 5 percent of shards is a lesson; the same run across all of them is a resignation letter.
Give every job a fuse
A failure budget is a number chosen in daylight: after N failures, the run stops itself. Without it, the third failed shard triggers a fourth retry, the fourth a fifth, each attempt smearing the mess wider. Your orchestrator has retry policies; the fuse is what admits the retries have become the problem.
Install the stop handle before opening night
Every automation gets its own kill switch, and a global one sits above the family. The unglamorous part is the drill: for the riskier grades, pulling the stop gets practised twice a year, because a brake that has never been squeezed is furniture. Whoever is on call holds the standing right to pull it without a permission slip.
Grade the danger before the debut
The Atlas drafts a four-grade scale for the estate. Grade zero only reads. Grade one acts reversibly on a single target. Grade two acts reversibly on many, and at that point rehearsal, batching and health checks stop being courtesies. Grade three is the irreversible, data-destroying class — it demands a second human’s sign-off, soft delete as the default, and a staged rollout. The security world reached the same summit long ago: the NIST catalogue files dual authorization under two codes, AC-3(2) and CM-5(4), and both reduce to one sentence — two people concur before a destructive act lands.
Keep receipts, not vibes
Is the job actually worth its upkeep? Honest arithmetic: runs per month, times the minutes a human used to spend, minus the minutes the machine still consumes, minus maintenance, divided by sixty. Walk the Atlas’s worked example and you get forty runs a month earning back a touch under eight hours, the ledger crossing into profit around the fifth month, and nineteen of twenty runs succeeding. Track the fallback rate too — how often a human gets paged anyway — because a robot that always needs supervision is a puppet, not an operator. The ledger also wants the embarrassing columns: how often the automation itself causes an incident, how often its guards trip, how fast a drill stops a runaway, and how much of the estate anybody will actually claim.
Automate with an expiry date
Automation rots silently. The schema it assumed drifted, the threshold it hardcoded fit last year’s volume, the one person who understood it left for a startup in Lisbon. Register every entry with a review date and a named owner, and treat “we built it in 2022” as history, not assurance. The classic anti-patterns wear familiar faces: the script in somebody’s home directory, the messy manual workflow copied verbatim into code, the single god credential every job shares, the auto-remediation quietly papering over a defect nobody investigates anymore. Two more deserve a shudder. First, approval gates on everything, which trains approvers to click yes until the gate protects nothing. Second, orphaned jobs still carrying admin rights — governance debt accruing interest at 3 a.m., compounding until an incident collects it all at once.
And when someone pitches the next great scheduled job, run the four-gate question. Does it repeat, and is the procedure actually understood? If the underlying human workflow is chaos, you will merely produce chaos on a timer — fix the source first. Does it need judgement mid-flight? Then a human belongs in the loop, with the machine preparing everything short of the decision. Finally, does it pay back, or is consistency itself the prize? Only a yes at all four gates earns the cron entry.
Where AutomationOps Meets Your Stack This Week
You need not boil the lake. The Atlas’s ninety-day arc, compressed to its bones:
- Inventory what actually exists, including the feral scripts.
- Register the twenty scariest, each with a grade and an owner.
- Move them behind one runner, give them their own identities, and pull the stop once — on purpose, in daylight.
Your stack neighbours have been solving cousin problems all along: the data-quality crowd explain Monte Carlo’s early-warning approach to broken data, there is a hands-on walkthrough of Great Expectations for validation work, and if the orchestration layer itself needs taming, read the piece on Dagster’s answer to pipeline chaos. Same instinct throughout: catch failure while it is small, and make the repeat safe by construction.
