{"id":4570,"date":"2026-11-16T15:35:00","date_gmt":"2026-11-16T15:35:00","guid":{"rendered":"https:\/\/dataopsschool.com\/blog\/?p=4570"},"modified":"2026-11-19T09:35:00","modified_gmt":"2026-11-19T09:35:00","slug":"nightly-load-went-quiet-sysops-story","status":"publish","type":"post","link":"https:\/\/dataopsschool.com\/blog\/nightly-load-went-quiet-sysops-story\/","title":{"rendered":"The Monday the Nightly Load Went Quiet: A SysOps Story for Data Teams"},"content":{"rendered":"<p>The story below is invented for illustration, a composite of the kind of morning many data teams will recognise. The people are not real and the numbers are made up, apart from the general facts about Linux hosts that come from the Ops Atlas. It is a tale about SysOps, the unglamorous care of the machines your pipelines run on, seen through the eyes of one small data team across a single Monday.<\/p>\n<h2>Monday, 8:40: the dashboard that stopped at Friday<\/h2>\n<p>Mira, an analyst on a four-person data team, opened the sales dashboard and noticed the latest row was from Friday. Nothing had flagged red. The orchestration screen showed every task green, because the tasks that should have run over the weekend had simply never been triggered. Nobody had been paged for the absence of something.<\/p>\n<p>She pinged Tomas, the engineer who looked after the shared Linux host where the ingestion scripts lived. He did not restart anything. A habit he had picked up from a mentor was to look before touching, and the Atlas lists restart-first among its anti-patterns because it wipes out the evidence. He saved a copy of the recent journal, the list of scheduled jobs and the output of a few status commands into a folder, and only then started reading.<\/p>\n<h2>The first clue: a job that ran, but not as intended<\/h2>\n<p>The ingestion was triggered by an old-style scheduler entry. When Tomas looked closely, he found that a recent edit had added a date stamp to the output filename using a percent-sign format. The Atlas flags exactly this trap: inside that scheduler&#8217;s entries a bare % acts as a line break, which sliced the command in two and handed the tail to the shell as stray input. The job had been failing silently since Saturday&#8217;s edit.<\/p>\n<p>The fix was easy, and what followed was more instructive. Instead of escaping the character and moving on, the team decided to retire the entry and use a systemd timer. The Atlas recommends that route, with a setting that makes missed runs execute when the machine comes back and a failure hook that notifies someone. For a team that wanted a record of what ran and when, a timer&#8217;s unit logs were worth the small learning curve.<\/p>\n<h2>The second clue: the disk had a deadline<\/h2>\n<p>While the journal was open, Tomas checked space. The volume was far fuller than anyone remembered. Using the arithmetic the Atlas teaches, he took the used amount and the weekly growth, subtracted the portion reserved for the system, and divided what remained by the daily growth. The answer was close to a week, shorter than the number the quick glance had suggested. Nothing had broken yet, but the pipeline&#8217;s staging area was the thing eating the space.<\/p>\n<p>That discovery reframed the whole morning. Missing data was a symptom; a disk heading for a wall was the disease. A metric the Atlas calls disk runway, the number of days before a volume fills, would have shown amber for a fortnight if anyone had been tracking it.<\/p>\n<h2>Lunch: writing down what had been tribal knowledge<\/h2>\n<p>Over lunch the team made a small list. Mira asked questions and Tomas answered; she typed. Which jobs run on this host? Who can log in? Where do logs live and how long are they kept? On that last point the Atlas had a useful detail: the system journal is capped by default at roughly a tenth of the filesystem, it tries to leave about fifteen percent free, and each file tops out at 4 GiB. It advises choosing the storage mode explicitly rather than relying on those defaults. Tomas set a deliberate cap and noted it in the repository.<\/p>\n<p>They also talked about how the team&#8217;s transformation layer fits in. For anyone curious about that side of the stack, <a href=\"https:\/\/dataopsschool.com\/blog\/master-data-transformations-with-dbt-in-modern-dataops-pipelines\/\">a hands-on piece about building transformations with dbt<\/a> covers the logic that runs after the data lands, and <a href=\"https:\/\/dataopsschool.com\/blog\/how-dagster-brings-order-to-your-data-chaos\/\">the article on orchestrating data work with Dagster<\/a> explains the scheduling layer above it. Both assume the host underneath is healthy, which was precisely the assumption that had just been tested.<\/p>\n<h2>The afternoon: a rehearsal nobody wanted<\/h2>\n<p>The last item on Tomas&#8217;s list was uncomfortable. If the host had died instead of merely stalling, how long would it take to bring it back? The Atlas draws a sharp line: a snapshot is not a backup, and the top rung of its maturity ladder asks for records of restores that were practised. The team had snapshots. They had never restored from one onto a clean machine.<\/p>\n<p>They picked a quiet Thursday, restored a copy to a scratch instance and timed the whole thing. The clock ran far past anyone&#8217;s guess, mostly because two credentials lived only in someone&#8217;s shell history. The exercise produced a checklist and a ticket to move those secrets somewhere proper.<\/p>\n<p>Nobody blamed anybody in the retrospective. Tomas pointed out that the scheduler edit was reasonable and the quirk obscure, and that the real gap was the absence of any check that told the team a nightly run had produced nothing. Mira added that on busy weeks she would never have spotted a stale row at all, because she normally reads the dashboard on Wednesdays. The lesson they wrote down was about visibility: a pipeline that fails loudly is an inconvenience, while one that fails quietly can mislead people for days.<\/p>\n<h2>What the team changed afterwards<\/h2>\n<ul>\n<li>Scheduled entries moved to timers, each with a failure notification and catch-up behaviour.<\/li>\n<li>A weekly check of days-of-disk-left, posted to the team channel.<\/li>\n<li>A one-page host record describing jobs, accounts, log retention and the restore procedure.<\/li>\n<li>An alert on missing data, not only on failed tasks, since the Monday failure had been an absence.<\/li>\n<\/ul>\n<p>The Atlas frames this kind of progress as moving up a ladder, from ad hoc to documented to baselined to measured. <a href=\"https:\/\/www.xopsschool.com\/sysops\/\">Its SysOps entry<\/a> places a thirty-day starter list at the bottom: inventory the host, remove any shared administrator login, collect basic performance history and set a few alerts. Tomas ticked off two of four before Friday.<\/p>\n<h2>The moral, such as it is<\/h2>\n<p>Mira&#8217;s closing note in the retrospective was short: pipelines are made of code and of machines, and only one of those shows up in the code review. The team did not become experts in a week. They simply began treating the host as something that deserved the same attention as their transformations, and that attitude carried more weight than any single tool or fix.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A made-up Monday on a small data team: a silent scheduler quirk, a disk with a deadline and a restore nobody had rehearsed, told as a short story.<\/p>\n","protected":false},"author":6,"featured_media":4571,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[814],"tags":[],"class_list":["post-4570","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-dataops"],"_links":{"self":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/4570","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=4570"}],"version-history":[{"count":1,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/4570\/revisions"}],"predecessor-version":[{"id":4701,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/4570\/revisions\/4701"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/media\/4571"}],"wp:attachment":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=4570"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=4570"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=4570"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}