{"id":4561,"date":"2026-11-16T05:35:00","date_gmt":"2026-11-16T05:35:00","guid":{"rendered":"https:\/\/dataopsschool.com\/blog\/?p=4561"},"modified":"2026-11-18T14:35:00","modified_gmt":"2026-11-18T14:35:00","slug":"data-platform-recovery-audit","status":"publish","type":"post","link":"https:\/\/dataopsschool.com\/blog\/data-platform-recovery-audit\/","title":{"rendered":"A DisasterRecoveryOps Audit Checklist for Data Platforms"},"content":{"rendered":"<p>If a regional outage hit your data platform tomorrow, could you say, within ten minutes, which dashboards come back first and how stale their numbers would be? Most teams cannot. This checklist is a DisasterRecoveryOps audit built for data engineers: twenty-odd questions grouped by theme, each of which can be settled with yes, no or a calendar date. Print it, run it with your lead, and treat every &#8220;no&#8221; as a ticket.<\/p>\n<h2>Section A: Objectives you can defend<\/h2>\n<ul>\n<li>Does every pipeline and serving layer have a written recovery time objective, signed off by the person who owns the business outcome?<\/li>\n<li>Is the recovery point objective expressed as a point in time (for example, &#8220;no older than 14:55&#8221;) rather than a vague amount of data? NIST guidance defines it that way, and it keeps arguments short.<\/li>\n<li>Have you split assets into tiers? The Ops Atlas sample uses three levels, from a 1 h \/ 5 min pairing for the most critical work down to 72 h \/ 24 h for the least, with 8 h \/ 1 h between.<\/li>\n<li>Did anyone push back when somebody asked for zero downtime and zero loss everywhere? If nobody did, the numbers are decoration.<\/li>\n<\/ul>\n<h2>Section B: The effective loss window<\/h2>\n<p>This is where data teams usually surprise themselves. The Atlas states that the real recovery point of a service equals the worst replication lag among all the stores it depends on. A warehouse replicated every few minutes is no help if the metadata catalogue or the orchestration database lags by an hour.<\/p>\n<ul>\n<li>Have you listed every store a critical pipeline needs: warehouse, object storage, catalogue, orchestrator state, secrets, schedules?<\/li>\n<li>Do you measure replication lag for each one, continuously, and alert on the worst?<\/li>\n<li>Can you restore an earlier version of a table, not just the newest copy? Replication copies corruption just as quickly as it copies good data.<\/li>\n<\/ul>\n<h2>Section C: Failover order and fencing<\/h2>\n<ul>\n<li>Is the sequence written down as fence, promote, then move traffic?<\/li>\n<li>Does the fence actually stop everything that writes? The Atlas warns that turning off a DNS route only redirects traffic that uses it. Batch jobs, queue consumers and clients with cached addresses can walk straight past it.<\/li>\n<li>Are scheduled jobs paused on the damaged side so they do not run twice after recovery?<\/li>\n<li>Do you know the difference between a planned switchover and an emergency failover? The first can be arranged with zero loss; the second may drop writes that had not yet replicated.<\/li>\n<\/ul>\n<h2>Section D: Reconciling what was lost<\/h2>\n<p>After an unplanned failover, some writes exist only on the old primary. The Atlas gives a tidy order of operations. Use it as a checklist item, not a suggestion.<\/p>\n<ol>\n<li>Preserve the old primary first. Snapshots can expire along with backup retention, so copy the one you need somewhere that will not age out.<\/li>\n<li>Compare against a restored copy, never against the live production system.<\/li>\n<li>Replay differences through the application path so that business rules apply, and make the replay idempotent so a second run does no harm.<\/li>\n<\/ol>\n<p>If your team already runs data quality checks, as described in <a href=\"https:\/\/dataopsschool.com\/blog\/great-expectations-for-data-validation-in-dataops-a-practical-guide\/\">this practical guide to validating data with expectations<\/a>, reuse those assertions to prove that the reconciled data is sound before declaring victory.<\/p>\n<h2>Section E: Access when things are broken<\/h2>\n<ul>\n<li>Do break-glass credentials exist for recovery work, and are they stored in a vault that does not depend on the region you are recovering from?<\/li>\n<li>Have you tested logging in with them within the last quarter?<\/li>\n<li>Does recovery require anyone to click through a console from memory? The Atlas calls that pattern &#8220;click ops&#8221; and lists it as an anti-pattern, because a stressed human is the least reliable automation you own.<\/li>\n<\/ul>\n<h2>Section F: Evidence that you rehearse<\/h2>\n<p>The Atlas example scorecard is useful because it is blunt. Its sample drill reports a recovery time of 64 minutes against a 60-minute target, a decision delay of 25 minutes against 16 allowed, a loss window of 2.5 minutes against 5, drill coverage at 67 percent and plan conformance at 25 percent. Mixed results like that are the norm, and listing them honestly is the point.<\/p>\n<ul>\n<li>What share of tier-one assets have been through a drill this year?<\/li>\n<li>Did the last drill stay inside the objective, including the time spent deciding?<\/li>\n<li>Have you rehearsed failback, the journey home? The Atlas counts it as a migration in its own right, with fresh ways to go wrong.<\/li>\n<li>Are drill results kept in one place, with dates, so an auditor or a new hire can read them?<\/li>\n<\/ul>\n<h2>Section G: Tooling choices, kept in proportion<\/h2>\n<p>Orchestrators help here because they make steps repeatable. The write-up on <a href=\"https:\/\/dataopsschool.com\/blog\/how-dagster-brings-order-to-your-data-chaos\/\">calming a messy pipeline estate using workflow orchestration<\/a> is a decent primer on why scripted, versioned workflows beat tribal knowledge. For selecting software more broadly, the <a href=\"https:\/\/dataopsschool.com\/blog\/the-simple-guide-to-picking-the-right-dataops-software\/\">simple guide to choosing DataOps software<\/a> can help you weigh recovery features alongside everything else.<\/p>\n<ul>\n<li>Can the failover steps be run from a script or pipeline instead of by a person with a screenshot?<\/li>\n<li>Are base images, quotas and capacity in the standby region checked on a schedule? Stale ones are a favourite hidden fault.<\/li>\n<li>Do control-plane dependencies, such as the service that launches new capacity, work when the region is degraded?<\/li>\n<\/ul>\n<h2>Section H: Governance and reading material<\/h2>\n<p>For background on how this discipline is framed, read <a href=\"https:\/\/www.xopsschool.com\/disasterrecoveryops\/\">the xOps School Atlas page covering DisasterRecoveryOps<\/a>. It also lists the standards people cite, including NIST SP 800-34, ISO 22301 and, for financial entities in the EU, the DORA regulation that requires yearly testing of recovery plans. If you work for a regulated customer, ask your compliance colleagues which of these apply to you; this checklist does not give regulatory advice.<\/p>\n<ul>\n<li>Is there a named owner for the recovery programme and a named person with authority to declare a failover?<\/li>\n<li>Is there a decision deadline? The Atlas formula subtracts your slowest recent drill time and a safety margin from the target. With a 60-minute objective, a 33-minute drill and a 5-minute margin, that gives 22 minutes to decide.<\/li>\n<\/ul>\n<h2>How to score it<\/h2>\n<p>Give one point per &#8220;yes&#8221; and ignore partial credit. Under twelve out of twenty-five: choose three items and fix them this month. Between twelve and twenty: schedule a staging switchover. Above twenty: run a production drill on the top tier and rehearse failback. Then repeat the audit next quarter, and see which answers changed because of work rather than because of optimism.<\/p>\n","protected":false},"excerpt":{"rendered":"<p>A no-nonsense checklist for data engineers: objectives, loss windows, fencing, reconciliation, access and drills, scored in an afternoon.<\/p>\n","protected":false},"author":6,"featured_media":4562,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[814],"tags":[],"class_list":["post-4561","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-dataops"],"_links":{"self":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/4561","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=4561"}],"version-history":[{"count":1,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/4561\/revisions"}],"predecessor-version":[{"id":4689,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/4561\/revisions\/4689"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/media\/4562"}],"wp:attachment":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=4561"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=4561"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=4561"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}