The ticket sat in the queue for eleven days. Not a pipeline bug, not a broken dbt model, not a slow query \u2014 a request to change two memory settings and one retention flag on the team’s own staging cluster. The request was trivial; the platform it targeted had never adopted GitOps, the discipline of recording a system’s intended state in a versioned repository and letting automated agents pull that record and reconcile the live system against it. The engineer assigned to the ticket spent most of those eleven days waiting: for a window, for a colleague who knew the environment, for permission to touch a system whose real state nobody could describe with confidence. Stories like this one are common in data platform work, and they rarely get reported as incidents, because nothing broke. Work simply slowed to the pace of the most fragile hand-off in the building. This piece walks the diagnosis the way a doctor would \u2014 symptom, findings, treatment \u2014 and shows why the treatment that finally held was the pull-based operating model described above.<\/p>\n
The team in question ships data products for a retail group: ingestion jobs, dbt transformations, a feature store, dashboards. Their pipeline code moved fast \u2014 versioned, reviewed, tested, merged. Yet any change to the surroundings of that code (compute sizing, a new environment, a warehouse credential rotation) travelled at a walk. When leadership asked why, the answers described symptoms rather than causes: “staging is special”, “only two people know the cluster”, “we find out something changed when the pipeline fails”. DataOps had disciplined the data; it had never disciplined the platform the data ran on.<\/p>\n
The cost was broader than the obvious delay. Onboarding a new engineer took weeks, because the platform’s behaviour lived in two heads and a pile of shell scripts. Capacity planning happened by folklore \u2014 a node pool was “about right”. And audits of the analytics environment, increasingly requested by the retail clients themselves, produced scurrying rather than answers. Slow, opaque, unrepeatable: the classic signature of configuration that exists only as accumulated handiwork.<\/p>\n
Worth naming: nothing in this story was a talent problem. The two people who understood the cluster were excellent. The fault was structural \u2014 knowledge that lives in a few heads cannot scale, and cannot be audited either.<\/p>\n
A short assessment surfaced three findings, and each is common enough to be a pattern of its own.<\/p>\n
One root pattern connects all three: the system’s true configuration existed only as the sum of whatever had been done to it, with no single place where intent was written down. Data teams recognise this disease instantly \u2014 it is exactly what unmanaged pipelines do to table schemas, and the remedy rhymes: make the intent explicit, version it, and let the machine enforce the match.<\/p>\n
The prescription was a pull-based operating model. Every element of the platform \u2014 namespaces, workloads, warehouse connection settings, the orchestrator itself \u2014 would be described declaratively in a repository, and an in-cluster agent would continuously pull that description and reconcile the live system against it. Nobody would log in to change production again; they would merge a pull request, and the agent would do the reaching. Security guidance supports this direction. The NIST publication SP 800-204C, written for microservices deployment, favours the pull arrangement for a plain reason: a pushing pipeline must be handed credentials that open the cluster, and credentials stored outside the boundary have a habit of escaping. Under the pull model the credentials never leave; the agent holds them, and the outside world gets read access to files. The Ops Atlas page on GitOps<\/a> documents this argument, and the OpenGitOps principles behind it, with unusual care.<\/p>\n Adapting the model to a data platform took two decisions worth stealing. First, environments became directories in one repository \u2014 staging\/, production\/ \u2014 rather than branches, so promotion meant copying a reviewed change between folders and looking at the resulting diff, not cherry-picking commits across long-lived branches that diverge in silence. Second, promotion keys on the exact artifact digest: an application version that passed staging is the same immutable version production receives, never a floating tag that could resolve to something new overnight.<\/p>\n Benchmarks published from reference labs give a sense of what mature GitOps delivery looks like in numbers, and the team’s trajectory moved toward them. The lab’s weekly sample found a median of roughly four minutes from merged change to healthy workload, with a p90 of twenty-six minutes \u2014 the tail dominated by large renders rather than by queueing for a human. About one deployment in five failed to apply cleanly, which sounds alarming until you compare it with the old world, where failures surfaced hours later as broken pipelines. The most instructive figure concerns drift: across a week, workloads spent about one percent of their runtime in a state that disagreed with the repository, and a pair of drift events accounted for nearly all of it. Drift, in other words, becomes measurable instead of anecdotal \u2014 and once measured, it becomes manageable.<\/p>\n The team’s own ledger told the same story in their currency. The eleven-day ticket class vanished: environment changes became pull requests with review times in hours. The two-person knowledge concentration broke up, because the repository now explained the platform to anyone who could read. And the CI system’s standing production credentials were retired \u2014 a change the security team noticed before engineering did.<\/p>\n Start smaller than the prescription sounds. One repository, one non-critical namespace, prune disabled while trust builds. Migrate one component at a time and let the agent manage itself after the first week, so the tool joins the discipline it enforces. Decide drift policy per resource before you need it: tolerate what the autoscaler owns, alert on the rest, and protect stateful things from automated deletion. Reference layouts usually organise a monorepo into clusters, infrastructure, tenants and applications, and one overlay should carry exactly one value change so reviews stay readable.<\/p>\n Two data-platform-specific notes belong in any such plan. Secrets \u2014 warehouse passwords, API keys for external services \u2014 need a dedicated mechanism such as encrypted files or an operator that syncs from a vault, never a plain commit. And the render step belongs in CI: generate the final manifests there, attach the rendered diff to the pull request, and let reviewers read output rather than templates. Both practices cost an afternoon and prevent the two most common first-month regrets.<\/p>\n The diagnosis-first habit matters more than any tool, and it generalises: Monte Carlo’s approach to stopping bad data before it reaches production<\/a> applies the same observe-compare-correct thinking to tables, and Great Expectations does it for data validation in DataOps pipelines<\/a>. Extending that discipline from data to the platform itself \u2014 with dbt transformations under modern DataOps pipeline control<\/a> \u2014 closes the loop. When every layer of the stack carries an explicit, versioned intent, the eleven-day ticket stops being a story you tell and becomes a warning from a world you left.<\/p>\n","protected":false},"excerpt":{"rendered":" A data platform team could ship pipeline code in hours but needed days for an environment change. Here is the diagnosis, and the GitOps fix that worked.<\/p>\n","protected":false},"author":6,"featured_media":4256,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[814],"tags":[],"class_list":["post-4255","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-dataops"],"_links":{"self":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/4255","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/users\/6"}],"replies":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=4255"}],"version-history":[{"count":1,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/4255\/revisions"}],"predecessor-version":[{"id":4286,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/4255\/revisions\/4286"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/media\/4256"}],"wp:attachment":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=4255"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=4255"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=4255"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}Ninety Days with GitOps, Measured Honestly<\/h2>\n
Starting a GitOps Migration in a Data Shop<\/h2>\n