Data engineers increasingly run their orchestrators, streaming jobs and notebooks on Kubernetes, which means the health of the cluster becomes part of the health of the data. When a scheduled load fails at 02:00, the root cause is sometimes a node that was drained, a quota that was too tight or a version that expired. This primer introduces KubernetesOps, the routine care of a cluster, in terms a data team can use, and every section ends on a short word list to help the vocabulary stick.
KubernetesOps scope: where the caretaker’s job starts and stops
The Ops Atlas, in its KubernetesOps entry, is candid that the term is informal. No standards body, book or certification uses it. Other tools and consultancies carry similar names, and the practice itself is more commonly called cluster administration or cluster operations. Inside its boundary the Atlas places the control plane, the machines, bundled extras, version changes, workload settings, scaling, capacity planning and fault-finding. Hardening for security, container images, delivery pipelines and the internal developer platform belong next door.
For an analytical reader, that scoping is the useful part. It tells you which problems belong to the cluster caretaker and which belong somewhere else. If a pipeline is slow because the transformation logic is wasteful, that is not a cluster matter. If it is slow because pods wait for room on nodes, it is.
Word list: the basics
- Cluster: a group of machines managed as one pool of capacity.
- Control plane: the part that decides where work runs and remembers the desired state.
- Node: a machine that actually runs your pods.
- Pod: the smallest unit that is scheduled, usually one application container plus helpers.
Versions: the clock that always runs
Kubernetes ships roughly three minor versions per year, and each receives patches for about fourteen months: twelve in standard support and two more in a maintenance phase reserved for critical fixes. Taken together, that means a cluster left alone will eventually fall out of support whether or not anything is broken. The Atlas therefore recommends tracking version lag, the number of minor versions behind the newest, and aiming for one or fewer in production, along with days remaining until your version reaches end of life.
Upgrading is deliberately sequential. You move one minor at a time, starting with the API server, then the controllers and scheduler, then the node agents, and the network proxy last. The node agents may be older than the API server by a limited margin but never newer. A data team that has scheduled a heavy quarterly backfill should plan the upgrade hops around it, much as you would freeze schema changes before a big reload.
Word list: versions
- Minor version: the middle number, for example the 37 in 1.37; where new features and removals arrive.
- Version lag: how far behind the newest minor your cluster is.
- Skew: the allowed gap between components of different versions.
- Deprecated API: an interface flagged for removal; still works for a while.
Resources: requests, usage and the evidence gap
The most data-flavoured part of the discipline is measuring requests against usage. Put simply, a request is the slice of CPU or memory that a pod asks the scheduler to hold back for it. The scheduler uses it to place the pod, regardless of how much the pod really consumes, and so do the autoscalers. Two ratios give you evidence: how much of the cluster is requested, and how much of that request is actually used. When requests are generous and usage is low, you pay for idle capacity and still see scheduling failures. The Atlas also suggests a floor for hygiene, with at least 98 percent of pods carrying requests, as a marker of a well-managed cluster.
A related caution concerns the built-in metrics component. The Atlas quotes the project as saying it is meant only for autoscaling purposes, so a dashboarding stack should come from a proper monitoring setup, not from that component.
Word list: capacity
- Request: the reserved amount a pod tells the scheduler it needs.
- Limit: a ceiling a pod may not exceed; its value is debated for CPU.
- Allocatable: what a node can hand to workloads after system overhead.
- Autoscaler: software that changes replica counts or node counts based on a signal.
Safety nets for stateful and scheduled work
Data workloads are less forgiving of surprise restarts than stateless web pods. Two protections matter. First, disruption budgets, which tell the cluster how many copies of a service must remain during voluntary maintenance. Second, the backup of the cluster’s own state store. For self-managed clusters, the Atlas stresses regular snapshots of that store and treats access to it as equivalent to root permission, so handle the copies as sensitive. Managed services take away part of this chore, yet the entry is clear that version-controlled configuration, separate data backups and a rebuild you have genuinely rehearsed remain your job.
Word list: protections
- Disruption budget: a rule limiting voluntary evictions of a service.
- Drain: asking a node to hand off its pods before maintenance.
- Snapshot: a point-in-time copy of the cluster state store.
- Restore readiness: evidence that a snapshot has been successfully replayed.
Where to go next in your own practice
Data teams already live by checks: schema checks, freshness checks, row counts. The mindset transfers well. For the data side of that habit, the practical guide to declaring data tests through the Great Expectations library shows how explicit expectations catch problems early, and the cluster equivalent is an explicit list of expected versions, requests and backups. If you also want a sense of how transformation layers fit together, see the guide to building modelled transformations with dbt and then ask where those jobs actually run.
A sensible first week looks like this: write down your cluster’s current version and its end-of-support date; list pods without requests; find any budget that allows zero disruption; and confirm when a restore was last rehearsed. None of that needs new software. It needs a spreadsheet, an afternoon and the willingness to treat the cluster as a dataset worth auditing.