Data teams rarely blame the container. When a nightly load dies, the first suspects are the source system, the schema or the warehouse. Yet a surprising share of pipeline mysteries come from how the job’s container was built, named and limited. The Ops Atlas groups that care under the label ContainerOps, and this article uses it as a diagnostic checklist: symptom, likely cause, fix.
Do not go hunting for ContainerOps in job postings or standards documents, since the Atlas coined the label itself. The practices underneath are old and solid. The Ops Atlas page on ContainerOps collects them, and the cases below draw on its figures only where stated.
Symptom one: the job vanishes with exit code 137
What you see: a transformation step stops mid-run, no stack trace, and the orchestrator logs an exit code of 137.
Likely cause: that number is 128 plus 9, meaning the process was ended by a forced kill. In a data workload the usual trigger is memory. A join or a full-file read pulls more into RAM than the container was allowed. The Atlas adds that on newer resource-control setups, the out-of-memory label can also reflect pressure on the whole host, not only the container’s own ceiling.
Fix: set an explicit memory limit sized from a measured peak plus headroom, and stream or chunk the biggest reads. Then count memory kills against the number of container-days you run, a ratio the Atlas suggests tracking, so you can see whether things improve over a quarter.
Symptom two: a stop request loses the last batch
What you see: redeploys leave half-written files or duplicated rows. Logs show the process never reacted to the shutdown request.
Likely cause: the container’s start command was written in the shell form, so a shell sits between the runtime and your program and swallows the polite stop signal. After a grace period, ten seconds by default in Compose according to the Atlas, the runtime sends the forced kill.
Fix: use the exec form of the start command, handle the termination signal in code by finishing or rolling back the current unit of work, and add the init option for jobs that start child processes. Data quality tooling helps you notice the damage, for example the checks described in the practical guide to validation with Great Expectations, but it cannot undo a forced kill halfway through a write.
Symptom three: the same code gives different results on different days
What you see: a rerun of last week’s job produces slightly different numbers, and nobody changed the code.
Likely cause: the image reference is a moving tag. Whoever built the base image published a new build under the same name, and your next pull picked up different libraries. The Atlas lists floating tags in production and deploying by tag among its anti-patterns.
Fix: deploy by digest, the content fingerprint of one exact build, and keep release tags immutable. To stop the digest from freezing security fixes, let a bot such as Renovate open pull requests for newer bases, then review and merge. Model-layer tools like those in our walkthrough of dbt transformations deserve the same reproducibility at the runtime layer under them.
Symptom four: the host disk fills and everything stalls
What you see: new runs fail to start, writes error out, and a look at the server shows disk space gone.
Likely cause: container logs. The Atlas notes that the default json-file logging has no rotation, so a verbose job will keep appending until the disk is full. Old images piling up on the host are the second usual suspect.
Fix: configure log rotation at the daemon or per container, set a retention and clean-up policy for images and registry tags, and ship logs elsewhere if you need them long term. Log pipelines themselves are outside this discipline, but the default on a single host is squarely inside it.
Symptom five: a job is slow although CPU looks fine
What you see: average processor usage is comfortably below the limit, yet runtime is longer than it should be and latency spikes appear in streaming components.
Likely cause: throttling. The Atlas walks through a sample case. A job capped at two CPUs burned ninety CPU-seconds inside one minute, which averages out at 1.5 CPUs and looks healthy. Even so, the kernel had paused it in 240 out of 600 accounting periods, a 40 percent hit rate, for six seconds in total. Averages hide bursts.
Fix: divide throttled periods by all periods, using the counters the kernel exposes per container, and aim to stay under 5 percent where latency counts. Either raise the cap or drop it for that workload and rely on other safeguards. Also check thread pools, because a program that asks the container how many CPUs exist may receive the host’s count and start far more workers than its allowance supports.
A quick triage table in words
- Sudden exit with 137: check memory limit, peak usage and host pressure.
- Half-finished writes after deploys: check start command form and signal handling.
- Results drift without code changes: check whether the image is pinned by digest.
- Disk full: check log rotation and image retention.
- Slow with low average CPU: check the throttle ratio and thread pool sizing.
Orchestration is a separate question
Everything above applies to a single container on a single host, which is exactly the layer the Atlas assigns to this discipline. Scheduling across many machines is a different topic with different tools. If your scheduler, for instance something like the one covered in our piece on bringing order to data chaos with Dagster, launches containers for you, these fixes still belong in the images and settings it launches.
The temptation is to fix each symptom with a patch inside the running container. The Atlas lists patching live containers through exec as an anti-pattern: the change disappears at the next restart and nobody records what was done. Rebuild the image, push a new digest and deploy that instead.
What this looks like in an incident review
When a pipeline fails, the post-incident write-up usually names a data cause: a late file, a changed column, a bad join. Add one more question to the template: what was the container’s state at the time? Record the image digest, the limits in force, the exit code and how the stop was requested. After a handful of reviews a pattern emerges, because the same couple of container-level causes keep returning, and those are cheap to remove for good.
It also helps to separate the quick fix from the durable one. Raising a memory limit at two in the morning is a legitimate quick fix. The durable one is understanding why the peak grew, writing it down and checking the limit again after the next significant data volume change.
Metrics worth putting on the team wall
Three numbers from the Atlas suit data teams well. Digest coverage is the share of production workloads that run by digest, with 100 percent as the target. Base-image age tells you how stale the oldest foundation under any job has become. The throttle ratio from symptom five completes the set. None of them needs a new tool, only a script that reads what the runtime already reports and a place to chart it weekly.
Your first Monday task
Pick the one pipeline that pages people most often. Look up its image reference, its memory limit, its start command form and where its logs go. That is four lookups, and in many teams at least one of them turns up the cause of last month’s incident.