Data platforms are among the most awkward things to put on a budget. A nightly job that cost a few dollars last quarter can cost ten times that after one new join, and nobody notices because the warehouse bill arrives as a single number. This how-to applies FinOps to a data platform in six steps, and its definitions and worked figures come from the Atlas page that covers FinOps.
The order matters. Each step produces something the next one needs, so resist the urge to jump straight to optimisation.
Step 1: Agree what “cost” means before you measure it
The Atlas builds its model on one equation: cost equals usage multiplied by rate. For a data team, usage is compute hours, bytes scanned and bytes stored; rate is whatever your contract or provider charges for each. Write both down for every component of the platform. Then pick which view of price you will report. The open cost-and-usage specification the Atlas describes separates cash cost as billed from the effective cost that spreads purchases over the period they cover, and it also carries list and contracted prices. Choose one primary view, say effective cost, and stick to it so that month-to-month comparisons mean something.
Step 2: Get the data out of the console
Decisions need a table, not a dashboard screenshot. Turn on the detailed billing export from your provider as early as you can. The Atlas notes a trap on one major cloud: the export backfills only the current and previous month, so anything you forget to enable early is simply never recoverable. If your providers offer the common open schema, use it, because it lets you join bills from several vendors without writing a translation for each. Any custom columns you add should carry the x_ prefix the specification requires, so they never clash with standard fields.
Step 3: Allocate, then measure what is left over
Tag every job, dataset and cluster with an owning team and a product. Then measure two numbers separately: the share of cost that carries a direct tag, and the share that is allocated once you spread shared costs across consumers by an agreed rule. One lab in the Atlas starts with an 11,800 cost base. Only 57.6 percent carried a direct tag; spreading the shared platform pool lifted the allocated figure to 89.0 percent, which still left 5.1 percent with no owner. The gap between the first and second figures is your shared-cost problem, and the final figure is your accountability problem. Data platforms usually have a big shared pool, such as the warehouse used by everyone, so expect the gap to be wide.
If you plan the work around orchestration, the article on putting scheduled data jobs in order shows where owner labels can be attached at the job level, which is the cheapest place to add them.
Step 4: Define a unit metric and chart it weekly
A total bill hides everything useful. The Atlas separates resource-efficiency measures, such as cost per gigabyte processed or per vCPU hour, from business measures, for example per customer account or per completed order. It adds a sensible warning: compare a metric against its own history inside one scope instead of ranking different products against each other, because their workloads are not alike. For a data team, a good starter is cost per thousand rows delivered to a downstream consumer, or cost per refreshed report. Chart it weekly. A rising total with a flat unit cost is growth; a rising unit cost is a finding.
Step 5: Set a cadence and wire in anomaly checks
Accountability in the Atlas is data times decision rights times cadence, so a data set with no meeting is half a practice. Its proposed calendar runs from a daily scan for odd spending, through a weekly team review and a monthly showback-and-forecast session, to quarterly commitment checks and a yearly budget and renewals round. Know the limits of your tools while you set this up: the Atlas notes that one provider’s anomaly alerts may arrive a full day late and only start working after ten days of history, and that its hourly chart reaches back just 14 days at a cent per request. Build the daily triage around those facts.
When a run does spike, check the transformation layer first. The walkthrough on building SQL transformations in modern pipelines is a good primer for finding which models are doing the heavy scanning.
Step 6: Only now, optimise
Rightsizing and commitments come last. For rightsizing, the Atlas lists a careful sequence: base the choice on utilisation figures that include the peaks, settle service-level guardrails first, alter a single dimension per change, watch through an entire business cycle, and undo the change if any service level slips. For commitments, aim at the lowest hourly level of eligible usage, since evaluation happens hour by hour and any idle portion of an hour’s commitment disappears.
Data platforms make that second rule vivid, because load is usually lumpy: heavy overnight, light at midday, heavy again at month end. The Atlas’s second lab shows the trap. Picture 100 units an hour by day, 40 overnight and a 30 percent discount: the commitment that paid off most was 28 an hour. Lifting it to 42 an hour pushed coverage up to 80.9 percent while the savings rate fell to 6.0 percent. Judge the purchase by the savings rate, which is net savings over the on-demand equivalent, and not by coverage.
What to watch for along the way
- Do not size a resource-based commitment to the footprint you had before you optimised; the Atlas describes this as paying for resources that no longer exist in their original form.
- Do not rightsize after buying a resource-based commitment, or you strand part of it.
- Do not add up two different recommendation lists, since the same saving is counted twice.
- Do not forecast from a trend alone when a launch or a backfill project is on the calendar.
How you know it is working
Three signs. Your unowned share is falling toward the low single digits. Anyone on the team can say what a thousand delivered rows cost this week and last. And a spike gets a named owner within a day. The Atlas cites sample targets of at least 85 percent allocation coverage for the middle maturity stage and over 90 percent for the top one, assessed capability by capability. Use them as signposts, not as a scoreboard.
One last caution from the Atlas: a tool purchase is not the same as the practice. The first working version of all six steps can live in a spreadsheet and a calendar invite. Prove the habit first, then automate the parts that hurt.