{"id":3662,"date":"2026-02-17T19:03:10","date_gmt":"2026-02-17T19:03:10","guid":{"rendered":"https:\/\/dataopsschool.com\/blog\/compaction\/"},"modified":"2026-02-17T19:03:10","modified_gmt":"2026-02-17T19:03:10","slug":"compaction","status":"publish","type":"post","link":"https:\/\/dataopsschool.com\/blog\/compaction\/","title":{"rendered":"What is Compaction? Meaning, Architecture, Examples, Use Cases, and How to Measure It (2026 Guide)"},"content":{"rendered":"\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">Quick Definition (30\u201360 words)<\/h2>\n\n\n\n<p>Compaction is the process of consolidating, deduplicating, or restructuring data and metadata to reduce storage, improve read\/write performance, and simplify downstream processing. Analogy: compaction is like compressing and organizing a cluttered desk into labeled folders. Formal: a deterministic operation that merges records or segments according to defined retention, key, or tombstone rules.<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">What is Compaction?<\/h2>\n\n\n\n<p>Compaction is a maintenance operation applied to storage or message systems to reduce fragmentation, remove duplicates, and consolidate state for faster access or lower cost. It is not merely compression; it often involves semantic logic such as retaining the latest update per key, honoring tombstones, or merging aged segments.<\/p>\n\n\n\n<p>Key properties and constraints:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Deterministic semantics per compaction policy.<\/li>\n<li>Often time or resource-bounded to avoid impacting live traffic.<\/li>\n<li>Can be online (concurrent with reads\/writes) or offline.<\/li>\n<li>Must preserve correctness regarding deletions and ordering rules.<\/li>\n<li>Has trade-offs: CPU, I\/O, temporary storage, and latency.<\/li>\n<\/ul>\n\n\n\n<p>Where it fits in modern cloud\/SRE workflows:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Data lifecycle management for logs, metrics, and event stores.<\/li>\n<li>Stream processing and storage backends (log-structured stores, time-series DBs).<\/li>\n<li>Cost control in cloud storage and cold\/hot tier transitions.<\/li>\n<li>Recovery assist in backup\/restore and snapshot compaction.<\/li>\n<\/ul>\n\n\n\n<p>Text-only diagram description:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Imagine a timeline of segments A\u2192B\u2192C. Compaction reads A,B,C, applies key-based rules, writes consolidated segment D, marks A,B,C for deletion, updates index, and swaps readers to D with minimal downtime.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Compaction in one sentence<\/h3>\n\n\n\n<p>Compaction consolidates fragmented or duplicate storage segments into a smaller, consistent set of data artifacts while preserving defined semantic rules.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">Compaction vs related terms (TABLE REQUIRED)<\/h3>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>ID<\/th>\n<th>Term<\/th>\n<th>How it differs from Compaction<\/th>\n<th>Common confusion<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>T1<\/td>\n<td>Compression<\/td>\n<td>Operates on bytes to save space<\/td>\n<td>Often confused as same task<\/td>\n<\/tr>\n<tr>\n<td>T2<\/td>\n<td>Garbage collection<\/td>\n<td>Frees unused memory objects not storage segments<\/td>\n<td>Scope and triggers differ<\/td>\n<\/tr>\n<tr>\n<td>T3<\/td>\n<td>Aggregation<\/td>\n<td>Produces reduced summaries of values<\/td>\n<td>Not always reversible vs compaction is state-preserving<\/td>\n<\/tr>\n<tr>\n<td>T4<\/td>\n<td>Snapshotting<\/td>\n<td>Captures a point-in-time copy<\/td>\n<td>Snapshots are copies, compaction rewrites storage<\/td>\n<\/tr>\n<tr>\n<td>T5<\/td>\n<td>Reindexing<\/td>\n<td>Rebuilds lookup indices<\/td>\n<td>Reindexing may not remove duplicates<\/td>\n<\/tr>\n<tr>\n<td>T6<\/td>\n<td>Vacuuming<\/td>\n<td>DB-specific cleanup operation<\/td>\n<td>Varies across engines and rules<\/td>\n<\/tr>\n<tr>\n<td>T7<\/td>\n<td>Tiering<\/td>\n<td>Moves data between storage classes<\/td>\n<td>Tiering moves, compaction rewrites<\/td>\n<\/tr>\n<tr>\n<td>T8<\/td>\n<td>Deduplication<\/td>\n<td>Removes identical data blocks<\/td>\n<td>Dedup is block-level, compaction is semantic<\/td>\n<\/tr>\n<tr>\n<td>T9<\/td>\n<td>Checkpointing<\/td>\n<td>Persists checkpointed state for recovery<\/td>\n<td>Checkpoints are for recovery, compaction is maintenance<\/td>\n<\/tr>\n<tr>\n<td>T10<\/td>\n<td>Merge<\/td>\n<td>Generic combining operation<\/td>\n<td>Merge may lack tombstone semantics<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\">Row Details (only if any cell says \u201cSee details below\u201d)<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>None<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">Why does Compaction matter?<\/h2>\n\n\n\n<p>Business impact:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Revenue: Reduced storage costs and faster analytics lead to lower operational cost and faster time-to-insight for revenue-generating features.<\/li>\n<li>Trust: Ensures query correctness and predictable latencies for SLAs, which preserves user trust.<\/li>\n<li>Risk: Poor compaction can cause data loss, inconsistent reads, or runaway bills.<\/li>\n<\/ul>\n\n\n\n<p>Engineering impact:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Incident reduction: Well-designed compaction prevents fragmentation-related slowdowns and spike-driven backpressure incidents.<\/li>\n<li>Velocity: Faster reduced datasets enable quicker feature development and testing cycles.<\/li>\n<li>Cost: Lower storage footprint and fewer IO ops in cloud environments reduce continuous costs.<\/li>\n<\/ul>\n\n\n\n<p>SRE framing:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>SLIs\/SLOs: Compaction influences availability and latency SLIs for storage layers and downstream consumers.<\/li>\n<li>Error budgets: Heavy compaction that causes latency spikes should be accounted against error budgets.<\/li>\n<li>Toil: Manual compaction tasks are toil; automation reduces this.<\/li>\n<li>On-call impact: Compaction failures can cause paging if not properly isolated or throttled.<\/li>\n<\/ul>\n\n\n\n<p>What breaks in production (examples):<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Sudden compaction job overload causes storage IOPS saturation, leading to increased read latencies and downstream timeouts.<\/li>\n<li>Bug in tombstone handling causes deleted keys to reappear after compaction, creating data integrity incidents.<\/li>\n<li>Compaction runs without enough temp space and fails mid-way, leaving partial state and inconsistent indexes.<\/li>\n<li>Unthrottled compaction after a restore overwhelms the cluster network and causes cross-region replication lag.<\/li>\n<li>Misconfigured retention rules compact essential audit logs accidentally, causing compliance violations.<\/li>\n<\/ol>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">Where is Compaction used? (TABLE REQUIRED)<\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>ID<\/th>\n<th>Layer\/Area<\/th>\n<th>How Compaction appears<\/th>\n<th>Typical telemetry<\/th>\n<th>Common tools<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>L1<\/td>\n<td>Edge<\/td>\n<td>Local log segment trimming on devices<\/td>\n<td>Disk usage, segment count<\/td>\n<td>See details below: L1<\/td>\n<\/tr>\n<tr>\n<td>L2<\/td>\n<td>Network<\/td>\n<td>Packet or telemetry aggregation before forward<\/td>\n<td>Batch size, throughput<\/td>\n<td>See details below: L2<\/td>\n<\/tr>\n<tr>\n<td>L3<\/td>\n<td>Service<\/td>\n<td>Message broker segment merge<\/td>\n<td>Partition lag, IO wait<\/td>\n<td>Kafka RocksDB<\/td>\n<\/tr>\n<tr>\n<td>L4<\/td>\n<td>Application<\/td>\n<td>Local caches consolidation<\/td>\n<td>Cache eviction rate<\/td>\n<td>Redis Memtier<\/td>\n<\/tr>\n<tr>\n<td>L5<\/td>\n<td>Data<\/td>\n<td>Time-series and column store compaction<\/td>\n<td>TS retention, cold reads<\/td>\n<td>Prometheus ClickHouse<\/td>\n<\/tr>\n<tr>\n<td>L6<\/td>\n<td>Kubernetes<\/td>\n<td>Pod-level log rotation and node cleanup<\/td>\n<td>Node disk pressure<\/td>\n<td>kubelet logrotate<\/td>\n<\/tr>\n<tr>\n<td>L7<\/td>\n<td>Serverless\/PaaS<\/td>\n<td>Managed store auto-compaction<\/td>\n<td>Invocation latency, storage cost<\/td>\n<td>Cloud-managed services<\/td>\n<\/tr>\n<tr>\n<td>L8<\/td>\n<td>CI\/CD<\/td>\n<td>Artifact pruning and repository GC<\/td>\n<td>Repo size, build time<\/td>\n<td>Artifact registries<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\">Row Details (only if needed)<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>L1: Edge devices compact logs to preserve flash lifespan and reduce sync payloads.<\/li>\n<li>L2: Network appliances aggregate flows to reduce telemetry volume sent upstream.<\/li>\n<li>L7: Managed services expose compaction tuning as parameters or auto-mode; specifics vary.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">When should you use Compaction?<\/h2>\n\n\n\n<p>When it\u2019s necessary:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>When storage fragmentation causes read latencies or high IOPS.<\/li>\n<li>When deduplication will meaningfully reduce storage bills.<\/li>\n<li>When retention rules or tombstones require consolidation for correctness.<\/li>\n<li>When long-lived segments hinder garbage collection or replication.<\/li>\n<\/ul>\n\n\n\n<p>When it\u2019s optional:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Small datasets with low growth where cost of compaction exceeds benefit.<\/li>\n<li>Short-lived ephemeral logs with short retention that rotate naturally.<\/li>\n<\/ul>\n\n\n\n<p>When NOT to use \/ overuse it:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Over-aggressive compaction during peak traffic windows.<\/li>\n<li>Compaction for the sake of neatness without telemetry or SLO justification.<\/li>\n<li>Rewriting cold immutable archives frequently.<\/li>\n<\/ul>\n\n\n\n<p>Decision checklist:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>If segment count &gt; threshold and read latency increased -&gt; schedule compaction.<\/li>\n<li>If storage cost growth rate &gt; budget and dedup ratio &gt; X -&gt; enable compaction with dedup.<\/li>\n<li>If retention TTL already enforces desired state -&gt; consider light-touch compaction.<\/li>\n<li>If replication lag spikes during compaction -&gt; throttle or shift to low-traffic windows.<\/li>\n<\/ul>\n\n\n\n<p>Maturity ladder:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Beginner: Scheduled nightly compaction with conservative resource caps.<\/li>\n<li>Intermediate: Adaptive compaction based on telemetry with throttling and retries.<\/li>\n<li>Advanced: Smart compaction integrated with autoscaling, cost policies, and predictive scheduling based on ML forecasts.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">How does Compaction work?<\/h2>\n\n\n\n<p>Step-by-step components and workflow:<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Discovery: Identify candidate segments or files for compaction based on policy (age, size, key range).<\/li>\n<li>Plan: Compute the target layout or merge strategy (e.g., key-based upsert merge).<\/li>\n<li>Reserve: Allocate temp storage and I\/O budget; acquire necessary locks or lease.<\/li>\n<li>Read: Stream source segments, applying tombstone, dedup, and retention semantics.<\/li>\n<li>Rewrite: Write compacted output in new segment(s) with consistent indexing.<\/li>\n<li>Swap: Atomically promote new segments and update metadata\/catalog.<\/li>\n<li>Cleanup: Delete old segments and release resources.<\/li>\n<li>Observe: Emit telemetry on throughput, latency, errors, and post-compaction read correctness.<\/li>\n<\/ol>\n\n\n\n<p>Data flow and lifecycle:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Raw writes create append-only segments -&gt; segments age -&gt; compaction selects multiple segments -&gt; produces consolidated segment -&gt; update metadata -&gt; old segments garbage-collected.<\/li>\n<\/ul>\n\n\n\n<p>Edge cases and failure modes:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Partial write failures leaving inconsistent catalogs.<\/li>\n<li>Race conditions between reads and segment deletion.<\/li>\n<li>Tombstone retention causing resurrected deletes if misapplied.<\/li>\n<li>Insufficient temp space leading to aborted compaction.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Typical architecture patterns for Compaction<\/h3>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Background worker compaction: Separate worker process merges segments, good for low interference.<\/li>\n<li>In-place compaction with locking: Single-node compaction with careful locks; good for single-writer systems.<\/li>\n<li>Incremental compaction \/ leveled compaction: Merge levels of increasingly larger segments; used in LSM stores.<\/li>\n<li>Time-window compaction: Compact only within fixed time windows; ideal for time-series.<\/li>\n<li>Tiered compaction with hot-cold separation: Compact hot tier differently than cold tier; good for cost optimization.<\/li>\n<li>Streaming compaction: Continuous small merges using stream processors to avoid large batch jobs.<\/li>\n<\/ol>\n\n\n\n<h3 class=\"wp-block-heading\">Failure modes &amp; mitigation (TABLE REQUIRED)<\/h3>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>ID<\/th>\n<th>Failure mode<\/th>\n<th>Symptom<\/th>\n<th>Likely cause<\/th>\n<th>Mitigation<\/th>\n<th>Observability signal<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>F1<\/td>\n<td>IO saturation<\/td>\n<td>High read\/write latency<\/td>\n<td>Unthrottled compaction<\/td>\n<td>Rate limit compaction<\/td>\n<td>IO wait metric spike<\/td>\n<\/tr>\n<tr>\n<td>F2<\/td>\n<td>Partial write<\/td>\n<td>Corrupt index<\/td>\n<td>Temp space exhausted<\/td>\n<td>Atomic swap with fsync<\/td>\n<td>Error logs during swap<\/td>\n<\/tr>\n<tr>\n<td>F3<\/td>\n<td>Tombstone loss<\/td>\n<td>Deleted items reappear<\/td>\n<td>Incorrect tombstone handling<\/td>\n<td>Add tombstone tests<\/td>\n<td>Data correctness tests fail<\/td>\n<\/tr>\n<tr>\n<td>F4<\/td>\n<td>Network overload<\/td>\n<td>Replication lag<\/td>\n<td>Large compaction traffic<\/td>\n<td>Throttle cross-region traffic<\/td>\n<td>Replication lag metrics<\/td>\n<\/tr>\n<tr>\n<td>F5<\/td>\n<td>Lock contention<\/td>\n<td>Operation timeouts<\/td>\n<td>Long compaction holds locks<\/td>\n<td>Reduce lock granularity<\/td>\n<td>Lock wait traces<\/td>\n<\/tr>\n<tr>\n<td>F6<\/td>\n<td>Out-of-memory<\/td>\n<td>Worker crash<\/td>\n<td>Unbounded in-memory buffering<\/td>\n<td>Stream-based processing<\/td>\n<td>OOM events in logs<\/td>\n<\/tr>\n<tr>\n<td>F7<\/td>\n<td>Thundering compactions<\/td>\n<td>Cluster-wide slowdowns<\/td>\n<td>Synchronized schedules<\/td>\n<td>Stagger compaction windows<\/td>\n<td>Correlated compaction job starts<\/td>\n<\/tr>\n<tr>\n<td>F8<\/td>\n<td>Inconsistent reads<\/td>\n<td>Read errors during swap<\/td>\n<td>Non-atomic metadata update<\/td>\n<td>Use atomic metadata update<\/td>\n<td>Read error rate<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\">Row Details (only if needed)<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>None<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">Key Concepts, Keywords &amp; Terminology for Compaction<\/h2>\n\n\n\n<p>Term \u2014 1\u20132 line definition \u2014 why it matters \u2014 common pitfall<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Append-only log \u2014 Write model where new data is appended \u2014 Simpler recovery and replication \u2014 Pitfall: unbounded growth  <\/li>\n<li>Segment \u2014 A unit file\/partition of appended data \u2014 Compaction works at segment granularity \u2014 Pitfall: tiny segments increase metadata cost  <\/li>\n<li>Tombstone \u2014 Marker indicating deletion \u2014 Ensures deletion survives compaction \u2014 Pitfall: premature tombstone expiry  <\/li>\n<li>Merge \u2014 Combining multiple segments into one \u2014 Reduces fragmentation \u2014 Pitfall: cost spikes during merge  <\/li>\n<li>Levelled compaction \u2014 Compacts segments into hierarchical levels \u2014 Balances read\/write cost \u2014 Pitfall: writes amplified if misconfigured  <\/li>\n<li>Size-tiered compaction \u2014 Merge equal-sized segments \u2014 Simpler but causes large merges \u2014 Pitfall: write amplification  <\/li>\n<li>Deduplication \u2014 Removing duplicate records \u2014 Saves storage \u2014 Pitfall: expensive key comparisons  <\/li>\n<li>Consolidation \u2014 Rewriting with semantic rules \u2014 Improves access patterns \u2014 Pitfall: inconsistent rules across versions  <\/li>\n<li>Snapshot \u2014 Point-in-time copy \u2014 Useful pre-compaction backup \u2014 Pitfall: snapshots consume storage  <\/li>\n<li>Checkpoint \u2014 Persisted state for recovery \u2014 Helps resume compaction \u2014 Pitfall: stale checkpoints  <\/li>\n<li>Atomic swap \u2014 Replace old segments atomically \u2014 Ensures correctness \u2014 Pitfall: non-atomic swaps cause races  <\/li>\n<li>Lease \u2014 Short term ownership token \u2014 Prevents multiple compactors on same target \u2014 Pitfall: stuck leases block progress  <\/li>\n<li>Compaction window \u2014 Time range used for selection \u2014 Controls scope \u2014 Pitfall: too wide causes big jobs  <\/li>\n<li>Compaction throttle \u2014 Limit on I\/O or CPU \u2014 Prevents saturation \u2014 Pitfall: inadequate throttling leaves impact  <\/li>\n<li>Temporary workspace \u2014 Storage used during compaction \u2014 Essential for intermediate writes \u2014 Pitfall: insufficient workspace fails jobs  <\/li>\n<li>Retention policy \u2014 Rules for keeping data \u2014 Guides compaction removal rules \u2014 Pitfall: ambiguous policies cause data loss  <\/li>\n<li>Merge metric \u2014 Metric tracking compaction jobs \u2014 Observability input \u2014 Pitfall: lack of metrics hinders tuning  <\/li>\n<li>Segmentation key \u2014 Key that defines segmentization \u2014 Affects compaction efficiency \u2014 Pitfall: bad key increases cross-segment merges  <\/li>\n<li>Garbage collection \u2014 Removing obsolete segments \u2014 Post-compaction cleanup \u2014 Pitfall: aggressive GC breaks in-progress readers  <\/li>\n<li>Checksum validation \u2014 Verifies segment correctness \u2014 Detects corruption \u2014 Pitfall: disabled checksums hide corruption  <\/li>\n<li>Read-after-write consistency \u2014 Guarantees about visibility \u2014 Compaction must preserve consistency \u2014 Pitfall: stale metadata exposes old data  <\/li>\n<li>Replica set \u2014 Copies across nodes \u2014 Compaction must coordinate across replicas \u2014 Pitfall: inconsistent compaction across replicas  <\/li>\n<li>Snapshot isolation \u2014 Isolation level for reads \u2014 Impacts compaction approach \u2014 Pitfall: reads see partial state  <\/li>\n<li>Write amplification \u2014 Extra writes caused by compaction | A cost metric to optimize | Pitfall: ignoring it increases costs  <\/li>\n<li>Compaction lag \u2014 Time between data write and compaction finishing \u2014 Indicates freshness | Pitfall: too long causes stale storage  <\/li>\n<li>Hot set \u2014 Frequently read or written keys \u2014 May avoid heavy compaction | Pitfall: compacting hot keys can hurt latency  <\/li>\n<li>Cold set \u2014 Rarely accessed data \u2014 Good compaction candidate | Pitfall: overcompacting cold data wastes cycles  <\/li>\n<li>Watermark \u2014 Lower bound for compaction selection | Helps incremental compaction | Pitfall: mis-set watermark stalls compaction  <\/li>\n<li>Index rebuild \u2014 Recreate lookup indices post-compact | Needed for consistency | Pitfall: expensive and forgotten  <\/li>\n<li>Differential merge \u2014 Merge only changed parts | Reduces work | Pitfall: complexity in correctness  <\/li>\n<li>Compactor coordinator \u2014 Component scheduling jobs | Avoids collisions | Pitfall: single point of failure  <\/li>\n<li>Backpressure \u2014 Flow control due to saturation | Compaction must respect it | Pitfall: no backpressure causes outages  <\/li>\n<li>Replayability \u2014 Ability to rebuild state from logs | Enables safe compaction | Pitfall: lost logs prevent rebuild  <\/li>\n<li>Checkpoint replay \u2014 Reapplying checkpoints | Shortens recovery | Pitfall: checkpoints inconsistent with logs  <\/li>\n<li>Consistency model \u2014 Strong vs eventual | Influences compaction design | Pitfall: mismatched expectations  <\/li>\n<li>Replay log \u2014 Source for rebuilding state | Often retained until compaction done | Pitfall: premature truncation  <\/li>\n<li>Time-to-compact \u2014 Duration for compaction job | Operational metric | Pitfall: long jobs lead to maintenance windows  <\/li>\n<li>Compaction policy \u2014 Rules governing selection and merge | Operational control | Pitfall: policies not versioned  <\/li>\n<li>Versioning \u2014 Schema\/version awareness in compactor | Preserves correctness across upgrades | Pitfall: neglecting schema changes  <\/li>\n<li>Metadata catalog \u2014 Tracks segments and versions | Critical for atomic swaps | Pitfall: unatomic updates cause partial visibility  <\/li>\n<li>Cold storage transition \u2014 Moving compacted output to cheaper tier | Cost optimization | Pitfall: retrieval latency ignored  <\/li>\n<li>Rate limiter \u2014 Enforces throughput caps | Protects cluster | Pitfall: too conservative hurts progress<\/li>\n<\/ol>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">How to Measure Compaction (Metrics, SLIs, SLOs) (TABLE REQUIRED)<\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>ID<\/th>\n<th>Metric\/SLI<\/th>\n<th>What it tells you<\/th>\n<th>How to measure<\/th>\n<th>Starting target<\/th>\n<th>Gotchas<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>M1<\/td>\n<td>Compaction throughput<\/td>\n<td>Work per second during compaction<\/td>\n<td>Bytes compacted \/ sec<\/td>\n<td>See details below: M1<\/td>\n<td>See details below: M1<\/td>\n<\/tr>\n<tr>\n<td>M2<\/td>\n<td>Compaction latency<\/td>\n<td>Time to compact unit<\/td>\n<td>End-to-end compaction time<\/td>\n<td>&lt; 30m typical<\/td>\n<td>Large units skew average<\/td>\n<\/tr>\n<tr>\n<td>M3<\/td>\n<td>IO utilization<\/td>\n<td>Disk IO consumed by compaction<\/td>\n<td>IOps and bandwidth per node<\/td>\n<td>&lt; 60% baseline<\/td>\n<td>Other jobs share IO<\/td>\n<\/tr>\n<tr>\n<td>M4<\/td>\n<td>Temp space usage<\/td>\n<td>Workspace used during compaction<\/td>\n<td>Peak temp bytes used<\/td>\n<td>&lt; 50% free disk<\/td>\n<td>Low disk causes failures<\/td>\n<\/tr>\n<tr>\n<td>M5<\/td>\n<td>Read latency impact<\/td>\n<td>Latency regression during compaction<\/td>\n<td>95th read latency delta<\/td>\n<td>&lt; 20% increase<\/td>\n<td>Spiky workloads mask impact<\/td>\n<\/tr>\n<tr>\n<td>M6<\/td>\n<td>Write amplification<\/td>\n<td>Extra bytes written because of compaction<\/td>\n<td>Total writes \/ input writes<\/td>\n<td>&lt; 3x<\/td>\n<td>Depends on compaction strategy<\/td>\n<\/tr>\n<tr>\n<td>M7<\/td>\n<td>Compaction error rate<\/td>\n<td>Failures per compaction job<\/td>\n<td>Failed jobs \/ total jobs<\/td>\n<td>&lt; 1%<\/td>\n<td>Transient network errors inflate rate<\/td>\n<\/tr>\n<tr>\n<td>M8<\/td>\n<td>Tombstone retention correctness<\/td>\n<td>Deleted keys removed properly<\/td>\n<td>Post-check queries for deleted keys<\/td>\n<td>100% correctness<\/td>\n<td>Race with deletes<\/td>\n<\/tr>\n<tr>\n<td>M9<\/td>\n<td>Segment count<\/td>\n<td>Number of active segments<\/td>\n<td>Active files per partition<\/td>\n<td>Target small number<\/td>\n<td>High churn increases count<\/td>\n<\/tr>\n<tr>\n<td>M10<\/td>\n<td>Storage reduction ratio<\/td>\n<td>Reduction after compaction<\/td>\n<td>Pre \/ post size ratio<\/td>\n<td>&gt;= 30% improvement<\/td>\n<td>Low duplication reduces effect<\/td>\n<\/tr>\n<tr>\n<td>M11<\/td>\n<td>Replication lag during compaction<\/td>\n<td>Replicas falling behind<\/td>\n<td>Replica lag in ms<\/td>\n<td>&lt; SLO window<\/td>\n<td>Large compactions spike lag<\/td>\n<\/tr>\n<tr>\n<td>M12<\/td>\n<td>Compaction job concurrency<\/td>\n<td>Number of compaction jobs running<\/td>\n<td>Concurrent jobs metric<\/td>\n<td>Controlled per node<\/td>\n<td>Thundering herd risk<\/td>\n<\/tr>\n<tr>\n<td>M13<\/td>\n<td>Swap atomicity failures<\/td>\n<td>Swap operation errors<\/td>\n<td>Atomic swap failures \/ ops<\/td>\n<td>0<\/td>\n<td>Non-atomic updates create inconsistency<\/td>\n<\/tr>\n<tr>\n<td>M14<\/td>\n<td>Recovery time after compaction failure<\/td>\n<td>MTTR for recovery<\/td>\n<td>Time to recover and resume<\/td>\n<td>&lt; 1 hour<\/td>\n<td>Lack of automation lengthens time<\/td>\n<\/tr>\n<tr>\n<td>M15<\/td>\n<td>Compaction cost per GB<\/td>\n<td>Cloud cost attributed to compaction<\/td>\n<td>Cost \/ GB compacted<\/td>\n<td>Budget bound<\/td>\n<td>Pricing varies by cloud<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\">Row Details (only if needed)<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>M1: Throughput is essential to know duration and capacity planning. Measure with bytes read and written over time, using histograms to capture variance. Consider percentiles as sudden drops indicate throttling.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Best tools to measure Compaction<\/h3>\n\n\n\n<h4 class=\"wp-block-heading\">Tool \u2014 Prometheus<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>What it measures for Compaction: custom metrics, histograms, counters for job durations and failures<\/li>\n<li>Best-fit environment: Kubernetes and cloud-native stacks<\/li>\n<li>Setup outline:<\/li>\n<li>Export compactor metrics via client libraries<\/li>\n<li>Use Pushgateway for batch jobs if needed<\/li>\n<li>Configure scrape intervals and retention<\/li>\n<li>Strengths:<\/li>\n<li>Wide adoption and integration<\/li>\n<li>Powerful alerting with PromQL<\/li>\n<li>Limitations:<\/li>\n<li>Long-term storage needs external solutions<\/li>\n<li>Not ideal for high-cardinality metrics without care<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Tool \u2014 Grafana<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>What it measures for Compaction: dashboards and visualizations of compaction metrics<\/li>\n<li>Best-fit environment: Any environment exposing metrics<\/li>\n<li>Setup outline:<\/li>\n<li>Add Prometheus or other data source<\/li>\n<li>Build dashboards for throughput, latency, errors<\/li>\n<li>Create templated panels for partitions<\/li>\n<li>Strengths:<\/li>\n<li>Flexible visualizations<\/li>\n<li>Alerting integration<\/li>\n<li>Limitations:<\/li>\n<li>No native metric storage<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Tool \u2014 OpenTelemetry<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>What it measures for Compaction: distributed traces of compaction workflows and spans<\/li>\n<li>Best-fit environment: Microservices and distributed compaction pipelines<\/li>\n<li>Setup outline:<\/li>\n<li>Instrument compactor code with spans<\/li>\n<li>Export traces to supported backend<\/li>\n<li>Correlate traces with metrics and logs<\/li>\n<li>Strengths:<\/li>\n<li>Trace-based diagnostics for multi-step jobs<\/li>\n<li>Limitations:<\/li>\n<li>Sampling may miss rare failures<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Tool \u2014 Cloud provider native metrics (AWS CloudWatch \/ GCP Monitoring)<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>What it measures for Compaction: VM\/IO metrics and managed service compaction telemetry<\/li>\n<li>Best-fit environment: Managed services and cloud VMs<\/li>\n<li>Setup outline:<\/li>\n<li>Enable service-level metrics<\/li>\n<li>Create dashboards and alerts in provider console<\/li>\n<li>Strengths:<\/li>\n<li>Near-source telemetry<\/li>\n<li>Limitations:<\/li>\n<li>Cost and noisy metrics<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Tool \u2014 Logging and ELK \/ Loki<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>What it measures for Compaction: job logs, errors, swap confirmations<\/li>\n<li>Best-fit environment: Any environment producing logs<\/li>\n<li>Setup outline:<\/li>\n<li>Tag logs with compaction job IDs<\/li>\n<li>Centralize logs and create parsers<\/li>\n<li>Build alerts on patterns<\/li>\n<li>Strengths:<\/li>\n<li>Rich text logs for postmortem<\/li>\n<li>Limitations:<\/li>\n<li>Parsing and noise management required<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\">Tool \u2014 Cost &amp; Billing tools<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>What it measures for Compaction: cost attributable to compaction IO and storage transitions<\/li>\n<li>Best-fit environment: Cloud cost-aware teams<\/li>\n<li>Setup outline:<\/li>\n<li>Tag resources and attribute costs per job<\/li>\n<li>Track per-GB compaction cost<\/li>\n<li>Strengths:<\/li>\n<li>Direct financial insight<\/li>\n<li>Limitations:<\/li>\n<li>Attribution can be complex<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Recommended dashboards &amp; alerts for Compaction<\/h3>\n\n\n\n<p>Executive dashboard:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Panels: Total storage reduction month-to-date; Cost savings from compaction; High-level compaction success rate.<\/li>\n<li>Why: Business stakeholders need ROI and risk overview.<\/li>\n<\/ul>\n\n\n\n<p>On-call dashboard:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Panels: Active compaction jobs per node; Compaction error rate; Read latency 95\/99 percentiles during compaction; Replication lag.<\/li>\n<li>Why: Helps quickly identify compaction-induced incidents.<\/li>\n<\/ul>\n\n\n\n<p>Debug dashboard:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Panels: Job-level traces and logs; Temp space usage over time; Per-partition compaction throughput; Swap operation latency; Tombstone counts.<\/li>\n<li>Why: Deep dive into failing compaction jobs.<\/li>\n<\/ul>\n\n\n\n<p>Alerting guidance:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Page vs ticket: Page for high-severity signals like compaction causing SLO breach, replication lag beyond SLO, or atomic swap failures. Create tickets for non-urgent failures like a single job retryable failure.<\/li>\n<li>Burn-rate guidance: If compaction-induced errors consume &gt;20% of error budget in 1 hour, page stakeholders.<\/li>\n<li>Noise reduction: Deduplicate alerts by job ID, group alerts by affected cluster, suppress alerts during planned maintenance windows.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">Implementation Guide (Step-by-step)<\/h2>\n\n\n\n<p>1) Prerequisites\n&#8211; Define compaction policy and SLOs.\n&#8211; Ensure adequate temp storage and I\/O capacity.\n&#8211; Versioned metadata catalog and atomic update capability.\n&#8211; Observability stack for metrics, logs, and traces.<\/p>\n\n\n\n<p>2) Instrumentation plan\n&#8211; Emit metrics: job start, end, bytes read\/written, errors.\n&#8211; Add tracing spans for planning, read, write, swap, cleanup.\n&#8211; Export tombstone and retention metrics.<\/p>\n\n\n\n<p>3) Data collection\n&#8211; Centralize logs and metrics.\n&#8211; Tag by partition, node, and job ID.\n&#8211; Retain historical metrics for trend analysis.<\/p>\n\n\n\n<p>4) SLO design\n&#8211; Define acceptable compaction latency and error rate.\n&#8211; Map to storage-read and write impact SLOs.\n&#8211; Define alert thresholds tied to error budget.<\/p>\n\n\n\n<p>5) Dashboards\n&#8211; Build executive, on-call, and debug dashboards.\n&#8211; Include pre\/post compaction comparisons.<\/p>\n\n\n\n<p>6) Alerts &amp; routing\n&#8211; Define paging criteria and escalation paths.\n&#8211; Integrate with incident management systems.<\/p>\n\n\n\n<p>7) Runbooks &amp; automation\n&#8211; Create runbooks for common failures and recovery steps.\n&#8211; Automate retries, throttling, and swap verification.<\/p>\n\n\n\n<p>8) Validation (load\/chaos\/game days)\n&#8211; Run load tests to simulate compaction under load.\n&#8211; Simulate failures: disk full, network partition, node restart.\n&#8211; Verify rollbacks and data correctness.<\/p>\n\n\n\n<p>9) Continuous improvement\n&#8211; Review compaction metrics weekly.\n&#8211; Tune thresholds and schedules.\n&#8211; Maintain versioned compaction policies.<\/p>\n\n\n\n<p>Pre-production checklist:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Validate compaction correctness on a copy.<\/li>\n<li>Verify atomic swaps and metadata updates.<\/li>\n<li>Ensure metrics emitted and dashboards present.<\/li>\n<li>Test failure injection for temp space exhaustion.<\/li>\n<li>Confirm rollback and resume behavior.<\/li>\n<\/ul>\n\n\n\n<p>Production readiness checklist:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Throttling configured and tested.<\/li>\n<li>Temp storage capacity verified per node.<\/li>\n<li>Alerting and runbooks in place.<\/li>\n<li>Scheduled windows and stagger strategy defined.<\/li>\n<li>Backup or snapshot plan before large compactions.<\/li>\n<\/ul>\n\n\n\n<p>Incident checklist specific to Compaction:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Identify impacted partitions and compaction job IDs.<\/li>\n<li>Check temp disk and IO metrics.<\/li>\n<li>If causing SLO breach, pause compactions and escalate.<\/li>\n<li>Inspect logs for swap atomicity and tombstone handling.<\/li>\n<li>Restore from snapshot if data corruption suspected.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">Use Cases of Compaction<\/h2>\n\n\n\n<ol class=\"wp-block-list\">\n<li>\n<p>Time-series DB retention compaction\n&#8211; Context: High-cardinality metrics over time.\n&#8211; Problem: Storage growth and slow queries.\n&#8211; Why: Consolidates data and removes expired points.\n&#8211; What to measure: Storage reduction, query latency.\n&#8211; Typical tools: Prometheus Cortex, Thanos.<\/p>\n<\/li>\n<li>\n<p>Log store key-value compaction\n&#8211; Context: Event-sourced systems using log stores.\n&#8211; Problem: Log growth and slow recovery.\n&#8211; Why: Keep latest state per key and remove old events for size control.\n&#8211; What to measure: Segment count, write amplification.\n&#8211; Typical tools: Kafka with log compaction, RocksDB.<\/p>\n<\/li>\n<li>\n<p>Message broker segment compaction\n&#8211; Context: Brokers retaining latest messages per key.\n&#8211; Problem: Consumer startup time due to many segments.\n&#8211; Why: Improves startup by reducing segments.\n&#8211; What to measure: Compaction throughput and replication lag.\n&#8211; Typical tools: Kafka, Pulsar.<\/p>\n<\/li>\n<li>\n<p>Column-store merge for analytics\n&#8211; Context: Data warehouse ingest with small files.\n&#8211; Problem: Many small files hurt query performance.\n&#8211; Why: Merge small files into bigger optimized files.\n&#8211; What to measure: Query latency and file count.\n&#8211; Typical tools: Parquet compaction via Spark or Flink.<\/p>\n<\/li>\n<li>\n<p>Mobile edge device log compaction\n&#8211; Context: Devices with limited flash.\n&#8211; Problem: Limited space and costly sync bandwidth.\n&#8211; Why: Compact before sync to reduce network cost.\n&#8211; What to measure: Sync payload size, flash wear metrics.\n&#8211; Typical tools: Device agents with local compaction.<\/p>\n<\/li>\n<li>\n<p>Backup snapshot compaction\n&#8211; Context: Frequent incremental backups.\n&#8211; Problem: Snapshot store growth and restore slowness.\n&#8211; Why: Consolidate incremental deltas into full images.\n&#8211; What to measure: Restore time and storage cost.\n&#8211; Typical tools: Backup tools with compaction routines.<\/p>\n<\/li>\n<li>\n<p>Cache consolidation\n&#8211; Context: Distributed caches with fragmented segments.\n&#8211; Problem: High miss rate due to fragmentation.\n&#8211; Why: Re-layout for locality and evict expired keys.\n&#8211; What to measure: Hit rate, eviction churn.\n&#8211; Typical tools: Redis clusters with GC.<\/p>\n<\/li>\n<li>\n<p>Cold-tier optimization\n&#8211; Context: Object store cold tier with redundant objects.\n&#8211; Problem: Cost with duplicate objects across partitions.\n&#8211; Why: Deduplicate and transition to archival storage.\n&#8211; What to measure: Archival cost savings, retrieval latency.\n&#8211; Typical tools: Object lifecycle policies + compaction jobs.<\/p>\n<\/li>\n<li>\n<p>Compliance log retention\n&#8211; Context: Legal retention with deletions.\n&#8211; Problem: Deletions must persist across rewrites.\n&#8211; Why: Compaction honors tombstones ensuring compliance.\n&#8211; What to measure: Tombstone correctness, audit logs.\n&#8211; Typical tools: Audit log stores with compaction policies.<\/p>\n<\/li>\n<li>\n<p>Stream processor state store maintenance\n&#8211; Context: Stateful stream processing with local stores.\n&#8211; Problem: Growing state cause slow processing and restarts.\n&#8211; Why: Compaction reduces store size and speeds up recovery.\n&#8211; What to measure: State store size and restore time.\n&#8211; Typical tools: RocksDB in stream processors.<\/p>\n<\/li>\n<\/ol>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">Scenario Examples (Realistic, End-to-End)<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">Scenario #1 \u2014 Kubernetes StatefulSet log compaction<\/h3>\n\n\n\n<p><strong>Context:<\/strong> StatefulSet producing append-only logs stored as per-pod local segments.\n<strong>Goal:<\/strong> Reduce node disk pressure and speed pod restarts.\n<strong>Why Compaction matters here:<\/strong> Local segments can grow unbounded and cause OOM or node eviction.\n<strong>Architecture \/ workflow:<\/strong> DaemonSet compactor scans per-pod directories, compacts segments, writes new compacted segment, updates per-pod metadata, cleans old files.\n<strong>Step-by-step implementation:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Schedule compactor DaemonSet with low CPU limit.<\/li>\n<li>Use hostPath read-only locks and a lease per pod.<\/li>\n<li>Emit Prometheus metrics for throughput and errors.<\/li>\n<li>Throttle compaction during high pod CPU usage.\n<strong>What to measure:<\/strong> Node disk free, compaction jobs per node, read latency.\n<strong>Tools to use and why:<\/strong> Kubernetes CronJob\/DaemonSet, Prometheus, Grafana for metrics.\n<strong>Common pitfalls:<\/strong> HostPath permission errors; compactor causing node disk pressure.\n<strong>Validation:<\/strong> Run chaos tests killing compactor and pods; verify restart times.\n<strong>Outcome:<\/strong> Reduced node disk usage and faster pod restarts.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Scenario #2 \u2014 Serverless managed-PaaS compaction for event store<\/h3>\n\n\n\n<p><strong>Context:<\/strong> Managed event store in serverless PaaS retaining last value per key.\n<strong>Goal:<\/strong> Control storage costs and ensure low-latency lookups.\n<strong>Why Compaction matters here:<\/strong> Serverless billing tied to storage; compaction reduces cost.\n<strong>Architecture \/ workflow:<\/strong> Managed compaction triggers based on partition size; service exposes compaction metrics.\n<strong>Step-by-step implementation:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Configure retention and compaction windows via service console.<\/li>\n<li>Monitor native compaction metrics and set alerts.<\/li>\n<li>Validate tombstone handling with test deletes.\n<strong>What to measure:<\/strong> Storage reduction ratio, compaction error rate.\n<strong>Tools to use and why:<\/strong> Managed PaaS compaction features, Cloud monitoring.\n<strong>Common pitfalls:<\/strong> Platform-specific behavior varies \/ Not publicly stated.\n<strong>Validation:<\/strong> Run load test with writes, deletes, and verify correctness.\n<strong>Outcome:<\/strong> Lower storage bills and maintained lookup performance.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Scenario #3 \u2014 Incident-response: Compaction caused replication outage<\/h3>\n\n\n\n<p><strong>Context:<\/strong> Large compaction job triggered during peak leading to replication lag and timeouts.\n<strong>Goal:<\/strong> Restore service and prevent recurrence.\n<strong>Why Compaction matters here:<\/strong> Compaction overloaded network and caused failover.\n<strong>Architecture \/ workflow:<\/strong> Compaction jobs run across multiple nodes sharing network links.\n<strong>Step-by-step implementation:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Immediately pause compactions via coordinator.<\/li>\n<li>Scale replicas and prioritize replication traffic.<\/li>\n<li>Inspect logs for swap failures and tombstone issues.<\/li>\n<li>Open postmortem and add throttle\/maintenance window.\n<strong>What to measure:<\/strong> Replication lag, number of paused\/running compactions.\n<strong>Tools to use and why:<\/strong> Monitoring, runbook, incident management.\n<strong>Common pitfalls:<\/strong> No compaction pause mechanism; lack of automated throttling.\n<strong>Validation:<\/strong> Simulate traffic during compaction in staging.\n<strong>Outcome:<\/strong> Incident resolved and throttling added.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Scenario #4 \u2014 Cost vs performance trade-off compaction<\/h3>\n\n\n\n<p><strong>Context:<\/strong> Data warehouse with frequent small files leads to high query latencies but compaction costs significant compute.\n<strong>Goal:<\/strong> Find balance between compaction frequency and query performance.\n<strong>Why Compaction matters here:<\/strong> Merging small files improves scan throughput but is costly.\n<strong>Architecture \/ workflow:<\/strong> Scheduled nightly compaction of ingestion buckets with staged merges.\n<strong>Step-by-step implementation:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Measure query latency improvement vs compaction compute cost.<\/li>\n<li>Adopt incremental compaction of hot partitions.<\/li>\n<li>Move cold partitions to cheaper archival storage without frequent compaction.\n<strong>What to measure:<\/strong> Cost per query, compaction cost per GB, average query latency.\n<strong>Tools to use and why:<\/strong> Spark for merge jobs, cost dashboards.\n<strong>Common pitfalls:<\/strong> Overcompacting cold partitions yields little benefit.\n<strong>Validation:<\/strong> A\/B testing with production queries.\n<strong>Outcome:<\/strong> Optimized schedule with net positive ROI.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Scenario #5 \u2014 Stream processor state compaction on RocksDB<\/h3>\n\n\n\n<p><strong>Context:<\/strong> Flink\/Faust state backend using RocksDB with many versions per key.\n<strong>Goal:<\/strong> Reduce state size and job restore time after failure.\n<strong>Why Compaction matters here:<\/strong> Smaller state reduces restore time during failover.\n<strong>Architecture \/ workflow:<\/strong> RocksDB compaction scheduled via configuration and adaptive triggers.\n<strong>Step-by-step implementation:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Configure RocksDB compaction style and target size ratios.<\/li>\n<li>Monitor compaction metrics and job restore times.<\/li>\n<li>Test stateful job upgrades with compaction enabled.\n<strong>What to measure:<\/strong> State size, restore duration, write amplification.\n<strong>Tools to use and why:<\/strong> RocksDB metrics, Flink metrics, Prometheus.\n<strong>Common pitfalls:<\/strong> Incompatible compaction settings across versions.\n<strong>Validation:<\/strong> Forced failover and measure restore latency.\n<strong>Outcome:<\/strong> Faster restorations and controlled state size.<\/li>\n<\/ul>\n\n\n\n<h3 class=\"wp-block-heading\">Scenario #6 \u2014 Mobile edge device compaction before sync<\/h3>\n\n\n\n<p><strong>Context:<\/strong> Consumer devices with intermittent connectivity sync logs to cloud.\n<strong>Goal:<\/strong> Reduce sync payload and sync time.\n<strong>Why Compaction matters here:<\/strong> Minimize bandwidth and improve user experience.\n<strong>Architecture \/ workflow:<\/strong> Local compactor prunes duplicates, compresses, and packages for sync.\n<strong>Step-by-step implementation:<\/strong><\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Implement compaction agent with periodic and on-sync triggers.<\/li>\n<li>Validate correctness with conflict resolution tests.<\/li>\n<li>Monitor sync payload and battery impact.\n<strong>What to measure:<\/strong> Sync payload size, sync success rate, battery usage.\n<strong>Tools to use and why:<\/strong> Device telemetry and server-side ingestion guards.\n<strong>Common pitfalls:<\/strong> Compaction causing CPU spikes affecting battery.\n<strong>Validation:<\/strong> Field testing with targeted user base.\n<strong>Outcome:<\/strong> Lower sync costs and improved user experience.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">Common Mistakes, Anti-patterns, and Troubleshooting<\/h2>\n\n\n\n<p>List of mistakes with Symptom -&gt; Root cause -&gt; Fix (15\u201325 items, including observability pitfalls)<\/p>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Symptom: Spike in read latency during compaction -&gt; Root cause: No throttling on compaction IO -&gt; Fix: Implement IO rate limiting and schedule during low-traffic windows  <\/li>\n<li>Symptom: Deleted keys reappear -&gt; Root cause: Tombstone handling bug -&gt; Fix: Add tombstone unit tests and retention enforcement  <\/li>\n<li>Symptom: Compaction job fails with disk full -&gt; Root cause: Insufficient temp workspace -&gt; Fix: Pre-check free disk and reserve quota before job start  <\/li>\n<li>Symptom: Numerous small files after compaction -&gt; Root cause: Incorrect merge policy creating tiny outputs -&gt; Fix: Tune target file size and merge heuristics  <\/li>\n<li>Symptom: Compaction orchestrator crash halts process -&gt; Root cause: Single point of failure -&gt; Fix: Add leader election and failover for coordinator  <\/li>\n<li>Symptom: High CPU usage across nodes -&gt; Root cause: Unbounded parallel compaction concurrency -&gt; Fix: Set concurrency limits per node and global cap  <\/li>\n<li>Symptom: Replication lag spikes -&gt; Root cause: Large cross-region compaction traffic -&gt; Fix: Throttle cross-region transfers and prioritize replication traffic  <\/li>\n<li>Symptom: Swap operation errors -&gt; Root cause: Non-atomic metadata updates -&gt; Fix: Implement atomic metadata writes and verify with checksums  <\/li>\n<li>Symptom: Metrics missing for compaction jobs -&gt; Root cause: Poor instrumentation -&gt; Fix: Add required counters, histograms, and correlation IDs  <\/li>\n<li>Symptom: Alerts flood during scheduled maintenance -&gt; Root cause: No maintenance window suppression -&gt; Fix: Configure alert suppression during planned compaction windows (observability pitfall)  <\/li>\n<li>Symptom: Post-compaction inconsistent search results -&gt; Root cause: Index rebuild skipped -&gt; Fix: Rebuild and validate index after compaction  <\/li>\n<li>Symptom: Thundering herd of compactions -&gt; Root cause: Synchronized schedules on many partitions -&gt; Fix: Stagger scheduling and random jitter  <\/li>\n<li>Symptom: Long recovery after compaction interruption -&gt; Root cause: Lack of resumable compaction -&gt; Fix: Support resumable jobs with checkpoints  <\/li>\n<li>Symptom: Unexpected cost increase -&gt; Root cause: Compaction compute billed in expensive region -&gt; Fix: Move heavy compaction jobs to cheaper zones or off-peak hours (cost pitfall)  <\/li>\n<li>Symptom: Compaction not reducing storage -&gt; Root cause: Low duplication and poor selection criteria -&gt; Fix: Re-evaluate policies and target segments with high duplication  <\/li>\n<li>Symptom: Unauthorized compaction changes -&gt; Root cause: Poor access controls for compactor -&gt; Fix: Enforce RBAC and authenticated job submission (security pitfall)  <\/li>\n<li>Symptom: Logs show intermittent swap failures -&gt; Root cause: NFS or shared filesystem semantics differ -&gt; Fix: Use filesystem with atomic rename guarantees or coordination layer  <\/li>\n<li>Symptom: Lack of visibility into job progress -&gt; Root cause: No tracing or progress metrics -&gt; Fix: Add progress percentages and span instrumentation (observability pitfall)  <\/li>\n<li>Symptom: High write amplification -&gt; Root cause: Wrong compaction style for workload -&gt; Fix: Switch compaction style or tune thresholds  <\/li>\n<li>Symptom: Compaction causing crashes on nodes -&gt; Root cause: Memory leaks in compactor process -&gt; Fix: Use streaming compaction and monitor memory, add limits (observability pitfall)  <\/li>\n<li>Symptom: Compaction stalls for certain partitions -&gt; Root cause: Hot partition locking -&gt; Fix: Use lock sharding and fine-grained locking  <\/li>\n<li>Symptom: Compaction jobs never end -&gt; Root cause: Infinite retry loop on transient errors -&gt; Fix: Exponential backoff and max retries with alerting  <\/li>\n<li>Symptom: Compliance audit fails -&gt; Root cause: Tombstone expiry misconfiguration -&gt; Fix: Audit tombstone retention and add verification jobs  <\/li>\n<li>Symptom: Compaction causes CPU steal on VMs -&gt; Root cause: Oversubscription on host -&gt; Fix: Set resource requests and limits in orchestrator<\/li>\n<\/ol>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">Best Practices &amp; Operating Model<\/h2>\n\n\n\n<p>Ownership and on-call:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Assign compaction ownership to storage or platform team.<\/li>\n<li>Define clear on-call rotations for compaction incidents.<\/li>\n<li>Create a compaction runbook accessible to on-call engineers.<\/li>\n<\/ul>\n\n\n\n<p>Runbooks vs playbooks:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Runbooks: Step-by-step actions for known failure modes.<\/li>\n<li>Playbooks: Higher-level decision trees for ambiguous incidents.<\/li>\n<\/ul>\n\n\n\n<p>Safe deployments:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Canary compaction settings on subset of partitions first.<\/li>\n<li>Provide rollback switches and throttles.<\/li>\n<\/ul>\n\n\n\n<p>Toil reduction and automation:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Automate scheduling, retries, and cleanup.<\/li>\n<li>Use policies rather than ad-hoc scripts.<\/li>\n<\/ul>\n\n\n\n<p>Security basics:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Enforce RBAC for compaction job submission.<\/li>\n<li>Validate artifacts with checksums and signed metadata.<\/li>\n<li>Encrypt temp workspace where required.<\/li>\n<\/ul>\n\n\n\n<p>Weekly\/monthly routines:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Weekly: Review compaction success rate and throughput trends.<\/li>\n<li>Monthly: Review policies and cost impact; test compaction resume scenarios.<\/li>\n<\/ul>\n\n\n\n<p>Postmortem reviews:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Always check whether compaction contributed to incident.<\/li>\n<li>Review throttling and scheduling decisions and incorporate into runbooks.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">Tooling &amp; Integration Map for Compaction (TABLE REQUIRED)<\/h2>\n\n\n\n<figure class=\"wp-block-table\"><table>\n<thead>\n<tr>\n<th>ID<\/th>\n<th>Category<\/th>\n<th>What it does<\/th>\n<th>Key integrations<\/th>\n<th>Notes<\/th>\n<\/tr>\n<\/thead>\n<tbody>\n<tr>\n<td>I1<\/td>\n<td>Metrics backend<\/td>\n<td>Stores compaction metrics<\/td>\n<td>Prometheus Grafana<\/td>\n<td>Short-term retention recommended<\/td>\n<\/tr>\n<tr>\n<td>I2<\/td>\n<td>Tracing<\/td>\n<td>Tracks compaction workflow spans<\/td>\n<td>OpenTelemetry backends<\/td>\n<td>Useful for multi-step jobs<\/td>\n<\/tr>\n<tr>\n<td>I3<\/td>\n<td>Logging<\/td>\n<td>Centralizes compaction logs<\/td>\n<td>ELK Loki<\/td>\n<td>Correlate with job IDs<\/td>\n<\/tr>\n<tr>\n<td>I4<\/td>\n<td>Orchestration<\/td>\n<td>Schedules compaction jobs<\/td>\n<td>Kubernetes CronJob<\/td>\n<td>Supports quotas and limits<\/td>\n<\/tr>\n<tr>\n<td>I5<\/td>\n<td>Storage engine<\/td>\n<td>Performs on-disk compaction<\/td>\n<td>RocksDB LevelDB<\/td>\n<td>Embedded compactor controls<\/td>\n<\/tr>\n<tr>\n<td>I6<\/td>\n<td>Message broker<\/td>\n<td>Provides log compaction features<\/td>\n<td>Kafka Pulsar<\/td>\n<td>Broker-level compaction<\/td>\n<\/tr>\n<tr>\n<td>I7<\/td>\n<td>Cloud monitoring<\/td>\n<td>Cloud native metrics for managed compaction<\/td>\n<td>CloudWatch Monitoring<\/td>\n<td>Varies per provider<\/td>\n<\/tr>\n<tr>\n<td>I8<\/td>\n<td>Cost\/FinOps<\/td>\n<td>Attributes compaction costs<\/td>\n<td>Billing APIs<\/td>\n<td>Tagging is critical<\/td>\n<\/tr>\n<tr>\n<td>I9<\/td>\n<td>Backup systems<\/td>\n<td>Snapshot before major compactions<\/td>\n<td>Backup solutions<\/td>\n<td>Ensure compatibility<\/td>\n<\/tr>\n<tr>\n<td>I10<\/td>\n<td>CI\/CD<\/td>\n<td>Deploy compaction configs safely<\/td>\n<td>GitOps pipelines<\/td>\n<td>Version compaction policies<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n<h4 class=\"wp-block-heading\">Row Details (only if needed)<\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>None<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">Frequently Asked Questions (FAQs)<\/h2>\n\n\n\n<h3 class=\"wp-block-heading\">H3: What is the difference between compaction and compression?<\/h3>\n\n\n\n<p>Compaction restructures or deduplicates data based on semantics while compression reduces bytes without changing structure.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">H3: Can I run compaction during peak traffic?<\/h3>\n\n\n\n<p>You can, but it&#8217;s risky; prefer throttling and adaptive scheduling to avoid SLO breaches.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">H3: How much temp storage do I need?<\/h3>\n\n\n\n<p>Varies \/ depends; plan for at least the size of data being compacted plus headroom, often 20\u201350%.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">H3: Does compaction affect consistency?<\/h3>\n\n\n\n<p>It can if not atomic; design for atomic swaps and validate tombstone semantics to preserve consistency.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">H3: How often should compaction run?<\/h3>\n\n\n\n<p>Depends on workload; start with daily for high-write systems and tune based on metrics.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">H3: Will compaction reduce my cloud bills?<\/h3>\n\n\n\n<p>Yes for storage-heavy workloads with duplication; measure compaction cost vs storage savings.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">H3: Is compaction automatic in managed services?<\/h3>\n\n\n\n<p>Varies \/ depends; many managed services offer auto-compaction but behavior differs.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">H3: How do I test compaction safely?<\/h3>\n\n\n\n<p>Run compaction on a copy of production data and inject failure scenarios in staging.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">H3: What telemetry is essential for compaction?<\/h3>\n\n\n\n<p>Throughput, latency, temp space, IO utilization, error rate, and swap atomicity.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">H3: How do I avoid compaction causing outages?<\/h3>\n\n\n\n<p>Throttle compaction, stagger jobs, use maintenance windows, and monitor impact.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">H3: Can compaction be resumable?<\/h3>\n\n\n\n<p>Yes; resumable or checkpointed compaction reduces risk of long retries.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">H3: What are common security concerns?<\/h3>\n\n\n\n<p>Unauthorized compaction, data leakage in temp storage, and improper access to compaction metadata.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">H3: How does compaction affect backups?<\/h3>\n\n\n\n<p>Compaction can change data layout; ensure backups reference consistent snapshots before compaction.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">H3: Should compaction be part of the CI pipeline?<\/h3>\n\n\n\n<p>Yes for config changes; test compaction policies and code changes via CI to prevent regressions.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">H3: How to correlate compaction with incidents?<\/h3>\n\n\n\n<p>Use job IDs in logs and metrics and add tracing spans to correlate compaction steps with errors.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">H3: Are there ML approaches to scheduling compaction?<\/h3>\n\n\n\n<p>Yes; predictive scheduling using workload forecasts can optimize when compaction runs.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">H3: How to handle schema changes during compaction?<\/h3>\n\n\n\n<p>Add version-awareness to compaction logic and perform migration steps with compatibility checks.<\/p>\n\n\n\n<h3 class=\"wp-block-heading\">H3: What SLIs should be prioritized?<\/h3>\n\n\n\n<p>Compaction error rate and read latency impact are typically highest priority.<\/p>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">Conclusion<\/h2>\n\n\n\n<p>Compaction is a foundational maintenance capability that reduces storage costs, improves performance, and preserves data correctness when implemented with careful policies, observability, and operational controls. Treat compaction as a first-class operational concern with SLOs, automated orchestration, and proper testing.<\/p>\n\n\n\n<p>Next 7 days plan:<\/p>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Day 1: Inventory where compaction applies and collect existing metrics.<\/li>\n<li>Day 2: Define compaction policies and SLOs with stakeholders.<\/li>\n<li>Day 3: Implement metrics and tracing for one compaction job.<\/li>\n<li>Day 4: Run compaction on non-prod data and validate correctness.<\/li>\n<li>Day 5: Build dashboards and set alerts; add runbooks.<\/li>\n<li>Day 6: Schedule canary compaction in production with throttles.<\/li>\n<li>Day 7: Review outcomes, tweak policy, and document lessons learned.<\/li>\n<\/ul>\n\n\n\n<hr class=\"wp-block-separator\" \/>\n\n\n\n<h2 class=\"wp-block-heading\">Appendix \u2014 Compaction Keyword Cluster (SEO)<\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Primary keywords<\/li>\n<li>Compaction<\/li>\n<li>Data compaction<\/li>\n<li>Log compaction<\/li>\n<li>Storage compaction<\/li>\n<li>Compaction policy<\/li>\n<li>Compaction scheduling<\/li>\n<li>Compaction metrics<\/li>\n<li>Compaction best practices<\/li>\n<li>Compaction architecture<\/li>\n<li>\n<p>Compaction troubleshooting<\/p>\n<\/li>\n<li>\n<p>Secondary keywords<\/p>\n<\/li>\n<li>Tombstone handling<\/li>\n<li>Segment compaction<\/li>\n<li>Background compaction<\/li>\n<li>Incremental compaction<\/li>\n<li>Leveled compaction<\/li>\n<li>Size-tiered compaction<\/li>\n<li>Compaction throughput<\/li>\n<li>Compaction latency<\/li>\n<li>Compaction throttle<\/li>\n<li>\n<p>Compaction runbook<\/p>\n<\/li>\n<li>\n<p>Long-tail questions<\/p>\n<\/li>\n<li>What is compaction in storage systems<\/li>\n<li>How does log compaction work<\/li>\n<li>How to measure compaction impact on latency<\/li>\n<li>When should I run compaction jobs<\/li>\n<li>How to avoid compaction causing outages<\/li>\n<li>How to test compaction safely in staging<\/li>\n<li>What metrics indicate compaction success<\/li>\n<li>How to design compaction policies for time-series data<\/li>\n<li>How to resume interrupted compaction jobs<\/li>\n<li>How compaction affects replication lag<\/li>\n<li>How compaction differs from compression and garbage collection<\/li>\n<li>How to attribute compaction costs in cloud billing<\/li>\n<li>How to handle tombstones during compaction<\/li>\n<li>How to implement atomic swap for compaction<\/li>\n<li>\n<p>How to compact RocksDB state stores<\/p>\n<\/li>\n<li>\n<p>Related terminology<\/p>\n<\/li>\n<li>Append-only log<\/li>\n<li>Segment file<\/li>\n<li>Tombstone marker<\/li>\n<li>Merge strategy<\/li>\n<li>Temp workspace<\/li>\n<li>Atomic swap<\/li>\n<li>Checkpointing<\/li>\n<li>Snapshot isolation<\/li>\n<li>Replica lag<\/li>\n<li>Write amplification<\/li>\n<li>Read-after-write consistency<\/li>\n<li>Compaction coordinator<\/li>\n<li>Retention policy<\/li>\n<li>Metadata catalog<\/li>\n<li>Compaction window<\/li>\n<li>Rate limiter<\/li>\n<li>Hot set and cold set<\/li>\n<li>Index rebuild<\/li>\n<li>Differential merge<\/li>\n<li>Compaction policy versioning<\/li>\n<li>Compaction orchestration<\/li>\n<li>Compaction scheduler<\/li>\n<li>Compaction concurrency<\/li>\n<li>Compaction cost per GB<\/li>\n<li>Compaction observability<\/li>\n<li>Compaction runbook<\/li>\n<li>Compaction playbook<\/li>\n<li>Compaction incident response<\/li>\n<li>Compaction validation<\/li>\n<\/ul>\n","protected":false},"excerpt":{"rendered":"<p>&#8212;<\/p>\n","protected":false},"author":5,"featured_media":0,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"footnotes":""},"categories":[],"tags":[],"class_list":["post-3662","post","type-post","status-publish","format-standard","hentry"],"_links":{"self":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/3662","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/users\/5"}],"replies":[{"embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/comments?post=3662"}],"version-history":[{"count":0,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/posts\/3662\/revisions"}],"wp:attachment":[{"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/media?parent=3662"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/categories?post=3662"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/dataopsschool.com\/blog\/wp-json\/wp\/v2\/tags?post=3662"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}