{"id":2435,"date":"2026-02-17T08:08:48","date_gmt":"2026-02-17T08:08:48","guid":{"rendered":"https:\/\/dataopsschool.com\/blog\/nmi\/"},"modified":"2026-02-17T15:32:08","modified_gmt":"2026-02-17T15:32:08","slug":"nmi","status":"publish","type":"post","link":"https:\/\/dataopsschool.com\/blog\/nmi\/","title":{"rendered":"What is NMI? Meaning, Architecture, Examples, Use Cases, and How to Measure It (2026 Guide)"},"content":{"rendered":"\n\n\n\n\n
Quick Definition (30\u201360 words)<\/h2>\n\n\n\n
Non-Maskable Interrupt (NMI) is a hardware-level interrupt that cannot be ignored by the CPU and signals critical conditions like hardware faults or watchdog timeouts. Analogy: an emergency alarm that always breaks through whatever you\u2019re doing. Formal: a high-priority, non-discardable interrupt vector delivered to the processor for immediate handling.<\/p>\n\n\n\n
\n\n\n\n
What is NMI?<\/h2>\n\n\n\n
\n
\n
What it is \/ what it is NOT\n NMI is a hardware-initiated interrupt line intended for critical, high-severity events that require immediate attention from the CPU and operating system. It is NOT a regular software interrupt, nor is it generally used for routine telemetry or graceful shutdowns.<\/p>\n<\/li>\n
Handler execution is constrained by the CPU and OS context; reentrancy and safety are concerns. <\/li>\n
Can indicate unrecoverable conditions (e.g., memory corruption) or survivable diagnostics (e.g., watchdog timeout). <\/li>\n
\n
Behavior can vary by platform, hypervisor, and BIOS\/UEFI firmware.<\/p>\n<\/li>\n
\n
Where it fits in modern cloud\/SRE workflows <\/p>\n<\/li>\n
Troubleshooting root-cause hardware incidents on hosts (bare metal or VMs with passthrough). <\/li>\n
Detecting stuck CPUs and kernel deadlocks via NMI watchdogs. <\/li>\n
Integrating with incident response for host-level severity incidents and correlated observability. <\/li>\n
\n
Postmortems and capacity planning when hardware error rates rise.<\/p>\n<\/li>\n
\n
A text-only \u201cdiagram description\u201d readers can visualize <\/p>\n<\/li>\n
Physical hardware sources (watchdog chip, thermal sensor, PCI device) send NMI signal to CPU(s). <\/li>\n
CPU interrupts current context and vectors to the NMI handler in the kernel\/firmware. <\/li>\n
NMI handler attempts minimal diagnostics, writes to special logs, and may trigger platform recovery or reboot. <\/li>\n
Observability agents collect NMI logs and map to infrastructure monitoring and incident pipelines.<\/li>\n<\/ul>\n\n\n\n
NMI in one sentence<\/h3>\n\n\n\n
NMI is a hardware-triggered, non-maskable interrupt delivered to the CPU to signal critical conditions that require immediate OS\/firmware attention.<\/p>\n\n\n\n
NMI vs related terms (TABLE REQUIRED)<\/h3>\n\n\n\n
\n\n
\n
ID<\/th>\n
Term<\/th>\n
How it differs from NMI<\/th>\n
Common confusion<\/th>\n<\/tr>\n<\/thead>\n
\n
\n
T1<\/td>\n
IRQ<\/td>\n
IRQs are maskable and used for normal device interrupts<\/td>\n
Confused as same urgency<\/td>\n<\/tr>\n
\n
T2<\/td>\n
SMI<\/td>\n
SMI runs in firmware\/SMM, not OS context<\/td>\n
Both are low-level interrupts<\/td>\n<\/tr>\n
\n
T3<\/td>\n
Watchdog timer<\/td>\n
Watchdog can generate NMI as reaction<\/td>\n
Not all watchdog events are NMIs<\/td>\n<\/tr>\n
\n
T4<\/td>\n
Machine Check<\/td>\n
Machine checks are CPU error reports, sometimes NMI-triggered<\/td>\n
Often conflated with NMI events<\/td>\n<\/tr>\n
\n
T5<\/td>\n
Kernel panic<\/td>\n
Panic is OS reaction; NMI may cause panic<\/td>\n
People think panic equals NMI<\/td>\n<\/tr>\n
\n
T6<\/td>\n
Exception<\/td>\n
Exceptions are CPU faults from instruction execution<\/td>\n
Some exceptions are handled differently<\/td>\n<\/tr>\n
\n
T7<\/td>\n
Hypervisor trap<\/td>\n
Hypervisor traps are virtualization events<\/td>\n
Might mask or reroute NMIs<\/td>\n<\/tr>\n
\n
T8<\/td>\n
ACPI event<\/td>\n
ACPI signals power\/thermal; can cause NMI on some boards<\/td>\n
Not typically an NMI source<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n
Row Details (only if any cell says \u201cSee details below\u201d)<\/h4>\n\n\n\n
None.<\/p>\n\n\n\n
\n\n\n\n
Why does NMI matter?<\/h2>\n\n\n\n
\n
Business impact (revenue, trust, risk) <\/li>\n
\n
Host-level failures can cause multi-tenant VM interruptions or data plane outages. Uninvestigated NMIs can hide systemic hardware issues leading to repeated downtime, SLA breaches, and customer churn.<\/p>\n<\/li>\n
Early detection of stuck CPUs or hardware errors via NMI reduces mean time to detect (MTTD) and helps avoid cascading faults. Clear NMI telemetry improves troubleshooting velocity and reduces firefighting toil.<\/p>\n<\/li>\n
\n
SRE framing (SLIs\/SLOs\/error budgets\/toil\/on-call) where applicable <\/p>\n<\/li>\n
\n
NMIs map to host-health SLIs (e.g., host-healthy-rate). Elevated NMI rates should consume error budget for infrastructure SLOs. On-call playbooks must include NMI triage to prevent noisy paging and to escalate only genuine host-level failures.<\/p>\n<\/li>\n
\n
3\u20135 realistic \u201cwhat breaks in production\u201d examples\n 1. Kernel deadlock on a host causing all containers to freeze; watchdog NMI triggers kernel handler and may force reboot.\n 2. Repeated Machine Check Exceptions signaled via NMI indicating failing DIMM, causing VM crashes and storage errors.\n 3. Thermal sensor NMI on a blade begins throttling or immediate shutdown, impacting throughput and auto-scaling decisions.\n 4. Faulty NIC triggers PCI-generated NMI, breaking network traffic for affected hosts in a cluster.\n 5. Firmware bug causes spurious NMIs across a hardware fleet, generating alert storms and masking true incidents.<\/p>\n<\/li>\n<\/ul>\n\n\n\n
L1: Hardware NMIs often originate from embedded controllers and require vendor firmware logs and IPMI\/OEM logs for full context.<\/li>\n
L3: Hypervisors may choose to swallow, forward, or emulate NMIs; behavior depends on platform and configuration.<\/li>\n
L5: Managed PaaS abstracts NMIs; operators usually see only higher-level platform health events and must rely on provider telemetry.<\/li>\n<\/ul>\n\n\n\n\n\n\n\n
When should you use NMI?<\/h2>\n\n\n\n
\n
When it\u2019s necessary <\/li>\n
\n
Use NMIs for detecting critical, unrecoverable hardware errors, stuck CPUs, and platform watchdog conditions where immediate attention is required.<\/p>\n<\/li>\n
\n
When it\u2019s optional <\/p>\n<\/li>\n
\n
Use an NMI watchdog in addition to regular telemetry if host-level hangs are rare but costly; enable selectively in high-value or high-risk instances.<\/p>\n<\/li>\n
\n
When NOT to use \/ overuse it <\/p>\n<\/li>\n
\n
Do NOT use NMI as a general-purpose monitoring signal or for frequent, low-priority events. Overuse leads to noisy alerting and may mask true emergencies.<\/p>\n<\/li>\n
\n
Decision checklist <\/p>\n<\/li>\n
If a host can hang and cause multi-tenant impact AND other diagnostics are insufficient -> enable NMI\/watchdog. <\/li>\n
If application-level retries or circuit breakers can recover without host intervention -> prefer application-level SLIs instead. <\/li>\n
\n
If you run fully managed compute where NMI is opaque -> rely on provider platform alerts, not local NMI tinkering.<\/p>\n<\/li>\n
Beginner: Enable basic NMI\/watchdog; collect kernel oops and report to central logging. <\/li>\n
Intermediate: Correlate NMI events with metrics and alerts; automate host cordon and replacement. <\/li>\n
Advanced: Fleet-wide NMI analytics, predictive failure models, automated firmware rollbacks, and integration with capacity and incident pipelines.<\/li>\n<\/ul>\n\n\n\n\n\n\n\n
How does NMI work?<\/h2>\n\n\n\n
\n
\n
Components and workflow\n 1. Hardware source (watchdog, thermal controller, PCI device) detects a critical condition.\n 2. Hardware asserts the NMI line to CPU.\n 3. CPU halts normal interrupt masking and vectors to the NMI handler (in kernel or firmware).\n 4. Handler performs minimal diagnostics (stack trace, CPU registers) and writes to protected log regions.\n 5. Depending on severity, handler may attempt recovery, notify management controllers, or initiate a controlled reboot.\n 6. Logged data is collected by host agents and sent to observability\/incident pipelines.<\/p>\n<\/li>\n
\n
Data flow and lifecycle <\/p>\n<\/li>\n
\n
Detection -> NMI delivered -> Handler executes -> Diagnostic capture -> Persist to ring buffer\/firmware logs -> Collection agent scrapes logs -> Ingest into central observability -> Correlation and alerting -> Triage and remediation.<\/p>\n<\/li>\n
\n
Edge cases and failure modes <\/p>\n<\/li>\n
NMI handler deadlocks or corrupts memory causing cascade. <\/li>\n
NMIs occur during low-level firmware initialization and produce incomplete diagnostics. <\/li>\n
Hypervisor masks or reroutes NMIs causing guest-visible symptoms without host-level signals. <\/li>\n
Multiple NMIs in close succession trigger log overflow and partial data capture.<\/li>\n<\/ul>\n\n\n\n
Typical architecture patterns for NMI<\/h3>\n\n\n\n
\n
Local NMI watchdog: Enable kernel NMI watchdog on hosts; used for VMs and bare metal to detect stuck CPUs. Use when debugging sporadic host hangs.<\/li>\n
Hardware watchdog + management controller: Dedicated BMC triggers NMI and then power-cycles host if OS is unresponsive. Use for resilient fleets with out-of-band management.<\/li>\n
Hypervisor-forwarded NMI: Hypervisor forwards host NMIs to guests for debugging; used in nested virtualization or when guest needs hardware-level visibility.<\/li>\n
Fleet telemetry aggregation: Agents collect NMI events and ship to centralized observability with correlation to kernel oops and machine-check logs. Use for fleet-wide failure analysis.<\/li>\n
Managed cloud abstraction: Rely on provider-supplied host health events and remediation; use when you cannot access host firmware or BMC.<\/li>\n<\/ul>\n\n\n\n
SMI \u2014 System Management Interrupt running in firmware \u2014 Can mask OS handling \u2014 Confused with NMI behavior <\/li>\n
BMC \u2014 Baseboard Management Controller \u2014 Out-of-band logs and power control \u2014 Not all providers expose BMC to tenants <\/li>\n
IPMI \u2014 Interface for managing BMC \u2014 Useful for fetching hardware logs \u2014 Security risk if not secured <\/li>\n
dmesg \u2014 Kernel ring buffer output \u2014 Primary local diagnostic source \u2014 Overwrites quickly without persistence <\/li>\n
oops \u2014 Kernel stack trace after fault \u2014 Essential for debugging \u2014 Misinterpreting stack levels <\/li>\n
crashkernel \u2014 Kernel reservation for kdump \u2014 Enables capture during panic \u2014 Not always configured by default <\/li>\n
kdump \u2014 Kernel crash dump mechanism \u2014 Stores memory images for postmortem \u2014 Large disk needs and complexity <\/li>\n
PCIe AER \u2014 PCI Express Advanced Error Reporting \u2014 Can cause NMIs for device errors \u2014 Misconfigured drivers suppress reports <\/li>\n
Thermal trip \u2014 Overheat event causing emergency interrupt \u2014 Prevents hardware damage \u2014 Lack of telemetry for gradual heating <\/li>\n
Watchdog timer \u2014 Timer that triggers recovery if system appears stuck \u2014 Can generate NMI \u2014 Overzealous timeouts lead to reboots <\/li>\n
CPU hang \u2014 CPU stops making progress \u2014 Detectable by NMI watchdog \u2014 Hard to reproduce in dev environments <\/li>\n
Hypervisor passthrough \u2014 Forwarding hardware interrupts to guests \u2014 Enables guest visibility \u2014 Can hide host-level problems <\/li>\n
Nested virtualization \u2014 VM inside VM \u2014 NMIs may be emulated \u2014 Complexity in debug trails <\/li>\n
Kernel panic \u2014 OS-level unrecoverable state \u2014 May be triggered by NMI actions \u2014 Panics should be captured via kdump <\/li>\n
Ring buffer \u2014 Circular log buffer in kernel \u2014 Holds recent messages \u2014 Overwrites with bursty logs <\/li>\n
ECC memory \u2014 Error-correcting RAM \u2014 Reduces MCEs \u2014 Not all platforms use ECC <\/li>\n
DIMM failure \u2014 Memory module hardware fault \u2014 Often appears as MCEs \u2014 Replacement required, not software fix <\/li>\n
Watchdog reset \u2014 Forced reboot from watchdog \u2014 Helps recover but may lose in-flight work \u2014 Need graceful drain before hard reset <\/li>\n
OOB management \u2014 Out-of-band control via BMC\/IPMI \u2014 Critical when OS is unresponsive \u2014 Limited in fully managed clouds <\/li>\n
Firmware log \u2014 Persistent log stored by firmware\/BMC \u2014 Survives reboots \u2014 Requires vendor tools to interpret <\/li>\n
Event correlation \u2014 Linking NMI with other telemetry \u2014 Essential for accurate triage \u2014 Lack of correlation causes noisy paging <\/li>\n
Panic kernel dump \u2014 Capture on panic for postmortem \u2014 Enables deep analysis \u2014 Large storage and retrieval complexity <\/li>\n
Telemetry agent \u2014 Host agent that ships logs\/metrics \u2014 Collects NMI indicators \u2014 Can be unavailable during host hang <\/li>\n
Kernel oops decode \u2014 Interpreting oops stack and registers \u2014 Critical for root cause \u2014 Often requires symbolized kernels <\/li>\n
Symbolized stack \u2014 Human-readable stack using kernel symbols \u2014 Needed for interpretation \u2014 Requires matching build IDs <\/li>\n
Firmware rollback \u2014 Reverting to older firmware to mitigate regressions \u2014 Useful for spurious NMIs \u2014 Needs controlled rollout <\/li>\n
Safe-mode boot \u2014 Boot mode with minimal drivers \u2014 Helps reproduce early-boot NMIs \u2014 Not always feasible in production <\/li>\n
Panic_on_oops \u2014 Kernel setting to panic on oops \u2014 Ensures kdump capture \u2014 Can reduce system availability if misused <\/li>\n
Watchdog threshold \u2014 Timeout value for watchdog \u2014 Balances detection vs false positives \u2014 Too short causes unnecessary reboots <\/li>\n
Node cordon \u2014 Mark node unschedulable in orchestration \u2014 Prevents placing workloads on unstable node \u2014 Needs automation to avoid human delay <\/li>\n
Auto-replace \u2014 Automated decommission and replacement after NMI \u2014 Reduces mean time to repair \u2014 Risk of replacing healthy nodes if false positive <\/li>\n
JVM safepoint \u2014 Java pause indicating thread stops \u2014 Can look like CPU hang \u2014 JVM-induced pauses can trigger NMI misdiagnosis <\/li>\n
Interrupt storm \u2014 Flood of interrupts from device \u2014 Can starve CPU and cause watchdogs to fire \u2014 Mitigate with driver fixes <\/li>\n
Vendor support ticketing \u2014 Escalation for hardware NMIs \u2014 Required for warranty parts \u2014 Slow if undiagnosed internally <\/li>\n
Fleet analytics \u2014 Aggregate NMI events across hosts \u2014 Useful for root-cause across fleet \u2014 Requires centralized ingestion and retention <\/li>\n
Recovery policy \u2014 What to do when NMI occurs (reboot, replace, escalate) \u2014 Ensures consistent response \u2014 Poor policy causes inconsistent handling<\/li>\n<\/ol>\n\n\n\n\n\n\n\n
How to Measure NMI (Metrics, SLIs, SLOs) (TABLE REQUIRED)<\/h2>\n\n\n\n
\n\n
\n
ID<\/th>\n
Metric\/SLI<\/th>\n
What it tells you<\/th>\n
How to measure<\/th>\n
Starting target<\/th>\n
Gotchas<\/th>\n<\/tr>\n<\/thead>\n
\n
\n
M1<\/td>\n
NMI count per host<\/td>\n
Frequency of NMIs on a host<\/td>\n
Count NMI events in logs per host per day<\/td>\n
0\u20130.01\/day<\/td>\n
Sparse events may indicate underreporting<\/td>\n<\/tr>\n
Deduplicate by root cause fingerprint and host group. <\/li>\n
Group alerts by cluster and time window (e.g., suppress repeat pages for same host within 10 minutes). <\/li>\n
Use suppression windows during platform maintenance.<\/li>\n<\/ul>\n\n\n\n\n\n\n\n
Implementation Guide (Step-by-step)<\/h2>\n\n\n\n
1) Prerequisites\n – Inventory of hardware and virtualization mappings.\n – Access to BMC\/IPMI or provider host health events.\n – Central logging and metrics pipeline.\n – Defined recovery policy (cordon\/replace vs repair).\n2) Instrumentation plan\n – Enable kernel NMI watchdog where appropriate.\n – Configure crashkernel and kdump for dumps.\n – Add host-agent hooks to mark NMI events as metrics and log them.\n3) Data collection\n – Collect dmesg, kernel oops, MCE logs, and BMC SEL entries.\n – Persist to central logs with retention and indexing.\n4) SLO design\n – Define host-health SLOs incorporating NMI frequency and recovery.\n – Set error budgets for host-induced incidents.\n5) Dashboards\n – Build executive, on-call, and debug dashboards.\n – Provide drilldowns from fleet to single-host views.\n6) Alerts & routing\n – Set thresholds for paging vs ticketing.\n – Route to hardware on-call, cloud ops, and relevant service owners.\n7) Runbooks & automation\n – Document step-by-step triage for NMI events.\n – Automate cordon, drain, and replacement actions where safe.\n8) Validation (load\/chaos\/game days)\n – Run controlled tests that simulate stuck CPUs and verify detection & remediation.\n – Include firmware-update scenarios and BMC failure modes.\n9) Continuous improvement\n – Monthly review of NMI trends, firmware rollouts, and vendor replacements.\n – Implement RCA feedback into host provisioning workflows.<\/p>\n\n\n\n
Include checklists:<\/p>\n\n\n\n
\n
Pre-production checklist <\/li>\n
Verify crashkernel\/kdump enabled. <\/li>\n
Test agent log shipping under simulated hang. <\/li>\n
Validate BMC access and SEL retrieval. <\/li>\n
\n
Define recovery policy for hosts in the test environment.<\/p>\n<\/li>\n
\n
Production readiness checklist <\/p>\n<\/li>\n
Baseline host NMI metrics and thresholds. <\/li>\n
Alerting rules with dedupe and grouping. <\/li>\n
Runbook available and indexed in incident portal. <\/li>\n
\n
Automation for safe cordon\/replacement validated.<\/p>\n<\/li>\n
\n
Incident checklist specific to NMI <\/p>\n<\/li>\n
Capture timestamp and host identifiers. <\/li>\n
Retrieve BMC SEL and kernel logs. <\/li>\n
Correlate with other telemetry (network, storage). <\/li>\n
Decide repair vs replace based on recurrence and vendor guidance. <\/li>\n
Record mitigation steps and update fleet actions.<\/li>\n<\/ul>\n\n\n\n\n\n\n\n
Use Cases of NMI<\/h2>\n\n\n\n
Provide 8\u201312 use cases:<\/p>\n\n\n\n
\n
\n
Bare-metal compute stability\n – Context: High-density hosting on bare metal.\n – Problem: Occasional host hangs impacting tenant VMs.\n – Why NMI helps: Detects CPU stalls and forces diagnostic capture.\n – What to measure: NMI count, reboots, kdump capture rate.\n – Typical tools: BMC, Prometheus, Fluentd.<\/p>\n<\/li>\n
\n
Kubernetes node reliability\n – Context: Node-level kernel issues causing pods to freeze.\n – Problem: kubelet not responding, disrupts stateful services.\n – Why NMI helps: Detects node-level hangs to trigger cordon\/replace.\n – What to measure: Node NMI events, node-exporter health.\n – Typical tools: Prometheus, kube-controller-manager automation.<\/p>\n<\/li>\n
\n
High-performance computing clusters\n – Context: Long-running compute workloads sensitive to host faults.\n – Problem: Silent CPU faults leading to silent corruption.\n – Why NMI helps: Machine Check Exceptions via NMI reveal hardware corruption.\n – What to measure: MCE rates, NMI per job.\n – Typical tools: MCE logs, fleet analytics.<\/p>\n<\/li>\n
\n
Network device faults on servers\n – Context: NIC causing packet loss intermittently.\n – Problem: Device driver issues trigger NMIs for serious PCI errors.\n – Why NMI helps: Immediate diagnostics and possible device isolation.\n – What to measure: PCIe AER incidents, NIC resets.\n – Typical tools: ethtool, kernel logs.<\/p>\n<\/li>\n
\n
Firmware regression detection\n – Context: Fleet firmware upgrades performed regularly.\n – Problem: New firmware triggers spurious NMIs on a class of boards.\n – Why NMI helps: Early detection to rollback firmware.\n – What to measure: NMI spike per firmware version.\n – Typical tools: Fleet analytics, firmware inventory.<\/p>\n<\/li>\n
\n
Managed cloud incident correlation\n – Context: Provider host degradation impacts VMs.\n – Problem: Provider signals host health but tenant lacks low-level logs.\n – Why NMI helps: When available via host health events, informs remediation.\n – What to measure: Provider host events vs tenant NMI symptoms.\n – Typical tools: Provider event streams, internal incident systems.<\/p>\n<\/li>\n
\n
Storage controller failures\n – Context: Local disks attached to hosts experiencing errors.\n – Problem: Storage IO lockups affecting VMs.\n – Why NMI helps: PCIe\/device NMIs indicate severe controller faults.\n – What to measure: Device NMIs, IO latency before event.\n – Typical tools: Smartctl, device driver logs.<\/p>\n<\/li>\n
\n
Safety-critical applications (telecom\/finance)\n – Context: Applications requiring durable correctness.\n – Problem: Silent hardware errors can corrupt transactions.\n – Why NMI helps: Ensures hardware error visibility and forced remediation.\n – What to measure: ECC and MCE logs correlated with NMIs.\n – Typical tools: ECC telemetry, MCE parsers.<\/p>\n<\/li>\n
\n
On-prem colo with limited redundancy\n – Context: Single-host services without cloud redundancy.\n – Problem: Host hangs cause prolonged service outages.\n – Why NMI helps: Automate detection and BMC-initiated recovery to reduce downtime.\n – What to measure: Time-to-reboot and service availability post-NMI.\n – Typical tools: IPMI, automated runbooks.<\/p>\n<\/li>\n
\n
Regression testing for OS updates <\/p>\n
\n
Context: Rolling kernel updates can affect NMI handling. <\/li>\n
Problem: Kernel changes cause different NMI behavior. <\/li>\n
Why NMI helps: Detect regressions in test labs before rollout. <\/li>\n
What to measure: NMI occurrence in canary hosts. <\/li>\n
Typical tools: CI labs, kdump validations.<\/li>\n<\/ul>\n<\/li>\n<\/ol>\n\n\n\n\n\n\n\n
Scenario #1 \u2014 Kubernetes node hang due to kernel deadlock (Kubernetes scenario)<\/h3>\n\n\n\n
Context:<\/strong> Production Kubernetes cluster with stateful services experiencing pod stalls.\nGoal:<\/strong> Detect node hangs quickly and automate replacement with minimal app impact.\nWhy NMI matters here:<\/strong> NMI watchdog can detect kernel stalls that kubelet and liveness probes cannot.\nArchitecture \/ workflow:<\/strong> Host-level NMI watchdog -> Kernel handler captures oops and marks node as unhealthy -> Host agent sends event to central logs -> Controller automation cordons node and drains workloads -> Replace host.\nStep-by-step implementation:<\/strong><\/p>\n\n\n\n\n
Enable kernel NMI watchdog on node images. <\/li>\n
Configure crashkernel and kdump to store dumps to network-backed storage. <\/li>\n
Add agent metric for “nmi_event_total” and ship to Prometheus. <\/li>\n
Create alert rule: any nmi_event_total > 0 on prod node -> Pager for hardware team. <\/li>\n
Automate cordon+drain when NMI occurs via operator with safety checks.\nWhat to measure:<\/strong> NMI events, time node cordoned, pod eviction success rate.\nTools to use and why:<\/strong> Prometheus for metrics, Fluent Bit for logs, kube-controller-manager for automation.\nCommon pitfalls:<\/strong> Watchdog false positives during CPU-intensive pods; ensure thresholds.\nValidation:<\/strong> Run simulated CPU hang tests in canary; verify cordon and replacement.\nOutcome:<\/strong> Faster detection and reduced downtime for affected services.<\/li>\n<\/ol>\n\n\n\n
Scenario #2 \u2014 Serverless function host hardware fault (Serverless\/managed-PaaS scenario)<\/h3>\n\n\n\n
Context:<\/strong> Managed serverless provider where tenant functions occasionally cold-fail without clear app errors.\nGoal:<\/strong> Correlate host-level faults with function failures and reduce function error rates.\nWhy NMI matters here:<\/strong> Underlying host hardware faults manifest as function invocation failures; NMIs indicate host-level issues needing provider action.\nArchitecture \/ workflow:<\/strong> Provider collects host events -> NMIs associated with host IDs -> Functions on same host flagged and invocations redirected -> Provider performs host replacement.\nStep-by-step implementation:<\/strong><\/p>\n\n\n\n\n
Ensure provider exposes host health events or internal BMC logs. <\/li>\n
Correlate function invocation failures with host event timestamps. <\/li>\n
Blacklist affected hosts in routing pool until resolved.\nWhat to measure:<\/strong> Host NMI events, function error rate per host, blacklisted-host count.\nTools to use and why:<\/strong> Provider internal telemetry, central event bus.\nCommon pitfalls:<\/strong> Tenant cannot access host-level logs in public clouds; rely on provider channels.\nValidation:<\/strong> Inject a simulated kernel hang in a non-prod region to verify routing changes.\nOutcome:<\/strong> Reduced invocation failures and faster provider remediation.<\/li>\n<\/ol>\n\n\n\n
Scenario #3 \u2014 Postmortem after recurrent NMIs (Incident-response\/postmortem scenario)<\/h3>\n\n\n\n
Context:<\/strong> Fleet shows recurring NMIs on a set of blades after firmware rollout.\nGoal:<\/strong> Root-cause analysis and rollback plan.\nWhy NMI matters here:<\/strong> Pattern of NMIs indicates regression introduced by firmware.\nArchitecture \/ workflow:<\/strong> Collect NMI events, firmware version mapping, BMC SEL, and service impact metrics -> RCA -> Firmware rollback plan and automated replacement.\nStep-by-step implementation:<\/strong><\/p>\n\n\n\n\n
Aggregate NMIs across hosts and group by firmware version. <\/li>\n
Validate with vendor logs and affected hardware SKU. <\/li>\n
Rollback firmware on canary subset, observe reduction in NMIs, then do controlled rollback.\nWhat to measure:<\/strong> NMI rate pre\/post rollback, service error budget consumption.\nTools to use and why:<\/strong> Fleet analytics, vendor firmware tooling.\nCommon pitfalls:<\/strong> Incomplete metadata linking firmware to host; ensure accurate inventory.\nValidation:<\/strong> Canary rollback and monitoring for recurrence.\nOutcome:<\/strong> Restored fleet stability and updated firmware rollout policy.<\/li>\n<\/ol>\n\n\n\n
Context:<\/strong> High-performance trading application prioritizing latency; ops must balance detection with performance.\nGoal:<\/strong> Tune watchdog sensitivity to avoid performance regressions while detecting real hangs.\nWhy NMI matters here:<\/strong> Aggressive watchdogs may cause reboots during short GC or high-CPU bursts; too lax and hangs go undetected.\nArchitecture \/ workflow:<\/strong> Parameterize watchdog timeout per instance class; observe impact on latency and NMI rate.\nStep-by-step implementation:<\/strong><\/p>\n\n\n\n\n
Baseline latency and CPU profiles. <\/li>\n
Enable NMI watchdog with conservative timeout on canary nodes. <\/li>\n
Adjust timeouts and correlate NMI rate vs latency impact. <\/li>\n
Choose per-instance class policy (e.g., lower sensitivity on ultra-low-latency instances).\nWhat to measure:<\/strong> Latency SLOs, NMI events, false reboot rate.\nTools to use and why:<\/strong> Prometheus, performance test harness.\nCommon pitfalls:<\/strong> Over-generalizing timeouts across different workloads; tune per class.\nValidation:<\/strong> Load tests and targeted chaos testing.\nOutcome:<\/strong> Balanced detection while preserving latency goals.<\/li>\n<\/ol>\n\n\n\n
Context:<\/strong> VM running Kubernetes, guest experiences fatal hangs; unclear if guest or host triggered.\nGoal:<\/strong> Determine whether NMIs originate in host or guest and fix accordingly.\nWhy NMI matters here:<\/strong> NMIs may be swallowed or emulated by hypervisor leading to ambiguity.\nArchitecture \/ workflow:<\/strong> Instrument both host and guest with NMI counters, enable hypervisor passthrough settings, compare logs.\nStep-by-step implementation:<\/strong><\/p>\n\n\n\n\n
Enable guest-visible NMI metrics. <\/li>\n
Correlate guest NMI events with host BMC SEL entries. <\/li>\n
If host-origin, escalate to hardware replacement; if guest-origin, patch guest kernel.\nWhat to measure:<\/strong> Host vs guest NMI event correlation rate.\nTools to use and why:<\/strong> QEMU\/KVM logs, host BMC logs, guest kernel logs.\nCommon pitfalls:<\/strong> Hypervisor default masks; must explicitly enable forwarding for accurate diagnosis.\nValidation:<\/strong> Reproduce on nested testbed.\nOutcome:<\/strong> Correct ownership and remediation path identified.<\/li>\n<\/ol>\n\n\n\n\n\n\n\n
Common Mistakes, Anti-patterns, and Troubleshooting<\/h2>\n\n\n\n
List 15\u201325 mistakes with:\nSymptom -> Root cause -> Fix<\/p>\n\n\n\n
\n
Symptom: No NMI logs after host crash -> Root cause: crashkernel not configured -> Fix: Reserve crashkernel and enable kdump. <\/li>\n
Symptom: Missing symbolized stacks -> Root cause: Kernel versions mismatched -> Fix: Keep kernel build and symbol store synchronized. <\/li>\n
Symptom: Noise during firmware maintenance -> Root cause: Not excluding canary hosts -> Fix: Use canary cohort and staged rollout. <\/li>\n
Symptom: Data loss after watchdog reboot -> Root cause: Hard reset without graceful drain -> Fix: Graceful drain automation before reboot. <\/li>\n
Symptom: Observability agent fails during hang -> Root cause: Agent runs in same kernel context -> Fix: Use out-of-band collection or pre-persisted logs. <\/li>\n
Symptom: Misattributed billing events -> Root cause: Reboots lead to VM migration counts -> Fix: Tag incidents and correlate with billing pipeline. <\/li>\n
PITFALL: Relying solely on dmesg -> CAUSE: dmesg ephemeral -> FIX: Persist to central logs and capture frequently. <\/li>\n
PITFALL: Not syncing clocks -> CAUSE: Misaligned timestamps -> FIX: NTP\/PTP across host and BMC. <\/li>\n
PITFALL: Missing correlation with higher-level metrics -> CAUSE: Siloed telemetry -> FIX: Integrate event bus and correlate traces\/metrics\/logs. <\/li>\n
PITFALL: Agent unavailable during hang -> CAUSE: Agent process requires kernel context -> FIX: Out-of-band logging via BMC. <\/li>\n
PITFALL: Over-indexed raw dumps -> CAUSE: Storage blowout -> FIX: Prioritize metadata and sample full dumps.<\/li>\n<\/ul>\n\n\n\n\n\n\n\n
Best Practices & Operating Model<\/h2>\n\n\n\n
\n
Ownership and on-call <\/li>\n
\n
Hardware\/BMC team owns NMI handling policies; service owners own application impact. Create a separate hardware on-call rotation for urgent host-level NMIs.<\/p>\n<\/li>\n
\n
Runbooks vs playbooks <\/p>\n<\/li>\n
Runbooks: Step-by-step remediation for a specific host NMI. <\/li>\n
\n
Playbooks: Higher-level escalation flows involving multiple teams and vendor engagement.<\/p>\n<\/li>\n
Firmware\/kernel changes must roll through small canaries with NMI and kdump monitoring. Implement automated rollback triggers on NMI spikes.<\/p>\n<\/li>\n
\n
Toil reduction and automation <\/p>\n<\/li>\n
\n
Automate cordon\/drain and host replacement. Use auto-replace policies for hosts with repeat NMIs and escalate to vendor only after automated triage.<\/p>\n<\/li>\n
\n
Security basics <\/p>\n<\/li>\n
Secure BMC\/IPMI access with rotation and network isolation. Limit who can read SEL logs. Audit all actions against hardware.<\/li>\n<\/ul>\n\n\n\n
Include:<\/p>\n\n\n\n
\n
Weekly\/monthly routines <\/li>\n
Weekly: Review NMI events for spikes and outstanding kdumps. <\/li>\n
Abstracted details; combine with tenant logs<\/td>\n<\/tr>\n<\/tbody>\n<\/table><\/figure>\n\n\n\n
Row Details (only if needed)<\/h4>\n\n\n\n
\n
I3: Ensure BMC credentials management integrates with secrets store and audit logs.<\/li>\n
I4: Validate crashkernel sizing in CI images to ensure kdump success.<\/li>\n<\/ul>\n\n\n\n\n\n\n\n
Frequently Asked Questions (FAQs)<\/h2>\n\n\n\n
What exactly triggers an NMI?<\/h3>\n\n\n\n
Hardware sources like watchdog timers, certain PCI errors, thermal events, and CPU machine checks can trigger an NMI.<\/p>\n\n\n\n
Can NMIs be disabled?<\/h3>\n\n\n\n
Varies \/ depends. Some platforms and hypervisors allow configuration; disabling removes a critical safety mechanism and is generally discouraged.<\/p>\n\n\n\n
Are NMIs visible inside VMs?<\/h3>\n\n\n\n
Sometimes. Hypervisor decisions determine whether NMIs are forwarded to guests, emulated, or swallowed.<\/p>\n\n\n\n
How do NMIs differ from SMIs?<\/h3>\n\n\n\n
SMIs execute in firmware SMM and are generally invisible to the OS; NMIs are delivered to the CPU and handled by the OS\/kernel.<\/p>\n\n\n\n
How do I capture useful data when an NMI happens?<\/h3>\n\n\n\n
Enable crashkernel\/kdump, persist kernel ring buffer to a central log, and collect BMC SEL entries.<\/p>\n\n\n\n
Can NMIs be the cause of data corruption?<\/h3>\n\n\n\n
NMIs often indicate hardware faults (like MCEs) that can lead to data corruption if not detected; NMIs themselves are signaling mechanisms.<\/p>\n\n\n\n
How to avoid false positives from watchdog NMIs?<\/h3>\n\n\n\n
Tune watchdog thresholds per workload and test under realistic load profiles.<\/p>\n\n\n\n
Do cloud providers surface NMIs to tenants?<\/h3>\n\n\n\n
Varies \/ depends. Public cloud providers typically surface aggregate host health events rather than raw NMIs.<\/p>\n\n\n\n
How to troubleshoot repeated NMIs on a host?<\/h3>\n\n\n\n
Collect kdump and SEL, compare firmware and hardware SKU, isolate with hardware swap, and engage vendor support.<\/p>\n\n\n\n
Is there a standard SLI for NMIs?<\/h3>\n\n\n\n
No universal standard; common SLI is hosts with NMI rate per period. Define based on fleet risk tolerance.<\/p>\n\n\n\n
Should NMIs page engineers?<\/h3>\n\n\n\n
Only if the event impacts production availability or is a repeated\/recurrent failure; otherwise log and ticket.<\/p>\n\n\n\n
How to test NMI handling?<\/h3>\n\n\n\n
Simulate hung CPUs in a controlled environment or use vendor-provided diagnostic tools.<\/p>\n\n\n\n
Will NMIs always produce stack traces?<\/h3>\n\n\n\n
Not always. Early-boot NMIs or severe memory corruption can prevent full stack captures.<\/p>\n\n\n\n
How to secure BMC\/IPMI access?<\/h3>\n\n\n\n
Network isolation, credential rotation, strong auth, and restricted access roles.<\/p>\n\n\n\n
What to do if NMIs increase after a firmware update?<\/h3>\n\n\n\n
Halt rollout, roll back canary firmware, gather diagnostics, and escalate to vendor.<\/p>\n\n\n\n
Can software cause an NMI?<\/h3>\n\n\n\n
Software cannot directly cause NMI, but software behavior (e.g., infinite interrupt disabling) can lead to watchdog timeouts that trigger NMIs.<\/p>\n\n\n\n
How long should we retain NMI-related logs?<\/h3>\n\n\n\n
Sufficient to perform RCA and trend analysis; typically months for fleet analytics and at least 90 days for incident traces.<\/p>\n\n\n\n
Do containers see NMIs?<\/h3>\n\n\n\n
Containers rely on kernel\/host; containers do not directly receive NMIs but will suffer host-level impacts.<\/p>\n\n\n\n
\n\n\n\n
Conclusion<\/h2>\n\n\n\n
Non-Maskable Interrupts are a critical, low-level signal for hardware and platform reliability. For 2026 cloud-native operations, treating NMIs as part of your observability and incident response fabric is essential\u2014especially for bare-metal fleets, high-availability services, and safety-critical workloads. Proper instrumentation, automation, and vendor collaboration turn NMIs from opaque emergencies into actionable diagnostics.<\/p>\n\n\n\n
Next 7 days plan (5 bullets):<\/p>\n\n\n\n
\n
Day 1: Inventory hosts and confirm crashkernel\/kdump settings on a canary cohort. <\/li>\n
Day 2: Enable minimal NMI telemetry and ship to central logs. <\/li>\n
Day 3: Create Prometheus metrics and dashboards for NMI events. <\/li>\n
Day 4: Draft runbook for NMI triage and test on non-prod hosts. <\/li>\n
Day 5\u20137: Run a controlled NMI simulation in staging and validate automation for cordon\/replace.<\/li>\n<\/ul>\n\n\n\n\n\n\n\n