Kubernetes Production Troubleshooting Guide

Why Kubernetes Incidents Still Take Hours to Resolve in 2026

Despite a decade of maturity, Kubernetes production incidents remain among the slowest to triage. Datadog’s 2026 Container Report puts median time-to-detect for pod-level failures at 4.2 minutes but median time-to-resolve at 47 minutes — an 11x gap driven almost entirely by unclear ownership between application, platform, and networking layers. The core problem isn’t tooling; it’s that most engineers troubleshoot Kubernetes the same way they troubleshoot a monolith: read logs, restart the process, hope. Kubernetes failures are systemic — they cascade across scheduler, kubelet, CNI, and etcd — and require a layered diagnostic method.

This guide gives you that method: a deterministic decision tree for the five failure classes that account for roughly 80% of production pages, based on aggregated postmortems from SRE teams at mid-size SaaS companies running clusters between 200 and 5,000 nodes.

Failure Class 1: CrashLoopBackOff and OOMKilled Pods

The single most common alert is a pod stuck in CrashLoopBackOff. The instinct is to check application logs first — that’s wrong. Check exit code first.

• Exit code 137 = SIGKILL, almost always OOMKilled. Run kubectl describe pod <name> and look at Last State: Terminated, Reason: OOMKilled. Fix: raise resources.limits.memory or fix a memory leak — check for unbounded caches or connection pools first, since 60%+ of OOMKills in JVM and Node.js workloads trace to those two causes.
• Exit code 1 or 2 = application-level failure. Read logs with kubectl logs <pod> --previous (the current container has already restarted).
• Exit code 143 = SIGTERM, often a misconfigured terminationGracePeriodSeconds colliding with a slow shutdown hook.

A frequently missed detail: kubectl top pod shows current usage, not peak. Use kubectl get --raw /apis/metrics.k8s.io/v1beta1 sampled every 10 seconds, or better, pull historical peaks from Prometheus (container_memory_working_set_bytes) — the OOMKill happened at a peak you can no longer observe live.

Failure Class 2: Networking — DNS, Service Discovery, and CNI Partitions

Networking failures are the hardest to diagnose because they’re rarely deterministic. A four-step isolation protocol:

1. Confirm it’s DNS. Exec into a debug pod and run nslookup <service>.<namespace>.svc.cluster.local. If this hangs, check CoreDNS pod health and kubectl logs -n kube-system -l k8s-app=kube-dns for SERVFAIL loops — a classic sign of upstream resolver saturation.
2. Confirm it’s not a NetworkPolicy. kubectl get networkpolicy -A — a policy applied to the wrong namespace selector silently blackholes traffic with no error surfaced to the caller.
3. Check CNI plugin health. For Cilium or Calico, check the daemonset pods on the affected node specifically — CNI failures are node-scoped, not cluster-wide, which is why symptoms look like “some pods work, some don’t.”
4. Check kube-proxy / iptables rule count. Clusters above ~2,000 services can see kube-proxy fall behind on iptables rule propagation; switching to IPVS mode resolves this in most cases.

Failure Class 3: Control Plane Degradation (etcd, API Server, Scheduler)

Control-plane issues manifest as slow kubectl responses, stuck rollouts, or pods stuck Pending. Check etcd first — it’s the most common root cause and the least monitored component.

• etcdctl endpoint status --write-out=table — watch for high RAFT TERM churn, which indicates leader elections under load.
• API server latency: query apiserver_request_duration_seconds in Prometheus, filtered by verb and resource. LIST requests on large CRDs (a common 2026 pattern with GitOps controllers) are the top cause of API server slowness.
• Scheduler: Pending pods with no scheduling events usually mean resource fragmentation, not resource scarcity — check node-level allocatable vs. requested, not cluster-wide totals.

Comparison Table: Diagnostic Signal by Failure Class

Building a Repeatable Runbook

The teams with the lowest MTTR don’t have smarter engineers — they have runbooks that map symptom to command to fix, removing decision-making from the critical path. Store these as versioned markdown in your incident-response repo, link them directly from PagerDuty alert descriptions, and review them quarterly against actual postmortems. The goal is that a mid-level engineer on their first on-call rotation can resolve 80% of pages without escalating.

If you’re preparing for interviews where Kubernetes troubleshooting scenarios come up — infrastructure and platform engineering roles increasingly test this in system design rounds — structured practice matters more than reading documentation. The 0-to-1 SWE Interview Playbook (https://www.amazon.com/dp/B0H256Z1MF?tag=sirjohnnymai-20) includes a full section on infrastructure debugging scenarios interviewers use to separate candidates who’ve memorized kubectl flags from those who understand failure propagation.

FAQ

Q: Why does kubectl top pod sometimes show low memory usage right before an OOMKill? A: kubectl top samples on an interval (typically 15-60 seconds depending on metrics-server config) and shows current usage, not the peak that triggered the kill. The OOM event is recorded by the kernel cgroup controller at the exact moment of the spike, which live metrics-server sampling will miss. Always cross-reference with container_memory_working_set_bytes history in Prometheus rather than trusting a live snapshot.

Q: How do I tell the difference between an application bug and a Kubernetes scheduling problem when a pod won’t start? A: Run kubectl describe pod and read the Events section first, before touching logs. FailedScheduling events mean it’s a cluster resource or affinity problem, never an application bug — the container hasn’t even started. CrashLoopBackOff with actual container logs means it’s application-level. Conflating these two is the single most common mistake junior engineers make during incidents.

Q: Is switching from iptables to IPVS mode for kube-proxy safe to do on a live production cluster? A: It requires a kube-proxy restart on every node, which briefly disrupts new connections (existing connections via conntrack are generally preserved). Do it node-by-node during a low-traffic window, verify service connectivity after each node, and have a rollback plan (revert the kube-proxy ConfigMap and restart the daemonset) ready before starting. Never do it as a single cluster-wide rolling change without incremental verification.