Kubernetes OOMKilled: Exit Code 137 Explained, and How to Fix It (2026)
OOMKilled in Kubernetes — what exit code 137 means, limits vs node memory vs JVM heap, how to diagnose with kubectl, and the fixes that actually hold.
Short answer: OOMKilled means the Linux kernel's Out-Of-Memory killer terminated your container because it exceeded its memory limit — exit code 137 (128 + SIGKILL). Diagnose with kubectl describe pod (Reason: OOMKilled), confirm with actual usage metrics, then fix one of three ways: raise the limit to match real usage, fix a memory leak, or cap the application's own memory (the classic: JVM heap larger than the container limit). The trap: OOMKills fire on limit breach even when the node has free memory.
What actually happened (the 30-second mechanism)
Kubernetes containers live in cgroups with a hard memory ceiling — your pod spec's resources.limits.memory. When a container's memory crosses that ceiling, the kernel OOM killer selects and SIGKILLs a process inside the cgroup. The kubelet reports the container as OOMKilled and restarts it per the pod's restart policy. Two clarifications that resolve most confusion:
- The limit, not the node, is the trigger. A container with a 512Mi limit gets OOMKilled at 512Mi even on a node with 200 GiB free. (A separate mechanism — node-pressure eviction — handles the node actually running out.)
- Exit code 137 = 128 + 9 = SIGKILL. Same code appears when something external kills the process, but in Kubernetes with
Reason: OOMKilled, it's memory.
Diagnosis: confirm it, size it, time it
# 1. Confirm the reason and exit code
kubectl describe pod <pod> -n <ns>
# Last State: Terminated Reason: OOMKilled Exit Code: 137
# 2. See actual usage vs limit (was it creeping or spiking?)
kubectl top pod <pod> -n <ns> # current only — history needs metrics
# 3. PromQL for the trend (the chart that decides the fix)
container_memory_working_set_bytes{pod="<pod>"}
/ container_spec_memory_limit_bytes{pod="<pod>"}
Reading the trend: slow creep upward over hours/days = memory leak (raising the limit delays the kill, doesn't fix it). Sharp spike on startup or under load = sizing problem (raise limit or reduce workload). Instant kill at boot = the app's own memory settings exceed the container limit (JVM class of problem).
Related guideKubernetes Monitoring: The Complete 2026 Guide→
The fixes, in order of how often they're right
Fix 1 — Size limits from measured reality. Chart p95 container_memory_working_set_bytes over a week; set requests at p95 and limits at ~1.5× p95. Requests drive scheduling; limits drive OOMKills — setting them thoughtfully is the whole game.
resources:
requests: { memory: "512Mi", cpu: "250m" }
limits: { memory: "768Mi", cpu: "1" }
Fix 2 — Cap the application's own memory. The JVM is the repeat offender: default heap ratios are calculated against what it sees, and older configurations saw the node's RAM, not the container's. On modern JVMs (JDK 11+, and 8u191+):
# Container-aware JVM settings
-XX:+UseContainerSupport -XX:MaxRAMPercentage=75.0
# Or be explicit: -Xmx must sit comfortably BELOW the container limit
# (limit 1Gi → -Xmx700m, leaving room for metaspace/threads/direct buffers)
Node.js equivalent: --max-old-space-size below the limit. Python: watch per-worker memory × worker count.
Fix 3 — Find the leak. If usage climbs monotonically until the kill: heap dumps (Java/Node), tracemalloc (Python), or a continuous profiler. The pattern "restarts every ~26 hours like clockwork" is a leak signature, and raising the limit just changes the period.
Fix 4 — Check the silent contributors. Page cache is not counted against the limit, but these are: tmpfs mounts, shared memory (/dev/shm), and off-heap buffers (Netty, JNI). Apps with heavy off-heap usage get OOMKilled while the heap looks fine.
Prevent: the alerts that catch it before the kill
# Memory above 85% of limit for 15 minutes — you get paged BEFORE the OOM
container_memory_working_set_bytes / container_spec_memory_limit_bytes > 0.85
# Every OOMKill as an event
kube_pod_container_status_last_terminated_reason{reason="OOMKilled"} == 1
Also chart the creep rate (deriv() of working set over 6h): a positive slope on a long-running service is a leak announcement weeks before the kill.
Related guideKubernetes Integration→
FAQ
Q: What does OOMKilled mean in Kubernetes?
The container exceeded its memory limit and the kernel's OOM killer terminated it (SIGKILL, exit code 137). It triggers on the container's cgroup limit — not on overall node memory — so pods get OOMKilled on nodes with plenty of free RAM. Check kubectl describe pod for Reason: OOMKilled.
Q: What is exit code 137?
128 + 9, meaning the process received SIGKILL. In Kubernetes with Reason: OOMKilled, it came from the kernel's out-of-memory killer enforcing the container's memory limit. The same code appears for any external SIGKILL, so always pair the exit code with the Reason field.
Q: How do I fix OOMKilled?
First read the memory trend: a spike means resize (set limits ≈1.5× measured p95 usage); a slow creep means a leak (profile and fix); an instant boot-kill means the app's own memory settings exceed the limit (set JVM -XX:MaxRAMPercentage=75 or an explicit -Xmx below the container limit; --max-old-space-size for Node.js).
Q: Why is my pod OOMKilled when the node has free memory?
Because the OOM killer enforces the container's cgroup limit, not node capacity. Node memory pressure triggers a different mechanism (eviction, shown as Status: Evicted). If you see OOMKilled with free node memory, the container limit is simply lower than the app's appetite — resize the limit or reduce usage.
Q: OOMKilled vs CrashLoopBackOff — what's the difference?
OOMKilled is a cause (killed for exceeding the memory limit); CrashLoopBackOff is a restart pattern (any repeated crash with growing backoff). A pod repeatedly OOMKilled at boot will display CrashLoopBackOff as its status — read Last State → Reason in describe pod to learn whether OOM is the underlying cause.
Sources: Kubernetes documentation (resource management, OOM behavior, cgroup v2), JVM container-support release notes. Verified 2026-08-07.
Contact us
Join the community
to join the community
Try Guance
Start online and pay only for what you use.
Get startedChoose a Guance plan