Kubernetes health checks: liveness, readiness, and startup probes
Set Kubernetes probe endpoints and timing around the actions you want, then use Pod events and exit status to diagnose restarts.
A checkout rollout starts, the new Pod appears, and its process begins warming a cache. Sending it requests now would cause errors. Restarting it would interrupt the warm-up and put it back where it started. Kubernetes health checks let the kubelet distinguish those two decisions.
Use readiness to decide whether a container can serve, liveness to detect failures a restart can fix, and startup to give a slow container a separate launch budget. The examples below are fragments for the same container in spec.containers[]. The application must serve /healthz and /ready on port 8080.
Start with the action a failure should trigger
Liveness protects against a process that is still running but has stopped making progress. After consecutive failures reach the threshold, the kubelet stops the container and applies its restart policy. Make /healthz check local progress. If the shared database goes down, restarting every application replica adds cold starts and reconnects without repairing the database.
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 10
periodSeconds: 10
failureThreshold: 3
Readiness controls eligibility for ordinary Service traffic. Once the threshold is reached, the kubelet marks the container unready. The Pod’s Ready condition reflects container readiness and any configured readiness gates. For a normal selector-based Service, the endpoint can stay in its EndpointSlice with ready: false. Traffic handling can lag behind that change. Existing connections are not forcibly closed by the readiness probe, and direct requests to the Pod can bypass the Service. Check publishNotReadyAddresses if the Service deliberately publishes endpoints before they are ready. EndpointSlice conditions
readinessProbe:
httpGet:
path: /ready
port: 8080
periodSeconds: 5
failureThreshold: 2
A readiness check should reflect what the instance can actually serve. If a dependency is essential for every request, it can be useful to mark that instance unready when the dependency is unavailable. If every replica can return cached results during the same outage, withdrawing every endpoint would also remove the working fallback. Keep the check cheap enough to run frequently.
Startup holds liveness and readiness until the startup probe succeeds. It runs only during startup of each container instance. If it reaches its failure threshold, the kubelet stops that container and applies the restart policy. With a 10-second period and a threshold of 30, plan for roughly five minutes of startup budget. The first check, scheduling, and individual check duration affect the exact timing.
startupProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 10
failureThreshold: 30
Choose what the probe can actually prove
| Mechanism | Success signal | What to watch |
|---|---|---|
| HTTP GET | Status 200–399 | The kubelet normally connects to the Pod IP. Return a direct response from the health endpoint. For HTTPS, the kubelet skips certificate verification. |
| TCP | The connection opens | An open port can pass even if the application closes the connection immediately. The kubelet connects from the node. A request from a client Pod tests a different path. |
| Exec | The command exits with code 0 | Run a short command. Pipes and redirects need an explicit shell. Frequent process creation can be costly at high Pod density. |
| gRPC | The health service reports SERVING |
Implement the gRPC Health Checking Protocol. The built-in probe uses a numeric port and defaults to plaintext. Use the optional service value if the server distinguishes checks. |
HTTP and TCP can use a named container port. The built-in gRPC probe has been stable since Kubernetes v1.27. Kubernetes v1.37 adds alpha TLS support behind the disabled-by-default GRPCContainerProbeTLS gate. In that mode the kubelet skips server certificate verification. gRPC probe TLS
For HTTP probes, keep the response small and avoid redirects. A redirect to another host can count as a successful probe with a warning, even though the health endpoint you meant to check was never contacted there. Kubernetes probe configuration
Trade off reaction time against false failures
The defaults are initialDelaySeconds: 0, periodSeconds: 10, timeoutSeconds: 1, failureThreshold: 3, and successThreshold: 1. Liveness and startup require successThreshold: 1. While a container is unready, the kubelet can run readiness checks sooner than the configured period. Startup gating and the initial delay both matter. In the Kubernetes v1.36.0 worker, the delay check uses the container’s running start time, so do not assume it starts over after the startup probe succeeds. Probe defaults · Kubelet v1.36.0 source
If /healthz sometimes takes more than a second at peak load, measure the endpoint before changing the policy. This alternative liveness configuration allows each check three seconds and needs three consecutive failures. Replace the earlier liveness fragment with it if those values fit your recovery budget. Slow checks, the probe schedule, and termination grace affect how long it takes to see a replacement.
livenessProbe:
httpGet: {path: /healthz, port: 8080}
periodSeconds: 15
timeoutSeconds: 3
failureThreshold: 3
Investigate the restart and the traffic symptom separately
CrashLoopBackOff means the kubelet is delaying another restart. Use the actual namespace, Pod, and container name in these read-only commands. Compare the container’s Last State, exit reason and timestamp with its previous logs. Then compare those timestamps with Pod events.
NS=shop
POD=checkout-abc
CONTAINER=checkout
kubectl -n "$NS" describe pod "$POD"
kubectl -n "$NS" logs "$POD" -c "$CONTAINER" --previous
kubectl -n "$NS" get events --field-selector "involvedObject.kind=Pod,involvedObject.name=$POD" --sort-by=.metadata.creationTimestamp
A startup or liveness failure followed by a container kill points to probe-triggered termination. OOMKilled in the last termination state points to a different investigation. An Unhealthy event alone cannot tell you which happened because readiness can also emit it. A Pod that stays up but disappears from ordinary routing should lead you to inspect readiness and the Service’s EndpointSlice.
kubectl -n shop get endpointslice -l kubernetes.io/service-name=checkout -o yaml
The API server’s default --event-ttl is one hour and clusters can change it. If you need a longer incident timeline, enable the DataKit Kubernetes event collector and check that it can read the intended namespaces. Match involved_namespace, involved_name, the event time, and the affected container’s logs or status. This keeps a brief readiness flap from looking like the reason a container restarted earlier. Pod lifecycle · API server settings
Further reading · Kubernetes architecture · Install Prometheus on Kubernetes with Helm
Contact us
Join the community
to join the community
Try Guance
Start online and pay only for what you use.
Get startedChoose a Guance plan
