Zero-Downtime Deployments With Kubernetes Readiness Probes

Zero-downtime deployments with Kubernetes readiness probes depend on a simple distinction. A container that has started is not necessarily ready to serve traffic. Rolling updates can keep a service available, but only when readiness checks, replacement capacity, and graceful shutdown work together. A successful rollout status is not proof that users saw no errors.

Why Kubernetes rolling updates can still drop traffic without readiness probes

An application has work to do between starting its process and handling its first request. It may still be opening database connections, loading configuration, or warming a cache. Without a readiness probe, Kubernetes considers the container ready as soon as it is running. The pod can then enter the Service’s EndpointSlice while the application is still getting ready to serve.

That gap turns an orderly deployment into a traffic failure. Kubernetes creates a new ReplicaSet and scales it up while scaling the old one down, but without a meaningful readiness check, it has no application-level signal to wait for. The rolling-update defaults of 25% for maxUnavailable and 25% for maxSurge also permit available capacity to fall below the desired replica count. Every pod can show 1/1 Running and kubectl rollout status can exit zero while users get 502s.

What readiness probes actually control inside the rollout loop

A readiness probe controls whether a pod belongs in the Service’s traffic rotation. When the probe fails, the EndpointSlice controller removes the pod from the matching Service EndpointSlices, and kube-proxy and ingress controllers update their routing. The pod keeps running and has a chance to recover. A liveness failure does something fundamentally different by triggering a container restart. Stopping traffic and restarting a process are not interchangeable remedies.

The three probe types divide that responsibility across the pod’s life. A startup probe owns the boot window and suppresses readiness and liveness checks until it succeeds and hands off. Readiness then checks whether this pod should get traffic right now. Liveness answers whether the container is unrecoverably stuck. Separate endpoints, including /healthz/started, /healthz/ready, and /healthz/live, should be allowed to disagree.

Pointing readiness and liveness at the same URL erases that distinction. Under load, a slow response can fail both checks, so Kubernetes removes traffic from the pod but also kills its container. The replicas that are left take on that load, slow down, and may fail their own liveness checks. A latency spike becomes a rolling restart of the Deployment. Even correctly separated probes cannot close the delay between a pod’s deletion and the routing layer finishing its updates, because that is a shutdown problem.

The four failure families that produce 502s even when probes exist

The first failure family is traffic arriving before the application can serve it. A missing probe causes this, but so does a /healthz handler that passes before the rest of the application finishes initializing. Readiness must describe this pod’s ability to serve.

The second family is traffic arriving after the application has stopped serving. Pod deletion starts two parallel tracks, with the kubelet running the preStop hook and shutdown signals while routing removal propagates to kube-proxy, ingress controllers, and any cloud load balancer. Those tracks do not wait for each other. Requests that still follow the old route can hit a socket as it closes and return 502s or connection resets. On a busy cluster, that routing-convergence gap can last several seconds.

The third family is capacity disappearing faster than its replacement becomes ready. Under sustained traffic, the default maxUnavailable of 25% can make that dip visible to users. Setting maxUnavailable to 0 and maxSurge to 1 instead requires a ready replacement before an old pod is removed. A correct probe cannot compensate for a rollout strategy that permits too much capacity to disappear.

The fourth family involves routing outside the cluster. Cloud load balancers that target pod IPs directly keep their own target registry, and deregistration goes through a cloud API path that is slower and varies more in how long it takes. Pod readiness gates can hold a replacement in NotReady until the load balancer controller reports it healthy in the target group. That protects admission of the new pod, but does not confirm deregistration of the old one. Both sides of the handoff still matter.

The configuration pattern that closes all four gaps

The configuration starts with replacement capacity. In the rolling-update strategy, maxUnavailable set to 0 preserves the desired available replica count, while maxSurge set to 1 permits one extra pod to fund the replacement. This gives the new application time to prove itself without first removing a healthy old replica.

Size failureThreshold multiplied by periodSeconds to cover worst-case startup time. For example, failureThreshold set to 30 and periodSeconds set to 5 provide a 150-second budget. That allowance applies only during startup. Liveness and readiness remain suppressed during this window, so normal initialization is not mistaken for failure.

After startup succeeds, readiness should be fast, cheap, and narrowly scoped. Use a short periodSeconds and a low failureThreshold, with two consecutive failures pulling the pod from rotation quickly. Check application initialization and whether the local connection pool is open, rather than testing downstream databases or external APIs. Enable readiness last, after migrations have run and caches are primed. Keep liveness on a different endpoint, use it only for conditions that cannot self-recover, such as a goroutine deadlock, and set its initialDelaySeconds higher than readiness’s so it cannot interrupt normal startup.

Once the probe passes, a nonzero minReadySeconds holds the rollout before the replacement counts as Available. The pod must stay ready for that whole period. Without the hold, a pod can pass once, crash two seconds later, and still have allowed the rollout to advance. The hold turns short-lived startup success into a stalled rollout rather than a silent replacement of healthy pods with broken ones. Where a cloud load balancer targets pod IPs, a readiness gate also ties admission to target-group health.

When an old pod is finally removed, graceful termination covers the routing delay. Size the preStop sleep against the actual deregistration latency of the load balancer in use, especially where cloud target-group updates are involved. terminationGracePeriodSeconds must exceed the sum of the preStop delay and the application’s in-flight request drain time. This is the total shutdown budget before forced termination, including the hook.

What happens when the probe checks the wrong thing — two production cases

A probe that checks too much can remove healthy serving capacity all at once. In one production case, a routine deploy caused 503s across the service for 90 seconds even though the previous version was still healthy. The health endpoint checked a downstream dependency, so a brief database blip marked every replica unready and left the Service with no endpoints. A failureThreshold of 1 and no startup grace made the transient failure an immediate removal from traffic. Checking a shared dependency had coupled every pod’s readiness to the same failure.

KEDA’s Metrics Server reported Ready while its gRPC connection to the keda-operator was down, so the HPA had no scale signals even though the replica looked healthy. The HTTP check passed, but the service could not do its job. The fix made the check active by triggering a non-blocking reconnect when the accessor observed an idle connection before returning. Both cases were probe-design failures, because Kubernetes acted on the answers it received.

Special cases that require a different probe strategy

Slow JVM services need enough time to boot to cover how long they actually take to warm up. Without a startup probe, liveness can kill a correctly written application before initialization is done, and each retry can hit the same limit until the pod enters CrashLoopBackOff. A startup probe with a high enough failureThreshold gives initialization time to finish before the faster steady-state checks take over. For Spring Boot, configure server.shutdown=graceful as well. Align spring.lifecycle.timeout-per-shutdown-phase with the Kubernetes terminationGracePeriodSeconds budget so in-flight requests can drain before SIGKILL.

Stateful workloads need checks that understand more than process responsiveness. A team expanding a Redis Operator deployment from one replica to two encountered unintended master promotion because the operator’s readiness probe used a simple ping that ignored replication status. The probe passed too early, so the rolling update led Sentinel to promote an empty replica and data was lost. Audit operator-managed probe logic before relying on defaults for stateful workloads.

GPU inference pods need the same separation between loading and serving as JVM applications. Their startup probe must cover the full model-load window before readiness gating begins, with the budget driven by model size and object-storage download time. KServe supports rolling updates with canary traffic splitting and native inference runtimes, but those features still require appropriately designed probes. Scale-to-zero makes the startup budget especially consequential. A probe that allows too little time can produce CrashLoopBackOff precisely when a traffic burst triggers the first scale-up.

PodDisruptionBudgets and minReadySeconds as the rollout's safety net outside the probe loop

Node drains, autoscaler evictions, and maintenance can disrupt pods without passing through the Deployment’s replacement loop. Readiness controls when a pod can get traffic, while minReadySeconds holds the rollout to check that the pod stays ready. PodDisruptionBudgets address the separate problem of voluntary eviction during cluster operations.

A PDB with maxUnavailable set to 0 permits no voluntary eviction of matching pods. Its disruptionsAllowed remains 0 regardless of the current replica count, blocking drains that need to evict those pods. That protection has a cost, because the autoscaler cannot consolidate underutilized nodes when it cannot move their workloads. The most protective setting is also the least friendly to spot-node recycling or tighter bin-packing.

Use maxUnavailable set to 1 for non-critical deployments and reserve 0 for services where even a momentary capacity dip is unacceptable. Document that tradeoff explicitly, so a blocked maintenance operation is understood as the chosen protection. Keeping services available during deploys and cluster operations directly supports SOC 2 availability criteria, and rolling updates backed by readiness checks provide a way to verify that this commitment is being met.

Verifying that the configuration is actually working

Verification has to follow requests through an actual rollout. kubectl rollout status reports ReplicaSet transitions. A 40-second 502 window belongs in the load balancer access log and application error rate. Run a continuous low-volume request loop against the service endpoint during a triggered rollout and watch for non-2xx responses. An error-free request stream across the full rollout is the evidence the configuration is meant to produce.

Watch kubectl get endpoints to check that pods enter and leave traffic rotation in the expected order, and check kubectl describe pod for probe failures and lifecycle events. Ingress and load balancer access logs show what users experienced, while kubectl rollout history establishes revision tracking for rollback. Remember that kubectl rollout undo uses the same readiness-gated replacement loop in reverse. Fix a faulty probe before rolling back, because the same wrong answer can drop traffic in either direction.