Kubernetes ships with a promise: change an image tag, and the platform replaces your pods gradually, keeping the old version serving until the new one can take over. For a lot of teams, that promise breaks the first time real traffic hits a real deploy. Users see a burst of 502s and everyone quietly learns to deploy at 2 a.m.
The gap between promise and reality is configuration. A stock Deployment does roll pods, but it has no idea when a container can actually serve requests. Without a readiness probe, a pod counts as ready the moment its container process starts — long before the app has finished booting, running migrations, or warming caches. The rollout proceeds, traffic shifts, and requests land on a process that isn’t listening yet.
This post walks through the pieces that turn a default Deployment into a genuinely zero-downtime one: the probe that actually gates traffic, the rollingUpdate knobs, graceful shutdown, PodDisruptionBudgets for node upgrades, and a complete manifest you can adapt.
The Default That Serves 502s
A stock Deployment already uses a rolling strategy: maxUnavailable: 25% and maxSurge: 25%. Old pods are terminated and new ones created in small batches until the rollout completes. The mechanics are fine — what’s missing is information. With no readiness probe defined, the kubelet has exactly one signal for readiness: is the container process running? The moment it is, the pod’s IP lands in the Service’s endpoints and traffic starts flowing to it.
Running is not the same as ready. A JVM app can take forty seconds to boot. The process can be up while models are still loading into memory. Every one of those windows is time when the cluster is happily routing live traffic to a pod that will drop the connection. That’s the 502 burst — on every deploy, because the rollout is working exactly as configured. The fix isn’t clever infrastructure; it’s telling the cluster what “ready” means.
Three Probe Types, One That Matters on Deploy Day
Kubernetes defines three kinds of container probes, and confusing them is the most common source of deploy pain. All three run periodically against the container, and all three support the same handlers — httpGet, tcpSocket, and exec — but they answer completely different questions.
The startup probe answers “has this process finished booting?” Until it passes, the other two probes don’t run at all. This is what lets a slow-starting app also have a strict liveness probe without getting restarted mid-boot.
The readiness probe answers “should this pod receive traffic right now?” It is the only probe wired into Service endpoint management. When it fails, the pod’s IP is pulled from the endpoints — no restart, just no traffic. During a rollout, this is the signal that gates progress: a new pod only counts toward the rollout once it’s actually ready.
The liveness probe answers “is this process wedged?” Fail it enough consecutive times and the kubelet restarts the container. It has nothing to do with traffic — it’s a self-healing mechanism for deadlocks and leaks, and misconfigured, it’s the fastest way to restart-loop your own service.
For zero-downtime deploys, readiness is the one that matters. But you configure all three, because they protect each other: startup covers slow boots, readiness covers traffic, liveness covers wedged processes. If your service speaks gRPC, expose an endpoint implementing the standard health-checking protocol and probe that instead.
containers:
- name: web
image: registry.example.com/web:1.42.0
ports:
- containerPort: 8080
startupProbe: # gates the other two until boot completes
httpGet:
path: /healthz
port: 8080
periodSeconds: 3
failureThreshold: 30 # allow up to ~90s to finish starting
readinessProbe: # the one deploys depend on: controls traffic
httpGet:
path: /ready
port: 8080
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 3
successThreshold: 1
livenessProbe: # controls restarts, never traffic
httpGet:
path: /healthz
port: 8080
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3One design note: point readiness at an endpoint that reflects genuine ability to serve — dependencies reachable, caches primed — while liveness checks only the process itself. If readiness fails when a database blips, your pods vanish from endpoints (good). If liveness fails on the same blip, Kubernetes restarts every pod simultaneously (very bad).
maxSurge and maxUnavailable: Trading Capacity for Speed
Once pods report real readiness, you can tune the rollout itself. Two knobs control it, and they pull in opposite directions. maxUnavailable is how many pods you’re willing to take offline during the rollout; maxSurge is how many extra pods you’re willing to run above the desired count while it happens. Both default to 25%, and both accept absolute numbers or percentages.
The defaults are a compromise. Allowing 25% unavailability means serving capacity dips by a quarter mid-rollout — fine with headroom, painful when you’re running hot. A 25% surge means the cluster must have room for the extra pods; if it can’t schedule them, the rollout stalls — on a full cluster, forever.
The combination that guarantees capacity during a deploy is maxUnavailable: 0 with maxSurge: 1: the platform starts one new pod, waits for it to report ready, and only then terminates an old one. You never serve below your desired replica count. The cost is one extra pod’s resources, and replacements happen one at a time — safe but slow on large fleets, which is why big deployments often widen the surge percentage while keeping unavailability pinned at zero.
spec:
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0 # never serve below the desired replica count
maxSurge: 1 # ...but pay with one extra pod while rolling
# Defaults for comparison: both 25%
# Fast, headroom-rich alternative: maxUnavailable: 0, maxSurge: 25%preStop Hooks and Graceful Termination
Readiness probes handle the start of a pod’s life. The end needs its own handling, because termination is asynchronous from two directions. When a pod is deleted, the API server marks it terminating, endpoint controllers remove its IP from the Service, and the kubelet sends SIGTERM to the process — all roughly in parallel, with no ordering guarantee. If your app closes its listener the instant SIGTERM arrives, requests still in flight — and requests routed during the last second before endpoint propagation completes — get connection resets.
The standard mitigation is embarrassingly low-tech: a preStop hook that sleeps a few seconds. The hook runs before SIGTERM is delivered, so the sequence becomes: pod marked terminating → endpoints updated → sleep covers the propagation window → SIGTERM arrives → your handler drains in-flight requests → exit. Ten seconds of sleep covers propagation on almost any cluster.
terminationGracePeriodSeconds (default 30) is the overall budget — preStop execution plus shutdown time must fit inside it, or the container gets SIGKILL mid-drain. If your app needs twenty seconds to drain connections, budget 45 or more.
containers:
- name: web
lifecycle:
preStop:
exec:
command: ["sleep", "10"] # cover endpoint propagation before SIGTERM
terminationGracePeriodSeconds: 45 # preStop + drain must fit inside thisNode Upgrades Need PodDisruptionBudgets
Everything above protects you from your own deploys. Node upgrades are the other half of the story: when a cluster upgrades, every node gets drained, and drain uses the eviction API — which does not consult your Deployment’s rollout strategy. Without a guardrail, a single drain can evict multiple replicas of your service at once. Same 502 burst, different trigger.
A PodDisruptionBudget tells the eviction API how much it’s allowed to take: minAvailable: 2 means any voluntary disruption must leave at least two ready pods, and a drain that would violate that waits and retries until it can proceed safely. Control planes and managed platforms honor budgets during upgrades — Azure’s container documentation, for instance, covers how node upgrades interact with them. One caution: don’t set minAvailable equal to your replica count, or nothing can ever be evicted and node drains stall.
Putting It Together: A Deployment Built for Zero Downtime
Here’s the whole picture in one file — Deployment and budget together, since they’re applied as a pair. Four replicas never dip below four ready pods during deploys; slow boots are gated by the startup probe; shutdown drains for ten seconds before SIGTERM with a 45-second grace budget; and node drains must leave two pods serving.
apiVersion: apps/v1
kind: Deployment
metadata:
name: web
labels:
app: web
spec:
replicas: 4
selector:
matchLabels:
app: web
strategy:
type: RollingUpdate
rollingUpdate:
maxUnavailable: 0 # capacity floor during deploys
maxSurge: 1 # surge budget for replacements
template:
metadata:
labels:
app: web
spec:
terminationGracePeriodSeconds: 45
containers:
- name: web
image: registry.example.com/web:1.42.0
ports:
- containerPort: 8080
startupProbe: # slow boots allowed, liveness held back
httpGet:
path: /healthz
port: 8080
periodSeconds: 3
failureThreshold: 30
readinessProbe: # traffic gate: rollout + Service endpoints
httpGet:
path: /ready
port: 8080
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 3
livenessProbe: # wedged-process recovery only
httpGet:
path: /healthz
port: 8080
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
lifecycle:
preStop:
exec:
command: ["sleep", "10"] # drain window before SIGTERM
---
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
name: web
spec:
minAvailable: 2 # voluntary disruption floor (node drains)
selector:
matchLabels:
app: webCommon Failure Modes
Almost every broken rollout traces back to one of these:
- The probe points at the wrong port or path — copied from another service’s manifest, or the health endpoint is bound to a management port that isn’t what the probe targets. The pod runs fine but never goes Ready, and
kubectl describe podshows repeated probe failures. - Liveness is too aggressive. Tight timeouts plus a small failure threshold means a GC pause or CPU contention fails the probe, Kubernetes restarts the container, and the restart burns cache and connections — which slows neighbors down and cascades. Fix: add a startup probe, loosen
failureThresholdandtimeoutSeconds. - No PodDisruptionBudget before a cluster upgrade — a node drain takes several replicas at once and you relive your deploy outage with a different cause. The inverse bug,
minAvailableequal to replicas, silently blocks node drains and stalls the upgrade instead. - preStop plus shutdown time exceeds
terminationGracePeriodSeconds, so the drain gets cut short by SIGKILL anyway.
# Apply and watch the rollout gate on real readiness
kubectl apply -f web.yaml
kubectl rollout status deployment/web # blocks until complete or stalled
# When pods sit in Pending/Ready-never, inspect probe events
kubectl describe pod -l app=web | grep -A 4 -i probe
kubectl get events --field-selector reason=Unhealthy
# Roll back a bad release in one command
kubectl rollout undo deployment/web
# Confirm the budget before upgrade season
kubectl get pdbWrapping Up
Zero-downtime deploys aren’t a feature you enable — they’re the aggregate of a handful of small honesties: telling the cluster when a pod can serve (readiness), separating boot time from wedged-process detection (startup versus liveness), never dropping below capacity mid-rollout (maxUnavailable: 0), letting connections drain (preStop plus a real grace period), and protecting replicas from node drains (PodDisruptionBudget). None of it is exotic, and together it’s the difference between deploying at 2 a.m. and deploying whenever. Adapt the manifest, watch rollouts with kubectl rollout status, and keep the failure modes handy for the day a pod refuses to go Ready.