Graceful Shutdown in Kubernetes: SIGTERM, preStop, and the Endpoint Race

It always happens during a deploy. You roll out a new version of your service, the old pods shut down “cleanly” as far as Kubernetes is concerned, and yet a handful of users get connection errors for a few seconds. Nothing crashed. Nothing errored in your logs. The requests were simply delivered to a process that was already gone.

This is the graceful shutdown problem, and in Kubernetes it is really two problems wearing the same costume. The first is that your application must actually handle the termination signal instead of dying mid-request. The second is subtler: there is a race between the control plane removing your pod from the service’s endpoint list and the load-balancing rules that clients have already loaded. Getting both right is mostly a matter of understanding the exact sequence Kubernetes runs when a pod dies, and then budgeting your time against it.

The termination sequence, step by step

When you delete a pod, the termination sequence is precise and worth memorizing. The pod is marked Terminating in the API server with a deadline — the termination grace period, 30 seconds by default. From that moment, two things happen in parallel:

  • On the node, the kubelet starts shutting the container down: it runs the preStop hook if one is defined, then sends SIGTERM to the main process. If the process is still alive when the grace period expires, it receives SIGKILL.
  • In the control plane, the pod is removed from the EndpointSlice objects that back its Service, so future traffic stops being routed to it.

Neither step waits for the other. The kubelet does not pause while the endpoints propagate, and the endpoints controller does not wait for your process to finish its in-flight requests. That is the whole race in one sentence: traffic can keep arriving at your pod during the window between “endpoint removed from the API” and “rules actually updated on every node and client”, and your process must survive that window.

Two details from the lifecycle docs matter more than people expect. First, the grace period clock starts before the preStop hook runs — the hook is not extra time, it comes out of the same budget, and the hook must complete before the TERM signal is even sent. Second, if the hook is still running when the grace period expires, the kubelet grants a one-off extension of just 2 seconds before killing everything. A hook that sleeps 10 seconds inside a 30-second budget that also needs 25 seconds for connection draining will not do what you think it does.

Step 1: actually handle SIGTERM

If your container’s main process does not catch SIGTERM, it dies immediately, dropping every in-flight request. For Go HTTP services the fix is a shutdown routine that stops accepting new connections, waits for the active ones to finish, and then exits:

package main

import (
	"context"
	"errors"
	"log"
	"net/http"
	"os"
	"os/signal"
	"syscall"
	"time"
)

func main() {
	mux := http.NewServeMux()
	mux.HandleFunc("/healthz", func(w http.ResponseWriter, r *http.Request) {
		w.WriteHeader(http.StatusOK)
	})

	srv := &http.Server{
		Addr:    ":8080",
		Handler: mux,
	}

	go func() {
		if err := srv.ListenAndServe(); err != nil && !errors.Is(err, http.ErrServerClosed) {
			log.Fatalf("listen: %v", err)
		}
	}()

	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGTERM, os.Interrupt)
	defer stop()

	// Block until the kubelet sends SIGTERM.
	<-ctx.Done()
	log.Println("SIGTERM received, draining connections")

	// Give in-flight requests a bounded window to complete.
	shutdownCtx, cancel := context.WithTimeout(context.Background(), 20*time.Second)
	defer cancel()

	if err := srv.Shutdown(shutdownCtx); err != nil {
		log.Printf("forced shutdown: %v", err)
	}

	// After the HTTP server is drained, close anything it depended on:
	// database pools, message-broker connections, metrics flushers.
	log.Println("shutdown complete")
}

http.Server.Shutdown closes the listener, closes idle connections, and waits for active requests to finish — but it will not wait forever, so the timeout on the context is your safety net. Pick that timeout deliberately: it must fit inside the pod's grace period with room to spare, because the grace period also has to cover the preStop hook and anything else the container does on the way down.

The most common mistake in this pattern is treating shutdown as an afterthought in code review: the server drains, but the database pool closes instantly under a still-running background job, or a message consumer acknowledges nothing and redelivery storms another replica. Anything the process started owns cleanup responsibilities in the same window.

Step 2: buy time with preStop

Handling SIGTERM correctly fixes dropped in-flight requests, but not the endpoint race. Clients — and node-level proxies using iptables or IPVS — learn about endpoint changes through watches that take time to propagate. During that propagation window, new connections can still be steered toward your dying pod. The standard mitigation is a short preStop delay, so the endpoint removal gets a head start before your process stops accepting traffic. Modern Kubernetes supports this directly with a sleep handler in the lifecycle hooks:

spec:
  terminationGracePeriodSeconds: 45
  containers:
    - name: api
      image: registry.example.com/api:1.42.0
      lifecycle:
        preStop:
          sleep:
            seconds: 5

Five seconds is a common starting point; clusters with large endpoint fan-out or slow client refresh intervals sometimes need more. What matters is the arithmetic. The grace period is a single budget that must cover the preStop hook plus your application's own drain time. With a 5-second sleep and a 20-second drain, a 30-second default grace period technically fits but leaves nothing for startup signal latency, so raising terminationGracePeriodSeconds to 45 is the honest configuration. Kubernetes documents this constraint explicitly: if the hook takes 55 seconds and normal shutdown takes 10, a 60-second grace period still results in a force kill, because 55 plus 10 exceeds 60. Budget from measured numbers, not guesses.

One more subtlety: if your pod has sidecar containers, their startup and shutdown ordering is handled specially by the kubelet, but plain containers receive TERM in arbitrary order. If a proxy sidecar must outlive the application container during shutdown, that ordering has to be arranged deliberately rather than left to chance.

Step 3: close the long-lived connections

HTTP request draining covers request-scoped traffic. Long-lived connections are a different beast: gRPC streams, WebSocket sessions, and database connections do not end on their own, so "wait for active requests to finish" can mean waiting for the heat death of the universe. The endpoint-list tutorial Kubernetes publishes for draining connections during pod termination is essentially a walkthrough of this problem — the server must actively close or finish these connections when it receives SIGTERM, and clients must treat a closed connection as a signal to re-resolve endpoints, not to retry the same address.

For gRPC specifically, servers should send GOAWAY on shutdown so clients reconnect to other pods; most client libraries do this automatically when the server closes its listeners cleanly, which is one more reason the orderly Shutdown path matters. Keep-alive HTTP clients deserve the same scrutiny: a client pinned to a connection pool it never refreshes will keep using stale endpoints for minutes regardless of how clean your shutdown is.

A shutdown checklist

  • Handle SIGTERM explicitly and drain HTTP traffic with a bounded timeout.
  • Add a short preStop sleep so endpoint removal propagates before you stop accepting connections.
  • Size terminationGracePeriodSeconds as hook time plus drain time, with margin — not the default 30 seconds if your drain needs 20.
  • Actively close long-lived connections (streams, WebSockets) during shutdown; never wait for them to end naturally.
  • Close dependency pools and flush metrics after the traffic drain, inside the same grace period.

None of this is exotic. It is five lines of lifecycle YAML and a shutdown function you write once per service. The reason dropped requests survive in so many codebases is that each individual piece — the signal handler, the hook, the grace period — looks optional in isolation, and the failure only appears in the seconds where all three intersect. Fix them together, once, and deploys stop being a minor outage.

Leave a Reply

Your email address will not be published. Required fields are marked *