Native Sidecar Containers in Kubernetes: Ordering, Jobs, and the End of Shutdown Races

The sidecar pattern has been part of the Kubernetes toolkit for as long as pods have existed: run a second container next to your application to ship logs, terminate TLS, sync secrets, or proxy traffic. It works, but the classic implementation — a second entry in spec.containers — has always had awkward edges. Sidecars and app containers start together with no ordering, die together with no ordering, and during a rolling update or a node drain, that lack of ordering causes very real production failures.

Kubernetes fixed this with native sidecar containers: declared as init containers with a restartPolicy: Always, they get exactly what the pattern always needed — startup ordering, independent restarts, and a shutdown guarantee. The feature was introduced as alpha in 1.28, went beta in 1.29, and is stable since 1.33. If you’re still defining sidecars the old way, this post walks through what changes, why the ordering matters, and how to migrate safely.

What Was Actually Broken

Three failure modes motivated the feature, and you’ve probably hit at least one.

Job containers that never finish. When a pod runs a Job, Kubernetes considers the pod succeeded only when all containers exit. A logging sidecar in spec.containers never exits — it tails a file forever — so the Job hangs in Running after the work is done. Teams resorted to hacks: sidecar images that watch for a done-file, or shared process-namespace signals, all to answer one question the container model couldn’t express: “is the main container finished?”

Startup races. A service-mesh proxy or a vault agent needs to be ready before the application starts making network calls. With everything in spec.containers, start order is undefined. The app boots, fails its first calls, and relies on retries to paper over the race — or crash-loops until the proxy happens to be up.

Shutdown races. On termination, containers get SIGTERM in parallel. If the app container exits first and the sidecar is still needed — say, to flush its log buffer or finish an in-flight proxy request — the pod can disappear from the network while work is unfinished. Reverse ordering is also what you want on startup: infrastructure first, application last.

The Declaration: An Init Container That Stays

A native sidecar is an init container with restartPolicy: Always. That one field changes its lifecycle semantics:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: api
spec:
  replicas: 2
  selector:
    matchLabels:
      app: api
  template:
    metadata:
      labels:
        app: api
    spec:
      initContainers:
        - name: service-proxy
          image: envoy:v1.31
          restartPolicy: Always   # this line makes it a sidecar
          ports:
            - containerPort: 15001
          # readinessProbe here contributes to Pod readiness
      containers:
        - name: app
          image: registry.example.com/api:1.8.2
          ports:
            - containerPort: 8080

The kubelet starts init containers in list order and waits for each to be running — or passing its startup probe — before starting the next. A sidecar’s readiness probe feeds the pod’s overall readiness, so the proxy is verified before the app container is even pulled. On termination, the kubelet keeps sidecars running until the main containers have fully exited, then stops the sidecars in reverse order of their appearance.

Because sidecars live in the init container list, they mix freely with regular init containers. A vault-agent init container that fetches secrets (runs once, exits) followed by a proxy sidecar (runs forever) composes into a deterministic startup sequence:

initContainers:
  - name: fetch-secrets          # regular init: runs once, exits
    image: vault-agent:1.18
    args: ["-config", "/vault/config.hcl"]
  - name: service-proxy          # sidecar: starts after secrets exist
    image: envoy:v1.31
    restartPolicy: Always
    startupProbe:
      httpGet:
        path: /ready
        port: 15000
      failureThreshold: 30
      periodSeconds: 1

What This Fixes in Practice

  • Jobs complete. Sidecars don’t block Job completion. The main container exits, the Job records success, and the sidecar is torn down with the pod. No done-file tricks.
  • Rolling updates drain correctly. During a rollout, the old pod’s proxy stays up while the app finishes in-flight requests — exactly the guarantee zero-downtime deploys need. This composes with terminationGracePeriodSeconds and preStop hooks: the app drains, then the mesh sidecar goes away.
  • Sidecars restart independently. A crashed sidecar restarts without taking the app container down, and you can raise its restart count alerts independently of the main container.
  • Cleaner probes and resources. Startup, liveness, and readiness probes work per sidecar, and resource requests/limits are enforced per container as usual — the scheduler sees the pod’s effective totals.

Migrating Existing Sidecars

Moving a sidecar from spec.containers to spec.initContainers with restartPolicy: Always is usually a cut-and-paste, but check these before you roll it out:

  • Order matters. Init containers start strictly in list order. Put sidecars that must be ready first (proxies, agents) at the top. If a sidecar has no startup probe, “running” counts as started — add a startup probe to real infrastructure sidecars so readiness is genuine.
  • App-to-sidecar localhost communication changes timing. If your app talks to the sidecar over localhost, it can no longer start “before” the sidecar in any observable way — calls simply succeed because ordering is guaranteed. But if the sidecar was previously restarting alongside the app and your app retried on connection refused, those retries become dead code; keep them anyway as defense.
  • Workload support. Native sidecars work in Deployments, StatefulSets, DaemonSets, and Jobs. If you run a very old cluster (pre-1.29), the feature gate is off and an init container with restartPolicy set will be rejected — so validate manifests against your minimum supported version in CI.
  • Don’t move everything. A helper container that genuinely shares the app’s fate — e.g., a busybox that tail-runs a debug command — can stay in spec.containers. The native form is for services the pod needs before the app starts and after it stops. The official adoption tutorial has a worked example.

One more migration note: because a sidecar delays the start of everything after it, a broken sidecar (bad image, failing startup probe) now blocks the pod from ever reaching the app container. Your init-container failure alerting should treat sidecar crash-loops with the same severity as app failures.

The Pattern in the Wider Ecosystem

The sidecar pattern itself hasn’t changed — a co-located process extending the app container without touching its code. What changed is that Kubernetes finally models its lifecycle explicitly. Service meshes like Istio and Linkerd moved their data plane injection toward native sidecars for exactly the ordering guarantees above; agents like OpenTelemetry collectors and secret rotators benefit identically. In environments outside Kubernetes — Docker Compose, Nomad — the same pattern exists, but you’re wiring the ordering yourself.

If your platform team has been patching over startup and shutdown races with retry loops and sleep statements, this is the cleanup those hacks were waiting for. The sidecar containers concept page and the pod lifecycle reference cover the full termination semantics, and it’s worth reading the termination section carefully before your next node-pool upgrade — the sidecar shutdown ordering is what makes drains quiet.

Leave a Reply

Your email address will not be published. Required fields are marked *