Every team starts with environment variables because every framework supports them. A DATABASE_URL in a .env file, a compose file passing secrets through environment:, maybe a Kubernetes Secret mounted as env vars in the pod spec. It works — right up until it doesn’t. Environment variables leak through crash dumps, child process inheritance, /proc/<pid>/environ, debug endpoints, and CI logs with depressing regularity. The reality that your secrets end up readable by anything that can inspect the process is not a hypothetical threat model; it’s the default behavior of the platform.
This post is about the step after environment variables: what a real secrets management setup looks like, the trade-offs between the mainstream options, and the rotation patterns that turn “we have a secrets store” into “compromised credentials stop mattering.” The OWASP Secrets Management Cheat Sheet is a good map of the territory; what follows is a practitioner’s route through it.
Why environment variables fail quietly
The core problem with env vars isn’t that they’re stored in memory — everything is stored in memory. It’s that they’re copied wholesale to every child process and exposed to anything that can read process metadata. A few concrete leak paths:
- Crash reports and core dumps capture the full environment block and often get pasted into issue trackers verbatim.
/procintrospection makes the environment of any process on the same host readable by root — and in containers sharing a PID namespace, by neighbors.- Shell history and CI logs catch secrets when someone debugs with
envorprintenv, or when a build tool echoes its environment on failure. - Composition inheritance means a process that shells out to a helper tool hands it your database password whether the helper needs it or not.
None of these are exotic attacks. They’re the failure modes of everyday debugging. The design goal of a better setup is simple to state: minimize the number of places a secret exists at rest, minimize the time it lives, and make every access an auditable event.
The two axes: where secrets live and how long they live
Every secrets tooling decision is a point on two axes. The first is storage location: a centralized manager (Vault, a cloud secrets manager, a Kubernetes Secret encrypted at rest) versus in-repo encrypted files (SOPS, Sealed Secrets). The second is lifetime: static secrets that live until someone rotates them versus dynamic secrets generated on demand and revoked automatically.
Static secrets in a vault are already a huge improvement over env vars, but they have a structural weakness: the secret exists continuously, so the window for theft is unlimited, and rotation depends on discipline and tooling. Dynamic secrets flip this around: they don’t exist until a client requests them, and the issuing system revokes them automatically when their TTL expires. A database credential that lives for one hour and is generated per-service per-session shrinks the blast radius of a leak from “until someone notices” to “about an hour,” and because issuance is an API call, every credential minted is logged with an identity attached.
That last point matters more than it sounds. With shared static credentials, when a leak surfaces, you can’t tell which service leaked it — everyone uses the same password. Dynamic secrets are naturally per-instance, which turns forensics from guesswork into a log query.
Comparing the mainstream options
Four approaches cover most real deployments. They’re not mutually exclusive — mature setups use several.
HashiCorp Vault: dynamic secrets and identity-based access
Vault (36k+ stars) is the heavyweight option. Its database secrets engine issues short-lived credentials on demand; its PKI engine issues certificates; its transit engine handles encryption as a service without storing data. Access is identity-based: workloads authenticate via Kubernetes service accounts, cloud IAM roles, or TLS client certs, and policies govern which paths they can read. The cost is operational — Vault is a stateful, clustered service that must be unsealed, backed up, and upgraded. It’s the right answer when you need dynamic secrets, cross-cloud secret brokering, or strict audit trails, and a heavy answer when you just have fifteen YAML files with API keys.
SOPS: encrypted files in Git
SOPS (23k+ stars, now a CNCF Sandbox project maintained under getsops after originating at Mozilla) takes the opposite approach: keep secrets as files, in the repo, encrypted with KMS keys (AWS, GCP, Azure), age, or PGP. The encrypted file commits to Git like any other artifact, so secret changes get review, history, and rollback for free. Decryption happens at deploy time. This is a superb fit for GitOps flows — Argo CD and Flux can apply SOPS-encrypted manifests — and for small teams that don’t want to run infrastructure. The trade-off is that secrets are still static: rotation means editing and re-encrypting, and access control is whoever holds the KMS key.
Sealed Secrets: GitOps-safe Kubernetes Secrets
Sealed Secrets (9k+ stars) solves one narrow problem well: you want to commit a Kubernetes Secret to Git without anyone — including cluster operators — being able to decrypt it from the repo. A controller in the cluster holds the private key; you encrypt a Secret with its public key using the kubeseal CLI, commit the resulting SealedSecret resource, and the controller decrypts it into a normal Secret in-cluster. Asymmetric encryption means the repo is safe by construction, but like SOPS it manages static secrets only, and it’s Kubernetes-specific.
External Secrets Operator: sync, don’t store
The External Secrets Operator (ESO) takes a third path: don’t put secrets in the cluster’s Git pipeline at all. Keep them in an external manager — AWS Secrets Manager, GCP Secret Manager, Azure Key Vault, Vault — and declare an ExternalSecret resource that references them. ESO syncs the values into native Kubernetes Secrets and keeps them refreshed. This gives you centralized management, provider-side audit logs, and IAM-based access control, while pods consume ordinary Secret volumes. For organizations already standardized on a cloud provider, ESO is usually the lowest-friction bridge between that provider and Kubernetes.
A quick decision guide: if your secrets are mostly static config and your deployment is GitOps-based, SOPS or Sealed Secrets covers you with almost no infrastructure. If you’re on a single cloud, that cloud’s secrets manager plus ESO gets you centralization and rotation without new components. If you need dynamic database credentials, certificate issuance, or cross-cloud brokering, that’s Vault territory.
Rotation: the pattern that actually reduces risk
OWASP’s cheat sheet groups rotation strategies from gradual (new keys for writes, old keys for reads) through rapid and scheduled rotation. The mechanics matter less than a property your system either has or lacks: can two credentials be valid at once? If yes, rotation is a deploy-time concern and can be fully automated. If no — if there’s exactly one database password and it changes atomically — rotation requires a coordination window and probably downtime or retry logic.
Dynamic secrets sidestep this entirely: each consumer mints its own short-lived credential, overlap is constant by design, and “rotation” is just the TTL expiring. For static secrets, the standard Kubernetes pattern from the OWASP guidance is a sidecar that authenticates with the secrets manager, fetches the current value, writes it to a shared in-memory volume, and refreshes periodically — while the application either watches the file for changes or reloads on a signal:
apiVersion: v1
kind: Pod
metadata:
name: my-app
spec:
serviceAccountName: my-app-sa
containers:
- name: app
image: my-app:1.4.2
volumeMounts:
- name: secrets
mountPath: /mnt/secrets
readOnly: true
- name: secrets-provider
image: hashicorp/vault:1.15.0
args: ["agent", "-config=/etc/vault/agent.hcl"]
volumeMounts:
- name: secrets
mountPath: /mnt/secrets
volumes:
- name: secrets
emptyDir:
medium: MemoryTwo details in that manifest carry most of the security value. The emptyDir with medium: Memory keeps the secret off the node’s disk, so it never lands in a container image layer, log, or tmpfs snapshot. And the sidecar authenticates with its own Kubernetes service account — the pod’s identity, not a long-lived token baked into a config map.
One caveat about Kubernetes Secrets specifically: base64 is encoding, not encryption. Secrets in etcd are only as safe as etcd’s encryption-at-rest configuration, which historically defaults to plaintext in many distributions. If your threat model includes someone getting read access to etcd — a compromised control plane, a forgotten backup — then RBAC alone isn’t the answer; enable etcd encryption or route secrets in from an external manager via ESO.
Application-side hygiene
How your code handles a secret after retrieval matters as much as where it came from. A few habits that prevent the post-delivery leaks:
- Read from a file, not the environment. Mount secrets as volumes (the sidecar pattern above, or plain Kubernetes Secret volumes) and read at startup or on change. Files can be permission-scoped; environments can’t.
- Never log configuration. Structured logging makes this easy to violate — a debug line that dumps the config struct ships the database password to your log aggregator. Redact by field name at the logger level.
- Zero what you can. Overwrite secret byte slices after use where the language allows it. It’s imperfect (garbage collection and string immutability get in the way), but it shrinks the window in core dumps.
- Watch dependencies. Secrets flow through libraries too — an HTTP client that logs all request headers will capture your Authorization header. Supply-chain review isn’t just about CVEs.
A pragmatic migration path
If everything currently runs on env vars, a realistic sequence, without a big-bang rewrite:
- Week 1: inventory. Grep repos and CI configs for credential patterns; get a real count of what exists and who consumes it. This is also where you find the secrets already committed to Git history — those need rotation regardless of what you adopt.
- Weeks 2-3: centralize. Pick one manager (cloud provider or Vault) and move secrets there, still delivering them as env vars if necessary. The win is a single access point with audit logging.
- Weeks 4-6: switch delivery. Move consumption from env vars to mounted volumes via a sidecar or ESO sync.
- Ongoing: shorten lifetimes. Move the highest-value credentials — databases first — to dynamic issuance or scheduled rotation with dual-key overlap.
The payoff compounds: once every secret access goes through one audited path, “which service used this credential and when” becomes a query instead of an investigation, and rotating a leaked key stops being an emergency. That’s the real goal — not a specific tool, but a world where credentials are boring, short-lived, and traceable.