Environment Variables Are Not a Secrets Strategy: A Ladder of Practical Alternatives

Environment variables are the default way applications receive configuration, and secrets ride along by inertia: a database password, an API token, a signing key — all delivered as process environment. It works until the first leak, and leaks come from more directions than most teams account for. The environment of a running process is not a private vault; it is a globally readable-at-the-edges string map that gets inherited, logged, dumped, and committed with surprising regularity.

This post maps the actual leak surfaces, then walks up a practical ladder of alternatives — encrypted files in git, Kubernetes-native secret delivery, external secret operators, and finally workload identity that eliminates static credentials altogether. The ladder is ordered by how much operational weight each step adds; climb as high as your threat model demands, not further.

Why Environment Variables Leak

The environment block is exposed in more places than developers usually realize:

  • Process introspection: any user on a shared host who can read /proc/PID/environ — or run ps e — sees every variable in cleartext.
  • Container tooling: docker inspect and the equivalent Kubernetes pod output print environment variables to anyone with read access to the resource.
  • Inheritance: every child process gets a copy. A debugging CLI, a shell you drop into, a crash reporter — all inherit your production credentials.
  • Debug endpoints: framework instrumentation that exposes runtime state (Spring Boot’s actuator env endpoint is the classic example) has repeatedly leaked secrets through misconfigured actuator exposure.
  • Crash data: core dumps and error-reporting breadcrumbs routinely capture environment snapshots.

Git is the other half of the problem. Credentials pasted into a .env file that gets committed once live forever in history, and history survives force-pushes on every clone. Scanning tools like gitleaks and trufflehog catch new leaks — and should run in CI — but the deeper failure is procedural: rotating a leaked credential means revoking and reissuing everywhere, and rewriting git history doesn’t help because the attacker already has the clone. Once a secret is static and long-lived, every leak is an incident.

Step Up 1: Encrypted Files in Git

The cheapest meaningful upgrade keeps secrets in git but encrypts them at rest. SOPS encrypts values while leaving keys in YAML or JSON files readable, so diffs stay useful, and it pairs naturally with age, a small modern encryption tool with explicit, simple key files:

# Encrypt a config file in place with a recipient's age public key
sops --encrypt --age age1replace-with-real-recipient-key --in-place config/secrets.yaml

# Decrypt to stdout for consumption — never write plaintext back to the repo
sops --decrypt config/secrets.yaml

The plaintext never touches disk in the repo, git history holds only ciphertext, and decryption happens at deploy time with the right private key. The weakness is key distribution: everyone authorized to deploy needs the private key somewhere, and “somewhere” tends to become a CI variable — better than plaintext in git, but still a static credential. This step is ideal for small teams and single-cluster setups where a full secret manager would be overkill.

Step Up 2: Kubernetes-Native Delivery

In Kubernetes clusters, two complementary tools dominate. sealed-secrets lets developers encrypt a Secret with the cluster’s public key using kubeseal; the resulting encrypted manifest is safe to commit, and the in-cluster controller decrypts it into a real Secret. The private key never leaves the cluster, so the flow is strictly one-way: you can commit what the cluster can decrypt, and nothing else.

external-secrets takes a different shape: the git repo contains only a declarative reference — an ExternalSecret pointing at a secret in a provider such as AWS Secrets Manager, GCP Secret Manager, or Vault — and the operator continuously syncs the real value into a cluster-native Secret:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: payments-db
  namespace: payments
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: prod-aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: payments-db-credentials
  data:
  - secretKey: password
    remoteRef:
      key: prod/payments/db
      property: password

The operator pattern has a rotation property sealed-secrets lacks: because the value lives in the provider and syncs on an interval, rotating the credential in the provider propagates to workloads without touching git at all. Whichever tool you use, also enable etcd encryption at rest for native Secrets — Kubernetes encodes them in base64, which is transport plumbing, not encryption — and keep RBAC tight, because “get secrets” in a namespace is effectively read access to everything that namespace protects.

Step Up 3: Dedicated Secret Managers and Dynamic Credentials

HashiCorp Vault remains the heavyweight option — centralized storage, leases, dynamic database credentials that are generated on demand and expire automatically, and audit trails on every read. Note that Vault moved to a non-open-source business license in 2023; the community fork OpenBao continues under the Linux Foundation with an MPL license if that matters to your org. The real payoff of this tier isn’t storage — it’s dynamic secrets. A database credential that lives for one hour and dies can leak without becoming an incident.

For CI pipelines, you can often skip static cloud credentials entirely with workload identity federation. GitHub Actions can mint a short-lived OIDC token per run, and your cloud provider’s IAM trusts that token to assume a scoped role — no stored cloud keys anywhere:

permissions:
  id-token: write   # required to request the OIDC token
  contents: read

jobs:
  deploy:
    runs-on: ubuntu-latest
    steps:
    - uses: aws-actions/configure-aws-credentials@v4
      with:
        role-to-assume: arn:aws:iam::123456789012:role/github-deploy
        aws-region: eu-west-1

The trust policy on github-deploy restricts assumption to your repository, branch, and workflow — so a leaked token is worthless outside the exact pipeline that minted it. Both AWS and Google Cloud document this pattern as their recommended replacement for long-lived service-account keys.

Picking a Step Without Over-Engineering

  • Small team, single repo, config-as-code: SOPS plus age covers you with near-zero infrastructure.
  • Kubernetes, gitops workflow: sealed-secrets for cluster-specific values, external-secrets when you want provider-side rotation to propagate.
  • Multi-cluster, compliance requirements, many consumers: Vault or OpenBao, ideally issuing dynamic credentials rather than storing static ones.
  • CI-to-cloud access: OIDC federation — there is almost never a reason left to store static cloud keys in a pipeline.

Whatever tier you land on, keep the scanners running. Credential hygiene is not a one-time migration; it is a loop of detection, rotation, and — increasingly — eliminating static secrets so there is nothing left to rotate. The environment variable is a convenient default, not a storage design, and treating it as one is how credentials end up in core dumps, actuator endpoints, and git history.

Leave a Reply

Your email address will not be published. Required fields are marked *