Secrets Management in Practice: From Hardcoded Credentials to Short-Lived Identity

Most breaches involving secrets do not involve sophisticated attacks. They involve a credential that was committed to a git repository years ago, pasted into a CI log, embedded in a container image, or shared across four services because rotating it “might break something.” The OWASP Secrets Management Cheat Sheet opens with exactly this observation: secrets are hardcoded in source and scattered across configuration files everywhere, and the fixes are organizational before they are technical.

This post is a practical tour of secrets management for engineering teams: where secrets actually leak, how to structure a secrets architecture that survives audits and incidents, and how to handle the two hardest operational problems — rotation without downtime, and cleanup after a leak. The examples lean on Kubernetes and cloud-managed vaults, but the principles transfer to any stack.

Where Secrets Actually Leak

Before designing a vault architecture, it helps to be honest about the leak vectors. In practice, almost all of them fall into five buckets — and the git history one is the most common by a wide margin:

• Source control history. A credential committed once lives in git history forever. Deleting the file in a later commit removes nothing — every clone still has it.
• CI/CD logs and artifacts. Commands that echo environment variables, test frameworks that dump config on failure, and build logs uploaded to third-party observability tools.
• Container images. A secret in a layer stays in the layer even when a later layer deletes the file. Image registries are effectively public archives with an access control list.
• Shared long-lived credentials. One database password used by five services means one leak compromises everything and no log can tell you which service leaked.
• Local developer machines. Dotfiles, shell histories, and unencrypted .env files synced to backup services.

The mitigation for the first three is scanning and hygiene: pre-commit secret scanning, log redaction, and multi-stage builds that never copy credential files into any layer. The mitigation for the last two is architectural, and it is the real subject of this post.

Centralize, Standardize, and Audit

The core recommendation of every serious secrets architecture is centralization: one system (or one system per environment tier) is the authoritative source for secrets, with access control and audit logging on every read. Cloud providers offer managed options — AWS Secrets Manager, Google Secret Manager, Azure Key Vault — and self-hosted deployments typically use HashiCorp Vault or OpenBao. The specific product matters less than three properties:

• Every access is authenticated, authorized, and logged. When an incident happens, “which service read this credential and when” must be answerable from an audit trail, not guessed.
• Availability matches your workloads. The vault is on the critical path of application startup and credential rotation. A vault outage that blocks every pod from starting is an outage multiplier — plan for replication, cached fallbacks, and break-glass procedures.
• One interaction pattern. Teams should consume secrets the same way regardless of which backend stores them. This is what integrations like the Kubernetes Secrets Store CSI Driver provide: a standard mount interface over multiple providers.

Sharing one secret across many services is the anti-pattern that undermines all of this. Prefer per-service credentials — ideally per-service identities — so access grants and audit logs have per-service resolution.

Short-Lived Credentials Beat Rotated Ones

The lifecycle of a static secret is a countdown to a leak that somebody forgot to respond to. The strongest architectural move is to shorten credential lifetimes until leakage stops being an emergency. In rough order of preference:

• Workload identity federation. Cloud IAM lets workloads authenticate as themselves — IRSA on EKS, Workload Identity on GKE, managed identities on AKS — with no stored credential at all. Kubernetes-issued tokens are short-lived JWTs that the cloud provider exchanges for scoped cloud credentials. This should be your default for anything that talks to cloud APIs.
• Dynamically issued credentials. Vault’s database secrets engine mints short-TTL database users on demand: the application requests credentials at startup, uses them for hours, and they expire. Nothing is stored anywhere to steal, and revocation is automatic.
• Automated rotation of static secrets. When a third party only offers a static API key, automate rotation: two active versions, an overlap window for in-flight processes, and a schedule. Manual rotation scheduled “quarterly” is a calendar entry that quietly becomes annual.

A standard pattern for the database case keeps credentials out of application config entirely. An external secrets controller syncs from the vault into a Kubernetes Secret, or a CSI-mounted volume delivers fresh material to the pod filesystem:

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: orders-db
  namespace: orders
spec:
  refreshInterval: 1h0m0s
  secretStoreRef:
    name: aws-secrets-manager
    kind: ClusterSecretStore
  target:
    name: orders-db-credentials
  data:
    - secretKey: username
      remoteRef:
        key: prod/orders/db
        property: username
    - secretKey: password
      remoteRef:
        key: prod/orders/db
        property: password

The application reads a normal Kubernetes Secret; the vault remains the single source of truth; rotation happens in the vault and propagates on the next sync. Nothing touches a developer’s laptop or a Helm values file.

In-memory handling deserves a mention here, because it closes the container-log and core-dump leak paths. Applications should read secrets once at startup, keep them out of structured logs and error messages, and — where the language makes it cheap — zero the buffers after use. This is hardening, not a substitute for short lifetimes.

Secrets in CI/CD Pipelines

Pipelines deserve their own section because they combine two risks: they hold the most powerful credentials in the organization (deploy keys, registry push tokens, cloud deploy roles), and their logs are copied to more places than almost any other output. The rules that matter most:

• Pipelines should hold no long-lived secrets at all where OIDC federation exists. GitHub Actions can exchange a workflow-signed OIDC token for short-lived cloud credentials scoped to a single role — the official OIDC configuration guide covers the provider setup. Deploy keys and registry tokens minted per-run expire with the run.
• Scope by repository and environment. A workflow building a library should not be able to touch production. Environment protection rules and per-environment secrets enforce this mechanically.
• Assume logs leak. Masked variables help, but the discipline is simpler: never pass secrets as command-line arguments (visible in process listings and often in logs), never echo them, and never write them to files that outlive the job.
• Separate build secrets from runtime secrets. The pipeline authenticates to deploy; the running application authenticates to the vault. A pipeline that ships the application its production database password has merged two trust domains that should never touch.

Detection: Assume Something Already Leaked

A secrets program without detection is a policy document. Scanning has to run in three places, because leaks do not respect boundaries:

• Pre-commit and server-side pushes to stop credentials entering history at all. GitHub push protection and local hooks like Gitleaks catch the common patterns — API key prefixes, PEM headers, connection strings — before a remote ever stores them.
• Scheduled scans of all repositories, including full git history, since the scanners and the key formats both evolve.
• Runtime and log scanning where feasible: log pipelines that redact high-entropy strings and alerts on secret-shaped data appearing where it should not.

Detection is also how you learn which of your own conventions are failing. If the scanner keeps finding JWTs in test fixtures, that is a workflow problem to fix, not just findings to close.

When a Secret Leaks: The Response Playbook

The moment you confirm a credential is exposed, the clock matters more than the analysis. The sequence that works:

• Revoke first, investigate second. A leaked key with no observed abuse is still an incident; a leaked key with observed abuse plus six hours of forensics is a breach. Rotation windows and dual-version schemes (above) are what make immediate revocation survivable.
• Assume it was used. Pull audit logs from the vault, cloud IAM, and the target system. Look for reads from unfamiliar principals, unusual IP ranges, or usage after the commit date of the leak.
• Remove it from history properly. Deleting the file is not enough. Rewrite history with git-filter-repo, coordinate force-pushes with the team, and invalidate every clone and fork that matters. Treat history rewriting as containment — the revocation in step one is the real fix.
• Write down how it got there. Every leaked credential entered the codebase through a specific workflow that felt reasonable to the person who used it. Fix the workflow, or the next leak is a matter of time.

What Good Looks Like

A team with mature secrets management has a short, checkable list of properties: no human knows any production credential; every workload authenticates as itself with short-lived credentials; the vault is the single source of truth with per-service audit trails; CI holds no long-lived secrets; scanners run at commit time and on a schedule; and rotation of whatever static secrets remain is automated with overlap windows. Each item on that list is independently achievable in a week or two, and none of them requires exotic technology — they require deciding that secrets hygiene is an architecture problem, not a checklist item.

The best starting point is small: pick one service, give it its own identity, move its credentials into the vault with per-service paths, and wire up rotation. The pattern you build for that one service becomes the template for everything else — and the first time an incident response consists of “revoked the key, audit log shows no other use, done,” the investment pays for itself.