Most security incidents in distributed systems don’t start with a zero-day exploit. They start with a leaked credential — a database password sitting in an environment variable, an API key committed to a config file, or a service account token that was supposed to be temporary but somehow became permanent. The credential was valid, it was exposed, and someone used it.
The standard response is to rotate credentials periodically and hope nothing leaks between rotations. But rotation is a band-aid. The real shift happening in modern infrastructure is toward dynamic secrets — credentials that are generated on demand, scoped to the requesting workload, and automatically invalidated shortly after use. Instead of protecting a long-lived secret, you eliminate the long-lived secret entirely.
Let’s walk through how dynamic secrets work, why they represent a fundamental improvement over static credential rotation, and how to implement them in practice using HashiCorp Vault and workload identity federation.
The Problem with Static Secrets
A static secret is a credential that exists in a fixed form until someone manually changes it: a database password, an API key, an SSH private key. The security of the entire system depends on that credential remaining confidential for its entire lifetime, which could be weeks, months, or indefinitely.
This creates several structural problems. First, every system that needs the credential is a potential leak vector — config files, CI/CD pipelines, environment variables, secrets managers, chat logs. Second, rotation requires coordination across every consumer, which means teams delay it or skip it entirely. Third, when a static secret is compromised, the attacker has the same access as the legitimate workload until someone notices and rotates the credential — a window that often stretches into weeks.
The most damaging pattern is the shared database credential: one password configured across every microservice that touches the database. When one service is compromised, the attacker has database access as far as that credential reaches, with no way to distinguish the attacker’s queries from legitimate traffic.
How Dynamic Secrets Work
Dynamic secrets flip the model. Instead of storing a fixed credential and distributing it, the secrets manager generates a unique credential each time a workload requests access. The credential is scoped to that specific workload, has a short TTL (typically 1–4 hours), and is automatically revoked when it expires.
Here’s what this looks like with Vault’s database secrets engine:
# Enable the database secrets engine
vault secrets enable database
# Configure the database connection
vault write database/config/my-postgresql \
plugin_name=postgresql-database-plugin \
allowed_roles="readonly,readwrite" \
connection_url="postgresql://{{username}}:{{password}}@db.internal:5432/myapp" \
username="vault-admin" \
password="super-secret-root-password"
# Create a role that generates read-only credentials (1-hour TTL)
vault write database/roles/readonly \
db_name=my-postgresql \
creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; \
GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \
default_ttl="1h" \
max_ttl="4h"
# Generate credentials on demand
vault read database/creds/readonly
# Key Value
# lease_id database/creds/readonly/abc123...
# lease_duration 3600
# username v-token-readonly-xyz789
# password random-32-char-stringEach call to vault read database/creds/readonly creates a fresh database user with random credentials and a guaranteed expiration. If the lease isn’t renewed within an hour, Vault revokes the user from the database automatically. No manual cleanup, no stale credentials left behind.
Workload Identity: Beyond Secrets Entirely
Dynamic secrets still require a workload to authenticate to Vault first. The next evolution eliminates even that step through workload identity federation. Instead of a workload holding a credential to get credentials, it proves its identity directly to the cloud provider using a trusted attestation.
In Kubernetes, this means using Service Account tokens (JWT-based since 1.24) that cloud providers can verify directly. AWS IRSA (IAM Roles for Service Accounts), Azure managed identities, and GCP Workload Identity all follow the same pattern:
# Kubernetes ServiceAccount annotated with AWS IAM role
apiVersion: v1
kind: ServiceAccount
metadata:
name: payment-service
annotations:
eks.amazonaws.com/role-arn: arn:aws:iam::123456789012:role/payment-service-role
---
# The pod uses this ServiceAccount
spec:
serviceAccountName: payment-serviceThe pod gets short-lived AWS credentials injected automatically — no static keys stored anywhere, no Vault dependency, no manual rotation. The trust relationship is established at the cloud platform level through OIDC federation.
Implementing Dynamic Secrets in Go
For services that still need a secrets manager, here’s a practical Go implementation using Vault’s API client that fetches dynamic database credentials and handles lease renewal:
package main
import (
"context"
"fmt"
"log"
"sync"
"time"
vault "github.com/hashicorp/vault/api"
vaultk8s "github.com/hashicorp/vault/api/auth/kubernetes"
)
// SecretManager wraps the Vault client to provide dynamic secrets
// with automatic lease renewal and revocation.
type SecretManager struct {
client *vault.Client
mu sync.Mutex
leases map[string]*vault.Secret
renewCtx context.Context
cancel context.CancelFunc
}
func NewSecretManager(addr string) (*SecretManager, error) {
config := vault.DefaultConfig()
config.Address = addr
client, err := vault.NewClient(config)
if err != nil {
return nil, fmt.Errorf("vault client: %w", err)
}
// Authenticate via Kubernetes service account
k8sAuth, err := vaultk8s.NewKubernetesAuth("payment-service")
if err != nil {
return nil, fmt.Errorf("k8s auth setup: %w", err)
}
if err := client.Auth().Login(context.Background(), k8sAuth); err != nil {
return nil, fmt.Errorf("k8s login: %w", err)
}
ctx, cancel := context.WithCancel(context.Background())
sm := &SecretManager{
client: client,
leases: make(map[string]*vault.Secret),
renewCtx: ctx,
cancel: cancel,
}
// Start the lease renewal loop
go sm.renewLeases()
return sm, nil
}
// GetDatabaseCreds fetches dynamic credentials from Vault.
// Each call returns a fresh credential with a short TTL.
func (sm *SecretManager) GetDatabaseCreds(role string) (string, string, error) {
secret, err := sm.client.Logical().Read("database/creds/" + role)
if err != nil {
return "", "", fmt.Errorf("read dynamic secret: %w", err)
}
if secret == nil {
return "", "", fmt.Errorf("no secret returned for role %s", role)
}
sm.mu.Lock()
sm.leases[secret.LeaseID] = secret
sm.mu.Unlock()
return secret.Data["username"].(string), secret.Data["password"].(string), nil
}
// renewLeases periodically renews all active leases before they expire.
// If renewal fails (e.g., Vault is unreachable), the secret will expire
// and the calling service should re-fetch.
func (sm *SecretManager) renewLeases() {
ticker := time.NewTicker(5 * time.Minute)
defer ticker.Stop()
for {
select {
case <-sm.renewCtx.Done():
return
case <-ticker.C:
sm.mu.Lock()
for id, secret := range sm.leases {
// Renew if more than half the TTL has elapsed
if secret == nil {
delete(sm.leases, id)
continue
}
_, err := sm.client.Sys().Renew(id, 0)
if err != nil {
log.Printf("lease renewal failed for %s: %v", id, err)
delete(sm.leases, id)
}
}
sm.mu.Unlock()
}
}
}
// RevokeAll cleans up all leases on shutdown. This immediately
// revokes the dynamic credentials from the target system.
func (sm *SecretManager) RevokeAll() {
sm.cancel()
sm.mu.Lock()
defer sm.mu.Unlock()
for id := range sm.leases {
if err := sm.client.Sys().Revoke(id); err != nil {
log.Printf("failed to revoke lease %s: %v", id, err)
}
delete(sm.leases, id)
}
}The CI/CD Pattern: OIDC Federation
CI/CD pipelines are another common place where static credentials accumulate. The old pattern stores AWS access keys as GitHub Actions secrets. The modern pattern uses OIDC federation — GitHub Actions generates a short-lived OIDC token, AWS verifies it, and issues temporary credentials for just that workflow run.
# .github/workflows/deploy.yml
name: Deploy
permissions:
id-token: write # Required for OIDC
contents: read
jobs:
deploy:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
# Request temporary AWS credentials via OIDC
- uses: aws-actions/configure-aws-credentials@v4
with:
role-to-assume: arn:aws:iam::123456789012:role/github-actions-deploy
aws-region: us-east-1
# No access keys stored anywhereNo long-lived AWS keys stored in GitHub secrets. Each workflow run gets a fresh set of temporary credentials scoped to the IAM role. The OIDC trust policy in AWS ensures only workflows from the specified repository and branch can assume the role.
Practical Migration Strategy
Moving from static to dynamic secrets is not an all-or-nothing migration. A pragmatic approach:
Phase 1: Centralize
Move all static secrets into a central vault. This doesn’t make them dynamic yet, but it gives you visibility, access logging, and a single point for rotation. Audit which secrets exist and who accesses them.
Phase 2: Automate Rotation
For secrets that can’t yet be made dynamic, automate rotation through the vault. The old and new credentials coexist briefly during rotation, with zero downtime. Rotation cadence depends on the secret type — database passwords every 30–90 days, API keys quarterly.
Phase 3: Go Dynamic
Replace static database credentials with dynamic secrets engines. Start with read-only database roles (lowest risk), then move to read-write roles as your team gains confidence. Each microservice gets its own role with minimal permissions, and every credential expires automatically.
Phase 4: Eliminate Secrets
Where the cloud platform supports it, replace secrets entirely with workload identity federation. Kubernetes pods authenticate via service account tokens. CI/CD pipelines use OIDC. The credential surface area shrinks to zero.
Common Pitfalls
Vault as a single point of failure. If every service depends on Vault for credentials, Vault going down means no new credentials can be issued. Mitigate with HA deployment (integrated storage with 3+ nodes), credential caching on the client with buffer time, and graceful degradation where services hold valid credentials long enough to survive a brief Vault outage.
Over-permissive dynamic roles. It’s tempting to grant broad permissions to dynamic database users to avoid access denied errors. But the whole point is least privilege. Create multiple roles with different permission scopes (read-only, app-specific schema, admin) and use the minimum that works.
Forgetting lease revocation on shutdown. If a service crashes without revoking its dynamic credentials, the database users persist until the TTL expires. This isn’t catastrophic (they’ll auto-expire), but it accumulates orphaned users. Implement graceful shutdown that revokes leases before exit.
Wrapping Up
The shift from static to dynamic secrets is one of the highest-leverage security improvements you can make in a distributed system. It reduces the credential attack surface from “every environment that ever held the password” to “ephemeral credentials that expire before an attacker can exploit them.” Combined with workload identity federation, you can reach a state where long-lived secrets simply don’t exist in your infrastructure.
Start with your highest-risk credentials — shared database passwords, CI/CD cloud keys — and work outward. Each secret you make dynamic is one less thing that can leak, one less rotation to coordinate, and one less entry in your incident response runbook.