Secrets Management for Developers: Environment Variables, Vaults, KMS, and Rotation Workflows

Overview

If you build or operate cloud-native applications, you are already doing secrets management whether you planned for it or not. Database passwords, API keys, signing keys, webhook secrets, TLS material, OAuth client secrets, and service credentials all have to live somewhere. The real question is not whether you store secrets, but whether your storage and delivery model matches the risk, scale, and operational maturity of your system.

A useful way to think about secrets management is to separate four concerns:

• Storage: where the secret is kept at rest
• Access: who or what can retrieve it
• Delivery: how the application receives it at runtime
• Rotation: how the secret changes without causing outages

That framing helps clear up a common source of confusion in the environment variables vs vault discussion. Environment variables are usually a delivery mechanism, not a complete secrets strategy. A vault is usually a secrets system that handles secure storage, access control, auditing, and sometimes dynamic credentials. A KMS is generally a key management and cryptographic control layer used to encrypt or decrypt data, protect master keys, or integrate with other secret stores. These tools often work together rather than competing directly.

For many teams, the progression looks like this:

1. Start with environment variables for local development and simple deployments
2. Move sensitive values out of source control and into a managed secret store
3. Use IAM or workload identity to control which services can fetch secrets
4. Add rotation workflows, auditability, and staged rollout procedures

The goal is not to create maximum complexity. It is to make the safe path the default path for developers and operators. If secret handling is awkward, people will copy values into chat, stash them in shell history, or hardcode them in test fixtures. Good secrets management reduces both security risk and delivery friction.

Checklist by scenario

Use the following checklist as a decision aid. The right answer depends on where the secret is used, who needs access, and how often it changes.

1. Local development

For local work, the main goals are developer speed, low accidental exposure, and reasonable parity with production.

• Keep secrets out of the repository, including test fixtures and example config files
• Use a local env file pattern such as .env.local or equivalent, and ensure it is ignored by version control
• Provide a checked-in template like .env.example with placeholder keys only
• Prefer low-privilege, development-only credentials instead of reusing shared staging or production values
• Document how secrets are obtained, refreshed, and revoked when a developer changes teams or leaves
• If possible, use short-lived credentials or local sign-in flows tied to corporate identity instead of long-lived copied secrets
• Scrub secrets from logs, shell history, and debug output

Use environment variables alone when: the team is small, credentials are low-risk and non-production, and there is a clear path to stronger controls later.

Use a vault-backed local workflow when: developers need many secrets, credentials change often, or you want centralized revocation and auditing.

2. CI/CD pipelines

Managing secrets in CI/CD is where many teams first feel real operational pain. Build systems, test runners, deployment jobs, and release automation all need access, but those systems also increase the blast radius of a leak.

• Store pipeline secrets in the CI platform's secure secret mechanism or an external vault, not in pipeline definitions
• Separate build-time, test-time, and deploy-time secrets instead of sharing one broad credential
• Use distinct secrets per environment so a staging leak does not become a production incident
• Prefer workload identity, OIDC federation, or short-lived tokens over static long-lived keys where supported
• Limit which branches, tags, or deployment contexts can access sensitive values
• Mask secret values in logs and verify that failure output does not reveal them indirectly
• Make rotation part of the pipeline design so changing credentials does not require manual edits in multiple jobs

If you are reviewing your pipeline design more broadly, it helps to pair this with a stage-by-stage release review such as CI/CD Pipeline Stages Explained: Build, Test, Security Scan, Deploy, and Rollback.

Use a vault or managed secret store when: multiple pipelines need access, you need audit logs, or secrets must be distributed consistently across jobs and environments.

Use KMS in this layer when: you need to decrypt encrypted configuration at runtime or pipeline execution time, or when your cloud platform's secret service is built on top of KMS-backed encryption policies.

3. Runtime secrets in production

Production is where ad hoc approaches become expensive. You want strong access controls, clear ownership, predictable rotation, and minimal human handling.

• Assign secrets to the application identity, not to individual engineers
• Use least privilege so each workload can fetch only the secrets it truly needs
• Prefer runtime retrieval or controlled injection from a secret store over baking secrets into container images
• Ensure the application can start, reload, or refresh secrets safely according to how updates are delivered
• Design health checks so temporary secret retrieval issues do not create cascading failures
• Audit access paths: application startup, sidecars, init containers, deployment systems, and manual break-glass procedures
• Define a fallback plan if the secret backend is unavailable

Secret delivery interacts directly with deployment behavior. If a secret change forces restarts, rollout strategy matters. This is one reason to think about secret rotation alongside release strategy, as covered in Blue-Green vs Canary vs Rolling Deployments and operational readiness topics like Container Health Checks Explained.

4. Database credentials and service-to-service auth

Some secrets deserve extra treatment because compromise has outsized impact.

• For databases, prefer distinct users per application or service boundary instead of a shared admin account
• Scope permissions narrowly by environment and purpose
• If your platform supports dynamic database credentials, consider them for highly sensitive systems
• Rotate application credentials on a schedule and after role changes, incidents, or infrastructure moves
• For service-to-service communication, prefer identity-based mechanisms and short-lived tokens over static shared secrets when practical
• Protect signing keys and token secrets with stricter controls than ordinary configuration values

If credential changes may coincide with schema or release changes, coordinate them with database and deployment planning rather than rotating in isolation. See Database Migration Checklist for Zero-Downtime Deployments for adjacent rollout concerns.

5. Third-party API keys, webhooks, and integration secrets

External integrations often linger for years, making them a common source of drift.

• Keep keys per integration and per environment, not one account-wide key reused everywhere
• Record owner, purpose, upstream system, and rotation method for each secret
• For webhooks, treat signing secrets as sensitive credentials and plan for dual-secret validation during rotation windows
• Ensure retries, replay logic, and validation code behave correctly during secret rollover
• Revoke unused credentials when integrations are retired or replaced

Webhook signing changes in particular benefit from a structured review process. A good companion read is Webhook Debugging Checklist.

6. Choosing between environment variables, vaults, and KMS

Use this simpler rule of thumb if you need a practical decision:

• Environment variables: good for delivering secrets into an app process, especially in simple or low-risk setups
• Vault or managed secrets service: good for central storage, access policy, auditing, dynamic secrets, and coordinated distribution
• KMS: good for key protection, envelope encryption, decrypt operations, and backing other secret systems

In other words, environment variables are usually the endpoint, a vault is often the operational system, and KMS is frequently the cryptographic foundation.

What to double-check

Before approving a secrets workflow, verify the details that tend to get skipped in architecture diagrams.

• Plaintext exposure points: Is the secret ever visible in logs, crash dumps, metrics labels, support bundles, screenshots, or chat transcripts?
• Repository hygiene: Are there old commits, examples, test data, or documentation pages that still contain real values?
• Secret scope: Does each app, environment, and pipeline use separate credentials, or are you relying on one broad shared secret?
• Access paths: Who can read the secret directly: developers, SREs, deployment bots, support staff, or third-party tooling?
• Auditability: Can you tell who accessed a secret, when, and through which system?
• Rotation readiness: Can the secret be changed without downtime? Does the application support dual values, staged rollout, or reload on change?
• Revocation procedure: If a leak is suspected, do you know exactly what to disable, replace, and redeploy?
• Bootstrap problem: How does the application authenticate to the secret store in the first place, and is that initial trust path acceptable?
• Expiration behavior: What happens when a token or certificate expires unexpectedly?
• Ownership: Is every secret assigned to a team and documented with a purpose and rotation policy?

One practical habit is to maintain a lightweight secret inventory with fields such as name, environment, owner, storage location, access method, rotation interval, downstream dependencies, and emergency contact. It does not need to be elaborate to be useful. The value is that it turns hidden operational knowledge into a repeatable process.

Common mistakes

Many secret incidents do not come from sophisticated attacks. They come from ordinary workflow shortcuts that accumulate over time.

Using environment variables as the entire strategy

Environment variables are convenient, but they do not solve discovery, auditing, revocation, ownership, or rotation by themselves. If your only control is “put it in an env var,” you likely still need a real source of truth behind it.

Reusing one secret across environments

Sharing credentials between development, staging, and production makes incidents harder to contain and harder to reason about. Separate values reduce blast radius and simplify revocation.

Hardcoding secrets during debugging

A quick fix during an outage often turns into a durable leak in source control or deployment config. Build secure debugging paths so engineers do not need unsafe shortcuts.

Ignoring rotation until after a scare

If you have never rotated a secret in a controlled way, you do not have a rotation process yet. You have a hope. Practice low-risk rotations before high-risk ones are forced on you.

Storing too many secrets in CI

When every job has access to every credential, the pipeline becomes a concentration point for risk. Narrow the scope of what each stage can read and prefer short-lived access where possible.

Forgetting application behavior during rollover

Teams often update the secret backend but forget that applications cache credentials, hold long-lived connections, or require restart behavior. Secret rotation is as much an application design problem as an infrastructure problem.

Treating all secrets equally

A webhook signing secret, a read-only sandbox API key, and a production token signing key do not deserve the same controls. Classify secrets by impact so your workflow reflects actual risk.

Related operational discipline matters here too. Team habits around branching, release flow, and deployment automation can either support or undermine secret hygiene. If your process is inconsistent, it is worth reviewing nearby workflow decisions such as Git Branching Strategies Compared.

When to revisit

Secrets management should be reviewed on a schedule and whenever the surrounding workflow changes. Use this as a recurring checklist rather than a one-time design document.

• Before seasonal planning cycles or annual security reviews
• When you add a new cloud account, environment, or deployment platform
• When you introduce a new CI/CD system, runner model, or identity provider
• When applications move from monolith to services, or from VM-based hosting to containers
• When a third-party integration is added, replaced, or deprecated
• After a team restructure that changes secret ownership or access patterns
• After any incident involving leaked credentials, unauthorized access, or failed rotation
• When compliance or internal audit requirements change

To make this practical, end each review with a short action list:

1. Identify the five highest-impact secrets in production
2. Confirm who owns each one
3. Verify where each is stored and how it is delivered
4. Test whether each can be rotated without a service outage
5. Remove one unused or over-privileged secret this cycle
6. Document one improvement to local development or CI/CD handling

If your team wants a simple standard, this is a reasonable baseline: keep secrets out of source control, use separate values per environment, rely on identity-based access where possible, centralize storage for anything important, and rehearse rotation before you need it urgently. That approach is not flashy, but it is durable, and durable is what most enterprise web app teams need.