Managing Kubernetes configuration well means separating ordinary application settings from sensitive values, making environment differences explicit, and promoting changes through reviewable artifacts. This checklist explains when to use ConfigMaps, Secrets, sealed secrets, external secret operators, Helm, and Kustomize, with practical templates for validation, promotion, and rollback.
Overview
Kubernetes configuration management has two related problems: storing values safely and moving the right values to the right environment. A development, staging, or production namespace may run the same application image, but it often needs different endpoints, feature flags, resource settings, credentials, and integration identifiers.
Keep the application image as environment-neutral as practical. Put deploy-time differences in versioned configuration and keep secret material in an approved secret-management path. This makes a release easier to inspect: the image, configuration references, and environment-specific overlays can each be reviewed instead of being hidden inside a manually edited cluster.
ConfigMaps are intended for non-sensitive configuration such as log levels, public service URLs, and feature switches. Kubernetes Secrets are designed for sensitive values, but storing a value in a Secret object does not by itself provide a complete secret-management process. Access control, encryption at rest, auditability, rotation, backup handling, and developer access still need to be addressed. Values in Secret manifests should also be treated carefully because common encoding formats are not the same as encryption.
| Option | Best fit | Review carefully |
|---|---|---|
| ConfigMap | Non-sensitive settings and small configuration files | Do not place credentials, tokens, private keys, or regulated data here |
| Secret | Sensitive values managed through Kubernetes-compatible resources | RBAC, encryption, audit logs, rotation, backups, and exposure through logs or manifests |
| Sealed secret | Encrypted secret resources committed to a repository for controlled delivery | Key lifecycle, controller ownership, recovery procedures, and repository access |
| External Secrets Operator | Synchronizing values from an external secret manager into Kubernetes | Provider authentication, refresh behavior, failure handling, and namespace permissions |
| Environment-specific manifests | Declaring differences between development, staging, and production | Duplication, unreviewed overrides, and drift between overlays |
Helm is useful when a chart exposes deliberate values and templates. Kustomize is useful when a common base needs overlays without turning every environment into a separate application definition. Either approach can work; the important control is that the rendered output is validated and reviewed before it reaches a cluster.
Checklist by scenario
When onboarding a service
- List every configurable value and classify it as non-sensitive, sensitive, or derived at runtime.
- Define safe defaults for local development without placing production credentials in the repository.
- Document which values are required for startup and which can be absent or changed later.
- Choose one ownership model for each secret: an external secret manager, a sealed-secret workflow, or a controlled Kubernetes Secret process.
- Define the namespace, service account, and RBAC permissions needed to read configuration.
- Render the deployment locally or in CI and inspect the final references, not only the source templates.
When adding development, staging, or production
Start with a shared base and keep differences small and intentional. A Kustomize layout might look like this:
apps/orders/base/
deployment.yaml
service.yaml
kustomization.yaml
apps/orders/overlays/
dev/kustomization.yaml
staging/kustomization.yaml
production/kustomization.yamlEach overlay should declare only what changes. For example, an overlay can reference an environment-specific ConfigMap, adjust replicas, or select a different external secret path. Avoid copying the complete Deployment into every environment; copied manifests make security and operational changes easy to miss.
With Helm, keep a stable set of chart values and separate environment files:
# values.yaml
config:
logLevel: info
publicApiUrl: ""
secretRefs:
- name: orders-runtime
autoscaling:
enabled: false# values-production.yaml
config:
logLevel: warn
publicApiUrl: https://api.example.invalid
autoscaling:
enabled: trueThe example shows structure, not a recommendation to put secret values in a values file. Prefer a secret reference or an external-secret resource. Add a CI check that rejects production manifests containing known secret patterns, private keys, or values supplied through an unapproved path.
When promoting a change
- Change the shared configuration or environment overlay in version control.
- Render the target environment using the same chart or overlay inputs used by deployment automation.
- Run schema, policy, YAML, and application-level validation.
- Review the diff for image changes, configuration changes, secret references, permissions, and resource settings.
- Apply to development, run focused tests, and record the resulting commit or artifact identifier.
- Promote the same reviewed artifact to staging, then production, changing only approved environment inputs.
- Record rollout health, configuration version, and any follow-up action.
This is especially useful when investigating a mismatch between environments. For deployment failure diagnosis, pair the configuration diff with the troubleshooting steps in CI/CD Pipeline Failure Troubleshooting.
When rotating a secret
- Confirm whether the consuming application supports a live refresh or requires a restart.
- Create the replacement value in the approved secret manager.
- Update the reference or synchronized object without exposing the value in CI logs.
- Deploy or restart according to the application’s documented behavior.
- Verify authentication, error rates, and dependent integrations.
- Revoke the old value only after the new path is confirmed, unless incident response requires immediate revocation.
- Update the rotation record and remove obsolete references from manifests and documentation.
What to double-check
Configuration and validation rules
Every required setting should have a clear failure mode. An application that starts with an empty database URL may fail later and less clearly than one that refuses to start. Define validation close to the application and reinforce it in deployment checks.
Required configuration:
- DATABASE_URL: non-empty URI; never printed in logs
- PUBLIC_API_URL: valid HTTPS URL for shared environments
- LOG_LEVEL: one of debug, info, warn, error
- ORDERS_TIMEOUT_MS: integer between 100 and 30000
- FEATURE_CHECKOUT_V2: boolean
Deployment checks:
- secret references exist in the target namespace
- referenced keys are present
- image tag is immutable or traceable to a build
- no production overlay points to a development endpoint
- rendered manifests pass policy and schema validationUse the policy engine and syntax already supported by your platform team. The exact rule language may differ, but the intent should be consistent: block privileged changes, unapproved registries, missing resource requests, plaintext sensitive values, and production references to non-production dependencies.
Secret access and observability
- Check which service accounts can read each Secret and whether access is narrower than namespace-wide access.
- Confirm that application logs, tracing attributes, crash reports, and command output cannot expose sensitive values.
- Check whether configuration changes trigger a rollout. A mounted or injected value may not be consumed until the process reloads it.
- Confirm that backups and exported manifests receive the same protection as the live cluster.
- Test the external secret provider’s unavailable or delayed state so the service has a deliberate failure behavior.
For broader rotation ownership and integration patterns, see the enterprise secrets management rotation runbook.
Promotion and rollback
A rollback plan should identify more than the previous application image. Configuration can be the cause of a failed release, so record the previous chart values, overlay commit, Secret or external-provider version, and any database or API compatibility requirement. If a secret was rotated, reverting the deployment may not restore compatibility; the rollback note must state whether the old credential remains valid and for how long according to the team’s operational policy.
Common mistakes
- Confusing encoding with protection. A base64-encoded value in a manifest should not be treated as confidential merely because it is not readable at a glance.
- Using one giant environment file. Large files conceal ownership and make a small change difficult to review. Separate application settings, secret references, and resource policies.
- Duplicating complete manifests. Separate copies drift. Prefer a shared base with focused Helm values or Kustomize overlays.
- Allowing a chart to accept arbitrary secrets. Define explicit secret references and reject unexpected keys where practical.
- Testing only template syntax. Valid YAML can still contain a wrong namespace, missing key, invalid URL, or unsafe permission.
- Changing configuration manually in production. An emergency change may be necessary, but it should be captured in version control or an incident record and reconciled afterward.
- Ignoring application reload behavior. Updating a resource does not guarantee that a running process has adopted the new value.
- Promoting values instead of intent. Production should receive the reviewed release configuration, not an unexamined copy of staging’s live state.
Configuration also intersects with API compatibility. When an endpoint, credential scope, or integration contract changes, coordinate the configuration change with the API’s rollout and versioning plan. The guide to API versioning strategies can help frame that review.
When to revisit
Revisit this checklist before seasonal planning cycles, when a new environment or service is introduced, after a Kubernetes or deployment-tool upgrade, and whenever the organization changes its secret provider, policy engine, or CI/CD workflow. Also review it after a deployment mismatch, credential incident, unexpected configuration exposure, or failed rollback.
A practical six-month review should confirm that resource APIs, Helm and Kustomize conventions, external-secret integrations, and policy rules still match the versions and capabilities your platform supports. Do not assume an example remains correct after a major ecosystem release; render and validate it against a representative cluster or test environment.
For the next production change, use this short action list: classify each value, select the storage mechanism, render the target environment, run validation and policy checks, review the diff, test the rollout, record the configuration version, and document rollback conditions. That sequence turns Kubernetes configuration from an informal collection of environment variables into a reviewable part of the cloud application delivery process.