A failed CI/CD run is easier to fix when you identify its failure domain before changing the pipeline. This guide provides a practical decision tree, stage-by-stage checks, log interpretation tips, safe rerun guidance, and a recurring review schedule for cloud deployments.
Overview
CI/CD pipeline troubleshooting should begin with evidence, not repeated reruns. A pipeline can fail because of application code, dependency resolution, the runner, credentials, permissions, artifacts, infrastructure, or the target environment. Similar symptoms can have very different causes: a deployment timeout may indicate a slow health check, a blocked network path, an unavailable dependency, or an expired credential.
Start by recording the commit, branch, workflow or pipeline run, runner image, target environment, and exact failing step. Preserve the original logs before retrying. If a rerun succeeds, the first failure still matters because intermittent failures often point to resource limits, race conditions, unstable dependencies, or environmental drift.
A failure-domain decision tree
- Did the job start? If no, inspect workflow syntax, branch rules, runner availability, concurrency limits, and required approvals.
- Did the runner initialize? If no, check the runner image, operating-system compatibility, tool installation, network access, and runner capacity.
- Did source checkout fail? Check repository access, ref names, submodules, large files, and the identity used by the job.
- Did dependency installation fail? Compare lockfiles, package-manager versions, registry access, cache behavior, and private package credentials.
- Did tests or static checks fail? Treat this as a code, configuration, test-data, or compatibility issue unless the logs show infrastructure symptoms.
- Did packaging or artifact upload fail? Verify paths, permissions, disk space, artifact size limits, retention settings, and naming conventions.
- Did deployment fail? Separate authentication and authorization errors from manifest, quota, networking, rollout, and application health failures.
- Did deployment finish but verification fail? Check health probes, routing, configuration, migrations, dependency availability, and logs from the deployed revision.
This sequence narrows the search without assuming that every red pipeline is a code defect. It also helps teams assign the incident to the right owner: application, platform, security, or release engineering.
What to track
A useful troubleshooting record contains more than the final error line. Track recurring variables that can reveal patterns across builds and deployments.
- Change context: commit, pull request, branch, release tag, changed files, and whether the failure is new or recurring.
- Execution context: pipeline name, job and step, runner operating system, runner image or toolchain version, region, and execution time.
- Dependency context: lockfile changes, package-manager version, registry URL, cache hit or miss, private package access, and dependency installation duration.
- Security context: identity, credential or token type, expiration state, required scopes, role assignment, and the last known successful use. Do not place secret values in tickets or logs.
- Build outputs: generated files, container image or package identifier, checksum where available, artifact path, retention period, and upload or download status.
- Deployment context: target account or project, cluster or service, configuration revision, migration status, rollout strategy, and health-check result.
- Operational impact: whether production changed, whether rollback was required, affected services, and the person who approved or initiated the action.
Stage-by-stage troubleshooting checklist
| Stage | Check first | Useful evidence |
|---|---|---|
| Trigger and queue | Workflow syntax, event filters, approvals, concurrency, runner availability | Run status, workflow configuration, queue time |
| Checkout | Ref, repository access, submodules, shallow-clone assumptions | Checkout logs, identity, commit SHA |
| Setup | Runtime versions, operating-system dependencies, environment variables | Tool versions, image identifier, setup output |
| Dependencies | Lockfile consistency, registry access, cache key, package credentials | Resolver output, cache result, registry response |
| Build and test | Compilation errors, failing tests, lint rules, test services and fixtures | First meaningful error, test report, exit status |
| Package and artifact | Output paths, disk space, permissions, artifact naming and retention | Directory listing, upload log, artifact metadata |
| Deploy | Identity, role, target, manifest, quota, network access | Provider response, rendered configuration, audit event |
| Verify | Health checks, routes, configuration, migrations, dependency reachability | Probe results, service logs, revision status |
Common exit codes and log symptoms
| Symptom or status | Likely failure domain | Next diagnostic action |
|---|---|---|
| Exit code 1 with a compiler, test, or lint message | Application code or quality gate | Fix the first actionable error and reproduce locally with matching tool versions. |
| Exit code 127 or “command not found” | Runner setup or script path | Print the runtime path and version; verify installation and shell assumptions. |
| Exit code 126 or “permission denied” | File mode, execution policy, or identity | Inspect file permissions, working directory, and the user running the step. |
| Timeout, connection reset, or DNS failure | Network, service availability, or resource pressure | Check endpoint resolution, firewall rules, service status, and timeout boundaries. |
| 401 or 403 response | Authentication or authorization | Confirm the selected identity, token validity, scopes, role bindings, and target account. |
| Artifact not found | Path, job dependency, naming, or retention | Compare the producer output with the consumer input and verify artifact lifetime. |
| Rollout stuck or health check failed | Deployment environment or application configuration | Inspect the new revision, probe path, startup logs, configuration, and dependency access. |
Exit codes are clues, not diagnoses. A numeric status should always be read with the surrounding log context and the first failure in the job.
Cadence and checkpoints
Pipeline reliability changes over time as runners, package registries, cloud APIs, dependencies, and application configuration change. Use a small operational review rather than waiting for a high-impact failure.
Per-run checkpoint
For every failed run, capture the failing stage, first meaningful error, commit, environment, runner image, and whether the failure reproduced. Mark whether the run changed any cloud resources before deciding to retry.
Monthly checkpoint
- Review the most frequent failed jobs and recurring error signatures.
- Check credentials, certificates, service connections, and secret rotation dates without exposing secret values.
- Review cache hit rates, build duration, artifact retention, and storage growth.
- Remove obsolete workflow steps, stale runners, unused variables, and undocumented manual approvals.
- Confirm that failure notifications reach the team responsible for remediation.
Quarterly checkpoint
- Test the pipeline against supported runtime and dependency versions.
- Review runner images and pinned actions, plugins, containers, and build tools.
- Validate least-privilege roles and deployment identities.
- Run a controlled rollback or recovery exercise where the risk is acceptable.
- Compare deployment frequency, failure categories, recovery time, and repeat incidents.
When a platform changes its default runtime, authentication flow, artifact behavior, or runner image, review the affected workflows immediately rather than waiting for the next scheduled checkpoint.
How to interpret changes
Look for correlations, not isolated red builds. A sudden increase in dependency failures after a lockfile or registry change points toward resolution or access. Failures beginning after a runner-image update suggest toolchain, operating-system, or preinstalled dependency drift. Deployment errors that affect only one environment usually indicate configuration, permissions, quotas, or network differences rather than application logic alone.
Separate deterministic failures from intermittent ones. A deterministic failure reproduces on the same commit and should be fixed before rerunning broadly. An intermittent failure deserves investigation into concurrency, external services, cache corruption, rate limits, resource exhaustion, and race conditions. Retrying can restore delivery, but it should not erase the record or become the permanent fix.
Use a controlled comparison when possible: run the same commit with the previous runner image, bypass the cache once, compare the target environment with a known-good environment, or execute the failing command with the same environment variables and tool versions. Change one variable at a time so the result remains interpretable.
Incident template
Title: [Pipeline] [service] - [failure summary]
First observed: [date and time]
Pipeline/run: [link or identifier]
Commit/release: [SHA or tag]
Target environment: [environment]
Failing stage and step: [stage / step]
First meaningful error: [sanitized log excerpt]
Failure domain: [code | dependency | runner | credential | permission | artifact | deployment | environment]
Impact: [what changed or was blocked]
Reproduction: [consistent | intermittent | unknown]
Immediate action: [fix, rollback, hold, or rerun]
Root cause: [confirmed cause or current hypothesis]
Prevention: [test, monitoring, permission, documentation, or automation change]
Safe rerun template
Rerun reason: [transient infrastructure issue | corrected configuration | verified code fix]
Original run: [identifier]
Commit unchanged: [yes/no]
Resources changed by original run: [yes/no/unknown]
Secrets or permissions changed: [yes/no]
Artifacts reused: [identifier and retention confirmed]
Environment verified: [yes/no]
Approval recorded by: [name or team]
Rollback plan: [brief action]
Post-rerun check: [tests, deployment status, health checks, logs]
Before rerunning a deployment, establish whether the original job reached the target system. A retry may create a second release, repeat a migration, or overwrite an artifact. Prefer immutable build outputs and explicit deployment identifiers so a rerun is deliberate rather than ambiguous.
When to revisit
Revisit this troubleshooting guide after any recurring failure, runner-image update, runtime upgrade, package-manager change, credential expiration, permission redesign, artifact-retention change, or migration of the pipeline to another CI platform. It is also worth reviewing after a rollback, an incident involving a failed health check, or a deployment that succeeded only after repeated retries.
At each monthly or quarterly review, update the exit-code examples, authentication instructions, runner details, package-manager behavior, artifact settings, and platform syntax used by your organization. Keep examples aligned with the versions actually supported by your repositories; avoid copying commands from old workflows without checking their security and compatibility implications.
For the next failure, use this short action plan:
- Freeze the evidence: save logs, identifiers, commit details, and environment information.
- Locate the first failing stage and classify the failure domain.
- Test one focused hypothesis using a controlled comparison.
- Check whether the failed run changed cloud resources before rerunning.
- Record the confirmed cause, prevention step, and review date.
For related operational work, see the enterprise secrets management runbook, the OpenAPI contract testing workflow, and the cloud deployment strategy comparison. These practices help reduce the number of pipeline failures that reach the deployment stage and make the remaining failures easier to diagnose.