Debugging Pipelines
CI Lint
Debug Trace
Variables
Root Cause
Learn how to debug GitLab CI/CD pipelines systematically. From CI Lint and pipeline editor validation to debug tracing, variable verification, and dependency checks, find and fix pipeline failures faster.
Debugging
Troubleshooting
GitLab CI/CD
Why Systematic Debugging Matters
Pipeline failures are inevitable. A missing variable, a typo in a script, a dependency that changed version — any of these can break a pipeline that worked yesterday. The difference between a team that recovers quickly and one that struggles is not the absence of failures; it's the presence of a systematic approach to debugging. Random trial and error wastes hours; a structured methodology finds the root cause in minutes.
GitLab provides several tools for this. The CI Lint tool validates syntax before you commit. The pipeline editor simulates pipeline creation, catching logic errors in rules and needs. CI_DEBUG_TRACE enables verbose logging when a job fails mysteriously. Variable verification confirms that the values your job sees are the values you expect. Each tool addresses a different class of problem, and knowing when to use each is the key [citation:1][citation:7].
The most common debugging mistake is jumping to conclusions. A job fails with "command not found," so you assume the image is wrong. But the real cause might be a missing before_script, a variable that didn't expand, or a dependency that failed to install. The systematic approach is to isolate: check the configuration, then the environment, then the script, then the dependency. Narrowing the problem space is faster than guessing.
Core Principle: Debugging is a process of elimination. Start with the tools that catch the most common problems (CI Lint), then move to runtime diagnostics (debug trace, variable checks), then to deeper investigation (dependencies, environment).
CI Lint
Validates syntax and simulates pipeline creation. Catches YAML errors, invalid keywords, and logic issues before commit.
Pipeline Editor
Live syntax validation, visualization, and full configuration view. The primary editing experience.
CI_DEBUG_TRACE
Enables verbose logging. Shows commands, variables, and environment. Use sparingly.
Variable Verification
Check which variables exist and their values. Many failures are variable mismatches.
Dependency Checks
Verify tool versions, pin critical dependencies, and check that installs succeeded.
Root Cause Analysis
GitLab Duo can analyze failed job logs and suggest fixes (Ultimate).
CI Lint and Pipeline Editor
The CI Lint tool is the first line of defense against pipeline failures. It validates the syntax of your .gitlab-ci.yml file against the GitLab CI/CD schema, catching typos, invalid keywords, and structural errors before you commit. When the project has a .gitlab-ci.yml file, CI Lint can also simulate pipeline creation, which validates not just syntax but also logic — checking that rules, needs, and include statements produce the expected pipeline [citation:1][citation:7].
The pipeline editor is the primary place to edit CI/CD configuration. It provides live syntax validation as you type, a visualization of the pipeline graph, and a "Full configuration" view that shows the merged result of all include statements. This is invaluable for debugging: if a job is missing or has unexpected configuration, the full configuration view shows exactly what GitLab sees, including any included templates [citation:5].
The "Validate" tab in the pipeline editor simulates a pipeline creation for a specific branch and pipeline source. This catches issues that syntax validation misses: a rule that never matches, a needs dependency on a job that doesn't exist, or a workflow rule that blocks the pipeline entirely. If the simulated pipeline is empty or missing expected jobs, the problem is in the configuration logic, not in the job execution [citation:5].
# CI Lint usage
# Build > CI Lint
# Paste your .gitlab-ci.yml content
# Click "Validate"
# Or use "Simulate pipeline creation" for deeper validation
# Pipeline editor
# Build > Pipeline editor
# - Edit tab: live syntax validation
# - Visualize tab: pipeline graph
# - Full configuration tab: merged config with includes
# - Validate tab: simulate pipeline creation
# Common syntax errors caught by CI Lint:
# - Invalid keyword (e.g., "stages" misspelled)
# - Wrong indentation
# - Missing required fields (script for jobs)
# - Invalid variable syntax
# - Duplicate job names
# Common logic errors caught by simulation:
# - rules that never match
# - needs on non-existent jobs
# - workflow:rules blocking all pipelines
# - include paths that don't exist
# Example: validate before commit
# 1. Open Pipeline editor
# 2. Make changes in Edit tab
# 3. Check syntax status at top (valid/invalid)
# 4. Go to Validate tab
# 5. Select branch and source
# 6. Click "Validate pipeline"
# 7. Review the simulated pipeline
# If validation fails:
# - Read the error message carefully
# - Check the line number (if provided)
# - Compare with GitLab CI/CD reference
# - Use Full configuration to see merged result
CI Lint Best Practices:
- Validate syntax before every commit — catches the most common errors
- Use the simulation feature to test logic, not just syntax
- Check the Full configuration view when includes are involved
- If a pipeline runs but jobs are missing, use simulation to debug rules
CI_DEBUG_TRACE
When a job fails and the log doesn't explain why, CI_DEBUG_TRACE enables verbose logging. Setting this variable to true makes the runner print every command it executes, along with the environment variables available to the job. This is the fastest way to see exactly what the runner is doing and what values it's using [citation:4].
The variable is powerful but dangerous. The debug trace includes all environment variables, which may include secrets. If a sensitive variable is not masked, it will appear in the log. For this reason, access to debug logging is restricted to users with Developer role or higher. It's also why you should never set CI_DEBUG_TRACE as a runner-level variable — the runner doesn't check permissions, so anyone with access to job logs could see the trace [citation:4].
Use CI_DEBUG_TRACE as a temporary diagnostic, not a permanent setting. Enable it on a specific job (not globally), run the pipeline, inspect the log, then disable it. If you need ongoing verbosity for a specific tool, add --verbose to that tool's command in the script instead of enabling full debug tracing. This limits the noise and the security exposure.
# Enable debug trace for a specific job
debug_job:
variables:
CI_DEBUG_TRACE: "true"
script:
- echo "This will produce verbose output"
# Or enable for the entire pipeline (temporarily)
# Settings > CI/CD > Variables
# Add CI_DEBUG_TRACE = true
# Disable after debugging
# Warning: debug trace shows all variables
# Ensure sensitive variables are masked
# Settings > CI/CD > Variables > Masked: ✓
# CI_DEBUG_TRACE output includes:
# - Every command executed
# - Environment variables
# - Shell expansions
# - Exit codes
# Do NOT set CI_DEBUG_TRACE on the runner
# Runner-level debug trace bypasses permission checks
# and is visible to all users with job log access
# Better: use --verbose for specific tools
script:
- npm test -- --verbose
- docker build --progress=plain .
# Or use a debug variable with default empty
variables:
DEBUG_VARS: ""
script:
- my-test-command $DEBUG_VARS /test-dirs
# Set DEBUG_VARS=--verbose when running manually
CI_DEBUG_TRACE Security:
- Exposes variables: All environment variables are printed, including secrets
- Access restricted: Only Developer+ can see debug logs (when set in GitLab, not runner)
- Never set on runner: Runner-level debug bypasses permission checks
- Use temporarily: Enable for one job, debug, then disable
Variable Verification
Many pipeline failures are variable mismatches. A job expects $DATABASE_URL but the variable is named $DATABASE_HOST. A variable is defined at the project level but the job runs on a protected branch where it's not available. A variable is masked, and the masking causes unexpected behavior. The first debugging step for any "command failed" or "connection refused" error is to verify the variables [citation:1][citation:7].
The simplest verification is to print the variables in a job. env | sort lists all environment variables. For a specific variable, echo "$VARIABLE_NAME" shows its value (unless masked, in which case it shows [MASKED]). This is quick and reveals whether the variable exists and has the expected value. If the variable is masked, you can't see its value, but you can check that it's not empty [citation:1].
Variable precedence is a common source of confusion. Pipeline variables (set via UI or API when triggering) override project variables, which override group variables, which override instance variables. Job-level variables override all of these. If a variable has an unexpected value, check the precedence order and look for an override at a higher level. The "Variables" page in project settings shows where each variable is defined.
# Verify variables in a job
debug_variables:
script:
- echo "CI_COMMIT_BRANCH=$CI_COMMIT_BRANCH"
- echo "CI_PIPELINE_SOURCE=$CI_PIPELINE_SOURCE"
- echo "DATABASE_URL=$DATABASE_URL"
- echo "API_KEY=$API_KEY" # Shows [MASKED] if masked
# List all variables (names only, not values)
- env | grep -E '^[A-Z_]+=' | cut -d= -f1 | sort
# Check variable existence (without exposing value)
check_variable:
script:
- |
if [ -z "$DATABASE_URL" ]; then
echo "ERROR: DATABASE_URL is not set"
exit 1
else
echo "DATABASE_URL is set (length: ${#DATABASE_URL})"
fi
# Variable precedence (highest to lowest):
# 1. Pipeline variables (UI/API trigger)
# 2. Project variables (UI)
# 3. Group variables (UI)
# 4. Instance variables (UI)
# 5. .gitlab-ci.yml job-level variables
# 6. .gitlab-ci.yml global variables
# 7. Predefined variables
# Debug: check if variable is protected
# Settings > CI/CD > Variables
# Look for the Protected and Masked columns
# Common variable issues:
# - Variable defined at project level, job runs on unprotected branch
# - Variable masked, but value too short (<8 chars) or contains newlines
# - Variable name typo ($DATABSE_URL instead of $DATABASE_URL)
# - Variable scope mismatch (staging vs production)
# - Variable overridden by pipeline trigger
Variable Debugging Checklist:
- Does the variable exist? (
echo $VAR is not empty)
- Is the value correct? (compare with expected value)
- Is it available in this context? (protected, environment scope)
- Is it overridden? (check precedence order)
- Is it masked? (affects visibility, not value)
Dependency and Version Checks
Dependency issues are a common cause of pipeline failures that seem mysterious. A tool that worked yesterday fails today because a new version was released with a breaking change. A library that installed successfully now fails because a transitive dependency changed. A base image that was stable now has a different default shell or missing package. The fix is to verify versions and pin critical dependencies [citation:1][citation:8].
The first step is to print the versions of the tools your job uses. node --version, python --version, docker --version, terraform --version — these take seconds and immediately reveal whether the environment is what you expect. If a job fails with a syntax error that the tool supports, the tool version is probably the issue. Pinning the version in the image tag (node:18.19.0 instead of node:18 or node:latest) prevents unexpected updates [citation:1].
For application dependencies, check that the install step succeeded. If npm ci or pip install fails, the job fails before the script runs. The error message usually explains why: a missing package, a version conflict, or a network issue. If the install succeeds but the script fails, the dependency is present but the wrong version — or the script is using a different dependency than expected. Lockfiles (package-lock.json, requirements.txt) pin versions, but only if the install command respects them (npm ci, not npm install).
# Verify tool versions before running script
job:
before_script:
- node --version
- npm --version
- docker --version
script:
- npm test
# Pin critical dependencies (avoid :latest)
variables:
ALPINE_VERSION: '3.18.6'
job1:
image: alpine:$ALPINE_VERSION # Never changes unexpectedly
script:
- my-test-script.sh
# Wrong: image tag may change
# image: alpine:latest # Could update with breaking changes
# Check dependency install output
job:
script:
- npm ci # Look for errors in this step
- npm test
# If npm ci fails:
# - Check package-lock.json exists
# - Check registry access (npm config get registry)
# - Check network connectivity
# - Check for version conflicts
# Debug: list installed packages
script:
- npm list --depth=0
- pip list
# Pin versions in lockfiles:
# - npm: package-lock.json (committed)
# - pip: requirements.txt (pinned versions)
# - go: go.sum (committed)
# - maven: pom.xml (explicit versions)
Dependency Debugging:
- Print tool versions in
before_script — catches unexpected updates
- Pin base images to specific versions, not
latest
- Use
npm ci (not npm install) to respect lockfiles
- Check install step output before assuming the script is wrong
- Pin transitive dependencies if they cause conflicts
Common Pipeline Issues
Some pipeline issues are so common that they have known causes and fixes. Jobs not running when expected is almost always a rules or only/except problem. Duplicate pipelines are usually caused by branch pipelines running alongside MR pipelines. A pipeline that doesn't start at all might have a workflow:rules that blocks it, or all jobs might be filtered out [citation:7].
The changes keyword is a frequent culprit. It always evaluates to true in certain pipeline types (scheduled pipelines, tag pipelines), which means a job intended for merge requests runs in contexts where it shouldn't. The fix is to combine changes with an if clause that restricts it to the intended pipeline type: if: $CI_PIPELINE_SOURCE == "merge_request_event" .
Byte order marks (BOM) in the .gitlab-ci.yml file can cause bizarre behavior: jobs missing, variables with wrong values, configuration ignored. The BOM is invisible in most editors, so this is hard to diagnose. If the pipeline behavior is inexplicable, check for a BOM with an external tool. Some editors insert it automatically when saving as UTF-8 .
# Issue: Job not running
# Check rules
job:
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
when: never # This blocks the job!
- when: always
# Issue: Duplicate pipelines
# Fix with workflow:rules
workflow:
rules:
- if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
when: never
- if: $CI_COMMIT_BRANCH
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
# Issue: changes always true
# Fix: restrict to branch/MR pipelines
job:
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
changes:
- "*.js"
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
# Issue: Pipeline only has .pre/.post jobs
# A pipeline with only .pre or .post stages does not run
# Add at least one job in a regular stage
# Issue: Byte Order Mark (BOM)
# Check with external tool:
# file .gitlab-ci.yml
# hexdump -C .gitlab-ci.yml | head -1
# If output starts with ef bb bf, remove the BOM
# Issue: YAML too big
# Split into parent-child pipelines
# Or move long scripts to standalone files
# Issue: "Checking ability to merge automatically" stuck
# - Refresh MR page
# - Close & reopen MR
# - Rebase with /rebase
# - Merge with /merge (if ready)
# Issue: "Project not found or access denied"
# - Check include path format (my-group/my-project)
# - Check user has access to included project
# - Check included project exists
Debugging Checklist:
- Pipeline not running? Check
workflow:rules and job rules
- Job missing? Check
rules, only/except, and changes
- Duplicate pipelines? Add
$CI_OPEN_MERGE_REQUESTS rule
- Inexplicable behavior? Check for BOM
- YAML too big? Split into child pipelines
- Merge blocked? Check pipeline status and "Pipelines must succeed" setting
Systematic Debugging Workflow
When a pipeline fails, the temptation is to look at the error message and guess. A better approach is to follow a workflow that isolates the problem. Start with the cheapest checks: CI Lint and syntax validation. These catch configuration errors, which are the most common cause. If syntax is valid, check the job log for the actual error. If the error is unclear, enable CI_DEBUG_TRACE and run again. If the error is a variable or dependency issue, verify those specifically [citation:1][citation:3].
If the error is in the script itself, reproduce it locally. Most job images can be pulled and run locally with Docker. Running the script in the same container image isolates whether the problem is environment-specific (CI runner configuration) or code-specific (the script itself). If it fails locally, the problem is in the script or the image. If it succeeds locally but fails in CI, the problem is in the CI environment — variables, runner configuration, or network access [citation:8].
For complex failures, root cause analysis tools can help. GitLab Duo (Ultimate) can analyze a failed job log and suggest fixes. Even without Duo, reading the log carefully — starting from the first error, not the last — often reveals the cause. The last error is usually a consequence; the first error is usually the cause. Look for the first line that says "error" or "failed" and start there.
# Systematic debugging workflow
# Step 1: Validate syntax
# Build > CI Lint > Validate
# Or Pipeline editor > Validate tab
# Step 2: Check the job log
# Pipeline > (pipeline) > (job)
# Read from the first error, not the last
# Step 3: If error is unclear, enable debug trace
job:
variables:
CI_DEBUG_TRACE: "true"
script: ...
# Step 4: Check variables
script:
- echo "VAR=$VAR"
- env | sort
# Step 5: Check dependencies
before_script:
- node --version
- npm --version
# Step 6: Reproduce locally
# docker run -it --rm node:18 sh
# Run the same script commands
# Step 7: Check for common issues
# - rules/only/except
# - changes keyword
# - BOM
# - Duplicate pipelines
# Step 8: If available, use Root Cause Analysis
# GitLab Duo Chat > "Analyze this failed job"
# Log reading tips:
# - Start from the first error
# - Look for the command that failed
# - Check the exit code
# - Check the lines before the error for context
# - If the error is in a tool, check the tool's docs
Troubleshooting Reference
# Job fails: "command not found"
# - Check the image has the tool installed
# - Check PATH includes the tool's location
# - Check the before_script installs the tool
# Fix: Install the tool or use an image that includes it
# Job fails: "permission denied"
# - Check file permissions (chmod +x)
# - Check user permissions (non-root user?)
# - Check volume mounts (if any)
# Job fails: "connection refused"
# - Check the service is running
# - Check the hostname/port
# - Check network access from runner
# Job fails: "variable not set"
# - Check variable exists (echo $VAR)
# - Check variable is not protected (if on unprotected branch)
# - Check variable scope (environment)
# Job fails: "YAML invalid"
# - Use CI Lint
# - Check indentation (no tabs)
# - Check syntax (colons, quotes)
# Pipeline doesn't run
# - Check workflow:rules
# - Check job rules
# - Check pipeline is not blocked by "Pipelines must succeed"
# Job not in pipeline
# - Check rules/only/except
# - Check changes keyword
# - Check stage exists
# Duplicate pipelines
# - Add workflow rule: if $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
# when: never
# Debug: print all variables
script:
- env | sort | grep -v MASKED
# Debug: check tool versions
script:
- node --version
- npm --version
# Debug: check current directory
script:
- pwd
- ls -la
# Debug: check exit code of previous command
script:
- false || echo "Previous command failed with $?"
Frequently Asked Questions
What's the first thing to check when a pipeline fails?
Check the job log for the first error, not the last. The first error is usually the cause; subsequent errors are consequences. If the error is unclear, use CI Lint to validate syntax and enable CI_DEBUG_TRACE for verbose output [citation:1].
What's the difference between CI Lint and the pipeline editor?
CI Lint validates a pasted snippet and can simulate pipeline creation. The pipeline editor provides live validation as you edit, plus visualization and full configuration views. Use the editor for editing, CI Lint for quick validation [citation:5][citation:7].
Is CI_DEBUG_TRACE safe to use?
It's safe if you use it temporarily and all sensitive variables are masked. It's not safe as a permanent setting or as a runner-level variable, because it exposes all environment variables in job logs [citation:4].
Why is my job not running?
Most likely a rules or only/except issue. Check that the rule matches the pipeline source ($CI_PIPELINE_SOURCE), branch ($CI_COMMIT_BRANCH), or MR context. Also check workflow:rules, which can block the entire pipeline .
Why are two pipelines running for one push?
You have both a branch pipeline and an MR pipeline for the same commit. Add a workflow rule: if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS when: never to suppress the branch pipeline when an MR is open .
How do I check if a variable is set in a job?
Add echo "VAR=$VAR" to the script. If the variable is masked, it shows [MASKED] instead of the value. You can also check if it's empty: [ -z "$VAR" ] && echo "not set" [citation:1].
What's a Byte Order Mark (BOM) and why does it matter?
A BOM is an invisible character that some editors insert at the start of UTF-8 files. It can cause GitLab to ignore parts of the configuration, leading to missing jobs or wrong variables. Check with hexdump -C .gitlab-ci.yml | head -1 — if it starts with ef bb bf, remove the BOM .
How do I reproduce a CI job locally?
Pull the job image with docker pull <image>, run it with docker run -it --rm <image> sh, and run the script commands manually. This isolates whether the problem is environment-specific or script-specific [citation:8].