Common GitLab Issues

Runner Errors YAML Syntax Permissions Cache & Registry

A practical reference for the GitLab issues that come up most often. From runner failures and YAML syntax errors to permission problems and registry access, understand the cause and apply the fix.

Runner Fixes GitLab

How to Use This Reference

Most GitLab issues fall into a small number of categories: runner problems, YAML configuration errors, permission and access issues, cache and registry problems, and pipeline behavior that doesn't match expectations. Each category has a set of common causes and a set of fixes. Knowing the category narrows the search dramatically — a "job stuck pending" is almost always a runner availability problem, not a YAML problem.

The issues in this reference are the ones that appear most often in GitLab support tickets and community forums. Each entry describes the symptom (what you see), the likely cause (why it happens), and the fix (how to resolve it). Some issues have multiple possible causes; the fix section lists them in order of likelihood. Start with the most common cause and work down.

The most important diagnostic tool is the job log. Every issue leaves traces in the log — the error message, the command that failed, the exit code, the environment. Before applying a fix from this reference, read the log. It often contains the answer directly. If the log doesn't explain the issue, the debugging tools (CI Lint, debug trace, variable verification) narrow it down .

Core Principle: Read the error message first. Most issues have a clear error — the challenge is interpreting it, not finding it.

Runner Issues

Jobs pending, stuck, or failing with runner-related errors. Usually availability, tags, or permissions.

YAML Issues

Syntax errors, invalid keywords, indentation problems. Caught by CI Lint before commit.

Permission Issues

401/403 errors, access denied, protected variables not available. Usually role or scope.

Cache Issues

Cache not hitting, stale cache, cache corruption. Usually key, policy, or path.

Registry Issues

Cannot push/pull images, authentication fails, storage full. Usually credentials or registry state.

Pipeline Issues

Duplicates, jobs missing, pipeline not running. Usually rules or workflow:rules.

Runner Issues

Runner issues are the most common category of GitLab CI problems. A job that stays in "pending" state isn't failing — it's waiting for a runner that matches its requirements. The requirements are the runner's tags (jobs are assigned to runners with matching tags), the runner's scope (shared, group, or project), and the runner's availability (is it paused, offline, or at capacity?). If no runner matches, the job waits indefinitely .

The "job stuck pending" issue is almost always one of these. Check the job's tags against the runner tags. Check whether the runner is shared (available to all projects) or specific (available only to its project or group). Check whether the runner is online and not paused. If the runner is online but not picking up the job, check whether it's at capacity — a runner can run only one job at a time by default, so a second job waits until the first finishes .

Runner failures with "exit code 137" or "Killed" usually indicate the job exceeded the runner's memory limit. The runner (or the underlying executor) kills the process. The fix is to reduce the job's memory usage, increase the runner's memory, or split the job into smaller pieces. For Docker executors, the limit is set in the runner's config.toml under memory and memory_swap.

                # Issue: Job stuck "pending"
                # Causes:
                # - No runner matches job tags
                # - Runner is paused or offline
                # - Runner is at capacity
                # - Runner is not assigned to the project
                # Fix:
                # - Check job tags: CI/CD > Pipelines > (job) > Tags
                # - Check runner tags: Settings > CI/CD > Runners
                # - Ensure at least one runner has all job tags
                # - Check runner status (online, paused)
                # - Scale runner capacity if needed

                # Issue: "This job is stuck because you don't have any active runners"
                # Cause: No runners available for this project
                # Fix:
                # - Register a runner for the project
                # - Or enable shared runners for the project
                # - Settings > CI/CD > Runners > Shared runners

                # Issue: "no matching runner found"
                # Cause: Job tags don't match any runner
                # Fix:
                # - Remove job tags (use any runner)
                # - Or add a runner with the required tags

                # Issue: Job killed (exit code 137)
                # Cause: OOM (out of memory)
                # Fix:
                # - Reduce job memory usage
                # - Increase runner memory in config.toml
                # - Split job into smaller pieces

                # Issue: "Failed to pull image"
                # Cause: Runner cannot pull the image
                # Fix:
                # - Check image name/tag
                # - Check registry credentials
                # - Check runner network access
                # - For private registry: add image pull secret

                # Debug runner status:
                # Settings > CI/CD > Runners
                # Check: online status, tags, paused state
                # Or via API: GET /projects/:id/runners

Runner Debugging Checklist:

Is the runner online? (Settings > CI/CD > Runners)
Is it paused? (paused runners don't accept jobs)
Does it have the job's tags? (tags must match exactly)
Is it shared or specific? (shared runners serve all projects)
Is it at capacity? (one job at a time by default)
Does it have network access to the registry? (for image pulls)

YAML and Configuration Issues

YAML syntax errors are the most common configuration issue, and the most preventable. YAML is indentation-sensitive: a single misaligned space breaks the structure. Tabs are not allowed — YAML requires spaces. Keys must be followed by a colon and a space. Strings with special characters (colons, hashes, brackets) must be quoted. These rules are simple but easy to violate, especially when copying configuration from documentation .

The CI Lint tool catches most syntax errors before commit. It validates the YAML structure and the GitLab CI schema, reporting the line number and a description of the problem. The pipeline editor provides live validation as you type. Both are faster than pushing a commit and waiting for a pipeline to fail. If you're editing a large file, validate after every few changes — small errors are easier to find than accumulated ones.

Beyond syntax, configuration logic errors can cause pipelines to behave unexpectedly. A job with a rules clause that never matches won't run. A needs reference to a job that doesn't exist fails validation. A workflow:rules that evaluates to never blocks the entire pipeline. The pipeline editor's "Validate" tab simulates pipeline creation, catching these logic errors that syntax validation misses .

                # Issue: YAML invalid
                # Causes:
                # - Tabs instead of spaces
                # - Inconsistent indentation
                # - Missing space after colon
                # - Unquoted special characters
                # - Duplicate keys
                # Fix:
                # - Use CI Lint to identify the error
                # - Check indentation (2 spaces consistently)
                # - Quote strings with special characters
                # - Remove duplicate keys

                # Issue: "jobs config should contain at least one visible job"
                # Cause: All jobs are hidden (start with .) or excluded by rules
                # Fix:
                # - Add at least one visible job
                # - Or fix the rules that exclude all jobs

                # Issue: "unknown keys in the YAML"
                # Cause: Invalid keyword (typo, deprecated, or wrong level)
                # Fix:
                # - Check the keyword in GitLab CI/CD reference
                # - Fix spelling
                # - Move to correct level (global vs job)

                # Issue: "This GitLab CI configuration is invalid"
                # Cause: Various — schema violation
                # Fix:
                # - Read the error message for the specific problem
                # - Check the line number
                # - Use the pipeline editor to see the error inline

                # Issue: "config contains unknown keys: only"
                # Cause: only/except was removed in GitLab 18
                # Fix:
                # - Replace only/except with rules
                # - Example:
                # rules:
                #   - if: $CI_COMMIT_BRANCH == "main"

                # Issue: BOM in .gitlab-ci.yml
                # Cause: Editor inserts UTF-8 BOM
                # Fix:
                # - Check: hexdump -C .gitlab-ci.yml | head -1
                # - If starts with ef bb bf, remove BOM
                # - Use an editor that doesn't insert BOM

                # Issue: Included file not found
                # Cause: Wrong path, wrong ref, or file doesn't exist
                # Fix:
                # - Check path format (starts with /)
                # - Check ref exists
                # - Check file exists in that ref

                # Validate with CI Lint:
                # Build > CI Lint > Paste YAML > Validate

YAML Best Practices:

Validate with CI Lint before commit
Use 2 spaces consistently — never tabs
Quote strings with colons, hashes, or brackets
Use the pipeline editor for live validation
Check the Full configuration view to see the merged result

Permission and Access Issues

Permission issues manifest as 401 (unauthorized), 403 (forbidden), or "access denied" errors. In GitLab CI, these usually come from one of three sources: the user's role is too low for the operation, the CI job token lacks permission, or the resource is protected (protected branch, protected variable, protected environment) and the job doesn't meet the criteria .

Protected variables are a common source of confusion. A variable marked as protected is only available to jobs running on protected branches or tags. If a job runs on a feature branch and expects a protected variable, the variable is empty. The job doesn't fail with "permission denied" — it fails because the variable is missing or empty. The fix is either to run the job on a protected branch or to remove the protection from the variable (if it's not actually sensitive).

CI job token permissions are another frequent issue. By default, CI_JOB_TOKEN has limited access to other projects. If a job in project A tries to access the API of project B, the token is only allowed if project B has explicitly allowed it (via the allowlist) or if the token is from a project the user can access. Since GitLab 16, the default is to restrict CI_JOB_TOKEN to the same project unless explicitly allowed .

                # Issue: 401 Unauthorized
                # Causes:
                # - Wrong credentials (username/password, token)
                # - Token expired or revoked
                # - User role insufficient
                # Fix:
                # - Check credentials (CI_JOB_TOKEN, CI_REGISTRY_PASSWORD)
                # - Check token scope/expiry
                # - Check user role (Maintainer+ for many operations)

                # Issue: 403 Forbidden
                # Causes:
                # - User role insufficient (Developer vs Maintainer)
                # - Protected branch/environment restriction
                # - CI_JOB_TOKEN restrictions
                # Fix:
                # - Increase user role if appropriate
                # - Check protected branch settings
                # - Check protected environment settings
                # - Check CI_JOB_TOKEN allowlist

                # Issue: Protected variable not available
                # Causes:
                # - Job runs on unprotected branch
                # - Variable marked protected
                # Fix:
                # - Run job on protected branch (main, release/*)
                # - Or unprotect the variable if not sensitive

                # Issue: "insufficient permissions to access this resource"
                # Cause: User or token lacks permission
                # Fix:
                # - Check user role (Maintainer, Owner)
                # - Check resource is not protected
                # - Check token scope

                # Issue: "Project not found or access denied"
                # Cause: Included project inaccessible
                # Fix:
                # - Check project path format (group/project)
                # - Check user has access to included project
                # - Check project exists (not deleted/renamed)

                # Issue: "The project is not allowed to access the CI_JOB_TOKEN"
                # Cause: CI_JOB_TOKEN restrictions
                # Fix:
                # - Enable allowlist for the target project
                # - Settings > CI/CD > Token Access
                # - Add the source project to allowlist

                # Debug: check current user/token
                script:
                  - echo "User: $GITLAB_USER_LOGIN"
                  - echo "Token: $CI_JOB_TOKEN"
                  - curl --header "JOB-TOKEN: $CI_JOB_TOKEN" \
                      "https://gitlab.example.com/api/v4/user"

Cache Issues

Cache issues are subtle because cache is an optimization — when it fails, the job still runs, it's just slower. The most common symptom is "cache not hitting": the job re-downloads dependencies every time. The cause is usually a cache key that changes too often. If the key is $CI_COMMIT_SHA, it changes on every commit, so the cache never hits. The fix is to key by the lockfile hash, which only changes when dependencies change .

The second common issue is stale cache: the cache hits, but it's outdated, so the job uses old dependencies. This happens when the key doesn't change even though dependencies did — for example, if the lockfile is updated but the key is static. The fix is to ensure the key includes the lockfile. If the lockfile changes, the key changes, and a new cache is created.

Cache corruption is rare but real. If a cache upload is interrupted or a runner crashes mid-write, the cache can be corrupted. The fix is to change the cache key (forcing a new cache) or clear the runner cache. In GitLab UI, you can clear the cache from the project's CI/CD settings. On self-hosted runners, delete the cache directory.

                # Issue: Cache not hitting
                # Causes:
                # - Cache key changes on every run (e.g., $CI_COMMIT_SHA)
                # - Cache path doesn't match what's installed
                # - Cache policy is pull (no upload)
                # Fix:
                cache:
                  key:
                    files:
                      - package-lock.json  # Changes only when lockfile changes
                  paths:
                    - node_modules/
                    - .npm/

                # Issue: Stale cache
                # Causes:
                # - Cache key doesn't change when dependencies change
                # - Cache shared across branches with different dependencies
                # Fix:
                # - Key by lockfile hash
                # - Or add branch to key: key: "$CI_COMMIT_REF_SLUG-$LOCKFILE_HASH"

                # Issue: Cache corruption
                # Causes:
                # - Interrupted upload
                # - Runner crash mid-write
                # - Disk full on runner
                # Fix:
                # - Change cache key (forces new cache)
                # - Clear runner cache
                # - Check runner disk space

                # Issue: Cache too large
                # Causes:
                # - Caching files that change (build outputs)
                # - Caching too many directories
                # Fix:
                # - Cache dependencies only (not build outputs)
                # - Use artifacts for build outputs
                # - Limit paths to necessary directories

                # Issue: Cache not shared across jobs
                # Causes:
                # - Different keys
                # - Different runners
                # - Different paths
                # Fix:
                # - Use the same key and paths for all jobs
                # - For different runners, use distributed cache (S3, GCS)

                # Clear cache:
                # UI: Build > Pipelines > Clear runner caches
                # Or API: POST /projects/:id/repository/files/...
                # Or change cache key to force new cache

                # Debug cache:
                script:
                  - ls -la node_modules/ 2>/dev/null || echo "No cache"
                  - du -sh node_modules/ 2>/dev/null

Registry Issues

Registry issues fall into two categories: authentication and storage. Authentication failures (401 Unauthorized) happen when the registry credentials are wrong or missing. In GitLab CI, the credentials are $CI_REGISTRY_USER and $CI_REGISTRY_PASSWORD, which are automatically available. If you're using a different registry (Docker Hub, AWS ECR), you need to configure the credentials explicitly .

Storage issues happen when the registry is full or the image is too large. The registry has a configurable storage limit (for self-managed, in gitlab.rb; for GitLab.com, in the plan). When the limit is reached, pushes fail with "quota exceeded" or "storage limit reached". The fix is to clean up old images (use the cleanup policy) or increase the limit.

Another common registry issue is the "latest tag" problem: a job pushes my-image:latest, and downstream jobs pull my-image:latest, but the tag has been updated between push and pull. The fix is to use immutable tags (commit SHA) and pass the tag explicitly between jobs. This also makes rollback reliable — the previous tag still exists in the registry .

                # Issue: "unauthorized: authentication required"
                # Causes:
                # - Not logged in to registry
                # - Wrong credentials
                # - Credentials not set
                # Fix:
                before_script:
                  - echo $CI_REGISTRY_PASSWORD | docker login -u $CI_REGISTRY_USER --password-stdin $CI_REGISTRY

                # Issue: "denied: requested access to the resource is denied"
                # Causes:
                # - No permission to push to registry
                # - Registry is read-only
                # - Project doesn't have registry enabled
                # Fix:
                # - Check project registry is enabled
                # - Check user role (Developer+ to push)
                # - Check registry visibility settings

                # Issue: "quota exceeded" or "storage limit reached"
                # Causes:
                # - Registry storage limit reached
                # - Too many images/tags
                # Fix:
                # - Run cleanup policy
                # - Settings > Packages and registries > Cleanup policies
                # - Or delete old tags manually
                # - Or increase storage (self-managed)

                # Issue: "Cannot pull image: manifest unknown"
                # Causes:
                # - Image/tag doesn't exist
                # - Wrong image name
                # - Registry unreachable
                # Fix:
                # - Check image name and tag
                # - Check image was pushed successfully
                # - Check registry access

                # Issue: Using :latest tag causes wrong image
                # Causes:
                # - :latest changes between push and pull
                # Fix:
                # - Use commit SHA tag (immutable)
                # - Pass tag explicitly between jobs
                build:
                  script:
                    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA .
                    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
                deploy:
                  script:
                    - docker pull $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA  # Same image

                # Debug registry access:
                script:
                  - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
                  - docker pull $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
                  - curl --header "PRIVATE-TOKEN: $GITLAB_TOKEN" \
                      "https://gitlab.example.com/api/v4/projects/:id/registry/repositories"

Pipeline Behavior Issues

Pipeline behavior issues are the ones where the pipeline runs, but doesn't do what you expect. A job doesn't run when it should. A job runs when it shouldn't. Two pipelines run for one push. These are almost always rules or workflow:rules issues, and they're diagnosable by examining the configuration against the pipeline context .

The most common is "job not running." The job has a rules clause that doesn't match the current pipeline source. For example, the rule is if: $CI_PIPELINE_SOURCE == "merge_request_event", but the pipeline is a branch pipeline (not an MR pipeline). The rule doesn't match, so the job is excluded. The fix is to ensure the rule matches the pipeline source, or to use workflow:rules to ensure the pipeline itself is the right type .

Duplicate pipelines are another common issue. A push to a branch with an open MR can trigger both a branch pipeline and an MR pipeline. The fix is a workflow:rules that suppresses the branch pipeline when an MR is open: if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS when: never. This leaves only the MR pipeline, which is the one that matters for the MR's merge eligibility.

                # Issue: Job not running
                # Causes:
                # - rules don't match pipeline source
                # - changes keyword doesn't match
                # - workflow:rules blocks pipeline
                # Fix:
                job:
                  rules:
                    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
                    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
                    # Add the context where you expect the job to run

                # Issue: Duplicate pipelines
                # Causes:
                # - Both branch and MR pipelines run
                # Fix:
                workflow:
                  rules:
                    - if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
                      when: never
                    - if: $CI_COMMIT_BRANCH
                    - if: $CI_PIPELINE_SOURCE == "merge_request_event"

                # Issue: Pipeline not running at all
                # Causes:
                # - workflow:rules evaluates to never
                # - All jobs excluded by rules
                # - No .gitlab-ci.yml file
                # Fix:
                workflow:
                  rules:
                    - if: $CI_PIPELINE_SOURCE == "push"
                    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
                    - when: always  # Or specific conditions

                # Issue: changes keyword always true
                # Causes:
                # - changes always true in scheduled/tag pipelines
                # Fix:
                job:
                  rules:
                    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
                      changes:
                        - "*.js"

                # Issue: "Checking ability to merge automatically" stuck
                # Causes:
                # - MR is in a pending state
                # - Pipeline is running
                # Fix:
                # - Refresh MR page
                # - Close and reopen MR
                # - Rebase with /rebase
                # - Wait for pipeline to finish

                # Debug pipeline source:
                script:
                  - echo "Source: $CI_PIPELINE_SOURCE"
                  - echo "Branch: $CI_COMMIT_BRANCH"
                  - echo "MR: $CI_MERGE_REQUEST_IID"
                  - echo "Open MRs: $CI_OPEN_MERGE_REQUESTS"

Frequently Asked Questions

Why is my job stuck in "pending"?

Usually a runner issue: no runner matches the job's tags, the runner is paused or offline, or the runner is at capacity. Check runner status in Settings > CI/CD > Runners and verify tags match .

Why is my job being killed (exit code 137)?

Out of memory. The job exceeded the runner's memory limit. Reduce memory usage, split the job, or increase the runner's memory limit in config.toml .

Why is my protected variable empty?

Protected variables are only available on protected branches or tags. If the job runs on a feature branch, the variable is empty. Run on main or a protected tag, or unprotect the variable if it's not actually sensitive .

Why is my cache not hitting?

The cache key changes on every run, or the paths don't match, or the policy is pull (no upload). Key by lockfile hash, use pull-push on build jobs, and verify paths match where dependencies are installed .

Why can't I push to the registry?

Either authentication or permission. Check docker login with $CI_REGISTRY_USER/$CI_REGISTRY_PASSWORD. Check your role (Developer+ to push). Check the registry isn't at its quota .

Why do I get duplicate pipelines?

Both branch and MR pipelines are running. Add a workflow rule: if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS when: never. This suppresses the branch pipeline when an MR is open .

Why is my job not running when I expect it to?

The rules clause doesn't match the pipeline source. Check $CI_PIPELINE_SOURCE, $CI_COMMIT_BRANCH, and $CI_MERGE_REQUEST_IID. Add the context where you expect the job to run .

What's a Byte Order Mark and how do I remove it?

A BOM is an invisible character at the start of UTF-8 files that can break GitLab parsing. Check with hexdump -C .gitlab-ci.yml | head -1. If it starts with ef bb bf, use an editor that saves without BOM or strip it with sed -i '1s/^\xEF\xBB\xBF//' .gitlab-ci.yml .

Common GitLab Issues

Runner Issues

YAML Issues

Permission Issues

Cache Issues

Registry Issues

Pipeline Issues

Debugging Pipelines

Runner Troubleshooting

Anti-Patterns

.gitlab-ci.yml

Best Practices