Runner Troubleshooting

Stuck Jobs DNS & Network OOM & Resources Cache & Docker

A practical guide to diagnosing and fixing GitLab Runner problems. From stuck jobs and tag mismatches to DNS resolution failures, OOM kills, and cache access errors, understand the cause and apply the fix.

Runner Fixes GitLab CI/CD

Where Runner Problems Come From

GitLab Runner sits between GitLab and your job's execution environment. Problems can originate from any of the three: GitLab not assigning the job, the runner not receiving or accepting the job, or the execution environment failing to run it. The first step in troubleshooting is determining which layer is failing. A "stuck" job that never starts is a GitLab-to-runner problem. A job that starts and immediately fails is an execution environment problem .

The runner's logs are the primary diagnostic tool. On Linux, journalctl -u gitlab-runner shows the runner's activity, including job assignments, errors, and heartbeats. Enabling debug logging (log_level = "debug" in config.toml) provides more detail, but it exposes secrets in the logs, so it should be used temporarily and carefully [citation:1]. The job log itself shows what happened inside the container, which is where most execution failures are visible.

Many runner issues are configuration mismatches. The job requests tags that no runner has. The runner is registered to a different project. The runner is paused or offline. The Docker executor's DNS configuration doesn't match the host's. These are not bugs — they're configuration errors, and they're fixable once identified. This guide covers the most common ones, organized by symptom.

Core Principle: A stuck job is almost always a runner availability or tag issue. A failing job is almost always an execution environment issue. Distinguish the two before debugging.

Stuck Jobs

Job stays "pending" forever. No runner matches tags, runner is offline/paused, or capacity is exhausted.

DNS Issues

"Could not resolve host" or authentication failures. Docker executor doesn't inherit host DNS.

OOM Kills

Exit code 137. Job exceeded memory limit. Reduce memory usage or increase limit.

Cache Issues

"403 Forbidden" on S3 cache. Missing ListBucket permission or wrong credentials.

Docker Executor

Service containers fail, image pull fails, or DNS doesn't work. Configuration-specific issues.

Logs & Debug

journalctl, debug logging, correlation IDs. The primary diagnostic tools.

Stuck Jobs (Pending Forever)

A stuck job is one that stays in "pending" state and never starts. The error message usually says "This job is stuck because you don't have any active runners online or available with any of these tags assigned to them: X." The message names the tag that no runner satisfies, which is the first clue .

The most common cause is a tag mismatch. A job with tags: [docker] requires a runner that has the docker tag. If no online runner has that tag, the job waits. The fix is either to add the tag to a runner or to remove the tag from the job (if it doesn't actually need a specific runner). A related issue is untagged jobs: a job without tags only runs on a runner that has "Run untagged jobs" enabled. If all runners are tag-restricted, untagged jobs hang .

Other causes include: the runner service isn't running (registered but not started), shared runners are disabled for the project, or the runner is at capacity (concurrent limit reached). The diagnostic workflow is: check if any runner is online for the project, check its tags against the job's tags, check if it's paused or at capacity, and check if the service is actually running .

                # Issue: Job stuck "pending"
                # Check the exact error message
                # "no active runners online or available with tags: docker"
                # -> Tag mismatch

                # Check runners for the project
                # Settings > CI/CD > Runners
                # Look for: online status, tags, paused state

                # Or via API
                curl -s --header "PRIVATE-TOKEN: $TOKEN" \
                  "https://gitlab.example.com/api/v4/projects/:id/runners" \
                  | jq '.[] | {description, online, paused, tag_list}'

                # Check runner service is running
                sudo gitlab-runner status
                sudo gitlab-runner verify

                # Check runner logs
                sudo journalctl -u gitlab-runner --no-pager | tail -30

                # Fix: Add tag to runner or remove from job
                # .gitlab-ci.yml
                job:
                  tags:
                    - docker  # Must match a runner's tag

                # Fix: Enable shared runners for the project
                # Settings > CI/CD > Runners
                # Toggle "Enable shared runners for this project"

                # Fix: Increase runner capacity
                # config.toml
                concurrent = 4  # Global concurrency
                [[runners]]
                  limit = 2  # Per-runner limit

                # Fix: Unpause runner
                # Settings > CI/CD > Runners > (runner) > Pause/Unpause

                # Fix: Start runner service
                sudo gitlab-runner start

                # Debug: check runner version compatibility
                # Runner and GitLab versions should match major version

Stuck Job Checklist:

Does the job have tags? Do any runners have those tags?
Is the runner online and not paused?
Is the runner service running? (gitlab-runner status)
Are shared runners enabled for the project?
Is the runner at capacity? (check concurrent and limit)

DNS and Network Issues

DNS issues are common with the Docker executor because containers don't always inherit the host's DNS configuration. If the host uses a VPN with its own DNS server, or an internal DNS service, the container may use a different resolver and fail to resolve internal hostnames. The symptom is "Could not resolve host" or an authentication failure that looks like a credential problem but is actually a DNS routing problem [citation:1].

The fix is to explicitly configure DNS for the Docker executor in config.toml. The dns setting specifies which DNS servers the container should use, and dns_search specifies the search domains. For hostnames that have no DNS record (only an IP), extra_hosts maps the hostname to an IP, equivalent to adding an entry to /etc/hosts [citation:14].

Another common network issue is that service containers (like a PostgreSQL database defined in services:) aren't reachable from the job container. This can happen when Docker's legacy links functionality is deprecated or when the job and services don't share a network. Setting FF_NETWORK_PER_BUILD: "true" creates a per-build network where job and services can resolve each other by alias [citation:14].

                # Issue: "Could not resolve host" or authentication failure
                # Cause: Docker executor doesn't use host DNS
                # Fix: Configure DNS in config.toml
                [[runners]]
                  [runners.docker]
                    dns = ["10.0.0.2", "1.1.1.1"]
                    dns_search = ["internal.example.com"]

                # Issue: Internal hostname has no DNS record
                # Fix: Add extra_hosts mapping
                [[runners]]
                  [runners.docker]
                    extra_hosts = ["artifacts.corp.local:10.20.0.15"]

                # Issue: Service containers not reachable
                # Cause: Job and services don't share network
                # Fix: Enable per-build network
                variables:
                  FF_NETWORK_PER_BUILD: "true"

                # Debug DNS inside job
                script:
                  - cat /etc/resolv.conf
                  - getent hosts gitlab.example.com
                  - nslookup gitlab.example.com
                  - ping -c 1 postgres  # Service alias

                # Issue: Docker 29.x.x + service containers fail
                # Error: "No HOST or PORT found"
                # Cause: Legacy links deprecated
                # Fix: Re-enable legacy links (temporary)
                # /etc/systemd/system/docker.service.d/override.conf
                # [Service]
                # Environment="DOCKER_KEEP_DEPRECATED_LEGACY_LINKS_ENV_VARS=1"
                sudo systemctl daemon-reload
                sudo systemctl restart docker
                sudo gitlab-runner restart

                # Or downgrade Docker to 28.x or earlier

DNS Debugging Steps:

Check /etc/resolv.conf inside the job container
Try getent hosts — if it fails, it's DNS
Check if the host can resolve the name (if yes, container DNS is misconfigured)
Add dns or extra_hosts to config.toml

OOM and Resource Issues

Exit code 137 is the signature of an OOM (out of memory) kill. The kernel (or Kubernetes) terminated the process because it exceeded the memory limit. For Docker executor, the limit is set in the runner's config.toml under memory and memory_swap. For Kubernetes executor, it's set via KUBERNETES_MEMORY_LIMIT and KUBERNETES_MEMORY_REQUEST [citation:4].

The fix depends on the cause. If the job legitimately needs more memory (a large build, a memory-intensive test), increase the limit. If the job has a memory leak or is allocating more than necessary, fix the job. If the job is running on a runner with insufficient total memory, move it to a larger runner or split it into smaller jobs. Profiling the job locally with docker stats reveals its actual memory usage [citation:4].

For Kubernetes executor, there's a known issue where rapid memory allocation (e.g., Java's -XX:+AlwaysPreTouch) can trigger a false-positive OOM kill even when the total memory is below the limit. The workaround is to remove the rapid allocation flag or to tune the container's memory settings. This is a Kubernetes behavior, not a GitLab Runner bug [citation:12].

# Issue: Job killed (exit code 137) # Cause: OOM (out of memory) # Fix: Increase memory limit or reduce usage # Docker executor (config.toml) [[runners]] [runners.docker] memory = "2g" memory_swap = "2g" # Kubernetes executor (CI variables) job: variables: KUBERNETES_MEMORY_REQUEST: "512Mi" KUBERNETES_MEMORY_LIMIT: "1Gi" KUBERNETES_CPU_REQUEST: "500m" # Debug memory usage locally docker run -it --rm --memory=1g sh docker stats # Check if job is actually OOM # Look for "Killed" or "OOMKilled" in job log # Check runner logs for memory-related errors # Fix: Split job into smaller pieces # Instead of one job that processes everything: job_shard_1: parallel: 4 script: npm test -- --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL # Fix: Reduce memory usage # - Use more efficient tools # - Set Node.js memory limit: NODE_OPTIONS=--max-old-space-size=1024 # - Stream data instead of loading it all # Kubernetes false-positive OOM # Remove rapid allocation flags (e.g., -XX:+AlwaysPreTouch) # Or increase limit significantly above actual usage

Cache Issues

Cache issues with self-hosted runners usually involve distributed caching (S3, GCS, MinIO). The most common error is "403 Forbidden" when the runner tries to access the S3 bucket. The cause is almost always a missing IAM permission. The runner needs s3:PutObject and s3:GetObject, but it also needs s3:ListBucket to check if the cache exists before downloading. If ListBucket is missing, the runner gets a 403 and the cache fails [citation:7].

For single-runner setups without distributed caching, cache is stored locally on the runner. This works, but it means the cache is only available on the runner that uploaded it. If you have multiple runners or autoscaling, a job picked up by a different runner won't find the cache. The fix is to configure distributed caching (S3-compatible object storage) so all runners share the same cache [citation:15].

Another common cache issue is "No URL provided, cache will not be downloaded/uploaded." This means the runner couldn't construct a valid cache URL, usually because the config.toml cache configuration is incomplete or malformed. Check the [runners.cache] section for the correct keys and values [citation:9].

                # Issue: Cache access denied (S3)
                # Error: "403 Forbidden, s3:ListBucket not allowed"
                # Cause: IAM policy missing ListBucket permission
                # Fix: Add ListBucket to the policy
                {
                  "Version": "2012-10-17",
                  "Statement": [
                    {
                      "Effect": "Allow",
                      "Action": [
                        "s3:PutObject",
                        "s3:GetObject"
                      ],
                      "Resource": ["arn:aws:s3:::my-bucket/*"]
                    },
                    {
                      "Effect": "Allow",
                      "Action": ["s3:ListBucket"],
                      "Resource": ["arn:aws:s3:::my-bucket"]
                    }
                  ]
                }

                # Issue: Cache not shared between runners
                # Cause: Local cache, not distributed
                # Fix: Configure distributed caching
                [[runners]]
                  [runners.cache]
                    Type = "s3"
                    Path = "gitlab-runner"
                    Shared = true
                    [runners.cache.s3]
                      ServerAddress = "s3.amazonaws.com"
                      BucketName = "my-cache-bucket"
                      BucketLocation = "us-east-1"
                      AuthenticationType = "iam"

                # Issue: "No URL provided, cache will not be downloaded/uploaded"
                # Cause: Invalid or missing cache configuration
                # Fix: Check [runners.cache] section in config.toml
                # Ensure all required keys are present

                # Issue: Cache corrupted
                # Fix: Change cache key or clear cache
                # UI: Build > Pipelines > Clear runner caches
                # Or: change cache key in .gitlab-ci.yml

                # Debug: check cache configuration
                script:
                  - echo "Cache key: $CI_COMMIT_REF_SLUG"
                  - ls -la node_modules/ 2>/dev/null || echo "No cache"

Docker Executor Issues

The Docker executor is the most common executor, and it has its own set of common issues. Image pull failures are usually authentication problems: the runner tries to pull a private image without credentials, or with the wrong credentials. The fix is to configure DOCKER_AUTH_CONFIG as a CI/CD variable with the registry credentials .

Service container failures are another common issue. Services (like a database) run alongside the job container, and the job connects to them by hostname. If the service isn't ready when the job starts, the connection fails. The fix is to add a health check or a wait script. For Docker 29.x.x, service containers may fail with "No HOST or PORT found" because legacy links were deprecated. The workaround is to re-enable legacy links or downgrade Docker [citation:8].

Authentication failures that look like credential problems can actually be DNS problems. If the runner can't resolve the registry hostname, it can't authenticate, and the error says "unauthorized" instead of "could not resolve host." Check DNS first if authentication fails with valid credentials [citation:1].

                # Issue: "failed to pull image" (unauthorized)
                # Cause: Private image, no credentials
                # Fix: Add DOCKER_AUTH_CONFIG as CI/CD variable
                # Compute the config:
                # echo -n "username:password" | base64
                # Then:
                # DOCKER_AUTH_CONFIG = {
                #   "auths": {
                #     "registry.example.com": {
                #       "auth": ""
                #     }
                #   }
                # }

                # Issue: Service container not reachable
                # Cause: Service not ready or network issue
                # Fix: Add health check / wait script
                job:
                  services:
                    - postgres:15
                  variables:
                    POSTGRES_DB: test
                    POSTGRES_USER: test
                    POSTGRES_PASSWORD: test
                  script:
                    - until pg_isready -h postgres -p 5432; do sleep 1; done
                    - npm test

                # Issue: Docker 29.x.x + services fail
                # Error: "No HOST or PORT found"
                # Fix: Re-enable legacy links or downgrade Docker
                # See DNS section above

                # Issue: Runner can't pull from GitLab registry
                # Cause: DNS, credentials, or registry access
                # Fix: Check DNS first, then credentials
                script:
                  - getent hosts registry.gitlab.com
                  - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

                # Issue: "dial tcp: lookup ... no such host"
                # Cause: DNS resolution failure
                # Fix: Add dns to config.toml (see DNS section)

Logs and Debugging

The runner's logs are the primary diagnostic tool for runner-level issues. On Linux with systemd, journalctl -u gitlab-runner shows the runner's activity. On other systems, the logs are in /var/log/gitlab-runner/ or the configured log directory. The logs show job assignments, errors, and the runner's heartbeat to GitLab [citation:1].

Debug logging (log_level = "debug") provides more detail, including the commands the runner executes and the variables it uses. But it also exposes secrets in the logs. For this reason, debug logging should be enabled temporarily and disabled after diagnosis. Masked variables are protected in job logs but not in runner logs, so a secret that's masked in the job log can appear in the runner debug log [citation:1].

GitLab Runner also generates a correlation ID for each API request. When a request fails, the correlation ID in the runner log can be matched with the GitLab server log to trace the request across both systems. This is useful for diagnosing issues that span the runner and GitLab, like job assignment failures or trace upload problems [citation:1].

                # View runner logs (systemd)
                sudo journalctl -u gitlab-runner --no-pager | tail -50

                # Enable debug logging (temporary)
                # config.toml
                log_level = "debug"
                # Then restart runner
                sudo gitlab-runner restart
                # Disable after debugging: log_level = "info"

                # Or run with debug (command line)
                sudo gitlab-runner stop
                sudo gitlab-runner --debug run
                # (Not recommended for shell executor)

                # Correlation ID example
                # Look for correlation_id in runner logs
                # Match with GitLab server logs
                # If correlation_id is a fallback UUID (not ULID),
                # the request didn't reach GitLab Workhorse
                # (problem is in WAF/CDN/proxy)

                # Check runner status
                sudo gitlab-runner status
                sudo gitlab-runner verify

                # Check runner version
                gitlab-runner --version

                # Check config.toml validity
                sudo gitlab-runner verify --config /etc/gitlab-runner/config.toml

                # Debug specific job
                # Look for job ID in runner logs
                # Search: journalctl -u gitlab-runner | grep ""

                # Common log patterns:
                # "Checking for jobs... received" -> job assigned
                # "Job succeeded" -> job finished
                # "Failed to update executor" -> executor issue
                # "ERROR: Job failed" -> job failure (check job log)

Frequently Asked Questions

Why is my job stuck in "pending" and never starts?

Usually a tag mismatch or runner availability. Check that a runner with the job's tags is online and not paused. Check that the runner service is running. Check that shared runners are enabled if you rely on them .

Why does my job fail with "Could not resolve host"?

The Docker executor isn't using the host's DNS. Add dns or extra_hosts to config.toml. For service containers, enable FF_NETWORK_PER_BUILD so job and services share a network [citation:14].

Why is my job killed with exit code 137?

OOM (out of memory). The job exceeded the memory limit. Increase the limit in config.toml (Docker executor) or via KUBERNETES_MEMORY_LIMIT (Kubernetes executor), or reduce the job's memory usage [citation:4].

Why does my S3 cache return 403 Forbidden?

The IAM policy is missing s3:ListBucket. The runner needs it to check if the cache exists. Add it to the policy with the bucket ARN (not the object ARN) [citation:7].

Why can't my runner pull a private image?

The runner doesn't have registry credentials. Configure DOCKER_AUTH_CONFIG as a CI/CD variable with the registry credentials. The format is a JSON with the base64-encoded username:password .

Why do service containers fail on Docker 29.x.x?

Docker 29 deprecated legacy links, which GitLab Runner uses for service discovery. Re-enable them with DOCKER_KEEP_DEPRECATED_LEGACY_LINKS_ENV_VARS=1 in Docker's systemd config, or downgrade to Docker 28.x [citation:8].

How do I enable debug logging safely?

Set log_level = "debug" in config.toml, restart the runner, reproduce the issue, then set it back to "info". Debug logs expose secrets, so never leave them enabled permanently [citation:1].

How do I check if my runner is registered and online?

Go to Settings > CI/CD > Runners. Or use the API: GET /projects/:id/runners. Check the online and paused fields. Also run sudo gitlab-runner verify on the runner host .

Stuck jobs are availability issues. Failing jobs are execution issues. Check the logs, isolate the layer, apply the fix.

Runner Troubleshooting

Stuck Jobs

DNS Issues

OOM Kills

Cache Issues

Docker Executor

Logs & Debug

GitLab Runners

Runner Executors

Common GitLab Issues

Debugging Pipelines

Runner Security