Pipeline Optimization
DAG & Parallelism
Caching
Monitoring
Resource Groups
Learn how to make GitLab CI/CD pipelines faster. From DAG dependencies and parallel execution to caching strategies, resource groups, and performance monitoring, cut pipeline times without sacrificing safety.
Speed
Monitoring
GitLab CI/CD
Why Pipeline Optimization Matters
Pipeline speed is not a luxury — it's a direct multiplier on developer productivity. A pipeline that takes 45 minutes means every push costs 45 minutes of waiting, every merge request takes 45 minutes longer to review, and every developer context-switches while waiting. Reduce that to 10 minutes, and you've given every developer back 35 minutes per push. Across a team of ten, that's hours of recovered focus every week [citation:2].
Optimization is also about cost. GitLab compute minutes are billed, and inefficient pipelines burn through them. A pipeline that rebuilds dependencies in every job, downloads every artifact in every stage, and runs tests sequentially is paying for compute that could be avoided. Caching, parallelization, and DAG dependencies aren't just faster — they're cheaper [citation:2][citation:4].
The most common bottleneck is the sequential stage model. By default, every job in a stage must finish before any job in the next stage starts. If your build stage has a fast frontend build and a slow backend build, the frontend tests wait for the backend build even though they don't depend on it. The needs keyword breaks this dependency, letting jobs start as soon as their actual inputs are ready .
Core Principle: Optimize for the critical path. Find the slowest chain of dependent jobs and shorten it. Everything else is secondary.
needs (DAG)
Jobs start as soon as their dependencies finish. Breaks the sequential stage barrier.
parallel:matrix
Run the same job with different variable combinations. Ideal for version matrices and test sharding.
Cache
Reuse dependencies across pipelines. Key by lockfile hash. Use pull-push for build, pull for test.
resource_group
Serialize deployment jobs without blocking unrelated jobs. Choose process mode (oldest_first, newest_first).
CI/CD Analytics
View pipeline duration trends, failure rates, and (with ClickHouse) job-level P50/P95 metrics.
Observability
Export traces and metrics to OpenTelemetry. Identify slow jobs and bottlenecks in production-like pipelines.
DAG with needs
The needs keyword is the single most impactful optimization for most pipelines. Without it, GitLab runs jobs stage by stage: all build jobs finish, then all test jobs, then all deploy jobs. With needs, a job can start as soon as the specific jobs it depends on finish, even if other jobs in the same stage are still running. This creates a directed acyclic graph (DAG) instead of a linear sequence .
A common pattern is the fan-out/fan-in structure. A single build job produces an artifact, multiple test jobs depend on that one build, and a deploy job depends on all the test jobs. Each test runs as soon as the build finishes, and deploy waits for all tests. This is more efficient than running all builds, then all tests, then deploy — especially when builds or tests have different durations .
Another pattern is immediate start. Jobs that don't depend on any previous job — linting, static analysis, secret detection — can use needs: [] to start immediately when the pipeline is created, without waiting for the build stage. They run in parallel with the build, giving faster feedback on issues that don't require compiled code .
# Fan-out/fan-in with needs
build:
stage: build
script: npm run build
artifacts:
paths: [dist/]
test_unit:
stage: test
needs: [build]
script: npm run test:unit
test_integration:
stage: test
needs: [build]
script: npm run test:integration
deploy:
stage: deploy
needs: [test_unit, test_integration]
script: ./deploy.sh
# Immediate start (no dependencies)
lint:
stage: test
needs: []
script: npm run lint
# Optional dependency (job may not exist due to rules)
deploy_optional:
needs:
- job: test_optional
optional: true
- job: test_unit
# needs with parallel:matrix (1:1 mapping)
build_matrix:
stage: build
parallel:
matrix:
- OS: [ubuntu, alpine]
NODE: [18, 20]
script: ./build.sh
deploy_matrix:
stage: deploy
parallel:
matrix:
- OS: [ubuntu, alpine]
NODE: [18, 20]
needs:
- job: build_matrix
parallel:
matrix:
- OS: ['$[[ matrix.OS ]]']
NODE: ['$[[ matrix.NODE ]]']
script: ./deploy.sh
needs Best Practices:
- Use
needs: [] for linting and scanning jobs that don't need build artifacts
- Use
needs to create independent execution paths for monorepos
- Use
optional: true when combining needs with rules that may skip jobs
- View the job dependency graph to verify the DAG structure
Parallel Execution
Some jobs are inherently serial — you can't test before you build. But many jobs can run in parallel: different test suites, different platform builds, different version matrices. GitLab's parallel keyword runs multiple instances of the same job, each with a different index. This is ideal for sharding a large test suite: split 1000 tests across 5 parallel jobs, each running 200 tests, and the suite finishes in one-fifth the time [citation:3].
parallel:matrix goes further: it runs the job with different variable combinations. Testing against Node 18 and 20 across Ubuntu and Alpine is four jobs, each with its own environment. This is essential for libraries and applications that need to support multiple versions or platforms. The matrix generates all combinations automatically, so you write the job once [citation:3].
Parallelism has a cost: it requires more runners. If you don't have enough runners, parallel jobs queue and the benefit disappears. Monitor runner availability and scale up if queue times are long. The goal is to find the sweet spot where parallelism reduces wall-clock time without creating a queue bottleneck [citation:2].
# Parallel sharding (5 jobs, each runs 1/5 of tests)
test_sharded:
stage: test
parallel: 5
script:
- npm test -- --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
# Matrix build (6 jobs: 3 Node versions × 2 OS)
test_matrix:
stage: test
parallel:
matrix:
- NODE_VERSION: ["18", "20", "22"]
OS: ["ubuntu", "alpine"]
image: node:$NODE_VERSION
script:
- npm test
# Matrix with different database versions
test_db:
parallel:
matrix:
- DB: ["postgres:14", "postgres:16"]
NODE: ["18", "20"]
services:
- $DB
script:
- npm run test:db
# Parallel with artifacts (each instance uploads)
test_parallel:
parallel: 4
script:
- npm test -- --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
artifacts:
when: always
paths:
- test-results-$CI_NODE_INDEX/
expire_in: 1 day
# Parallelism limits:
# - Max 200 permutations per job
# - Requires sufficient runners
# - Queue time increases if runners are scarce
Parallelism Caveats:
- Runner capacity: Parallel jobs queue if runners are unavailable
- Cost: More parallel jobs = more compute minutes (unless they finish faster overall)
- Artifact collisions: Each parallel instance should use a unique artifact name
- Test isolation: Parallel tests must not interfere with each other (use separate databases or namespaces)
Caching Optimization
Cache is the difference between downloading dependencies from the internet in every job and reusing them from the runner. For a Node.js project, node_modules can be hundreds of megabytes; downloading it every time adds minutes to every job. With a correctly keyed cache, the dependencies are downloaded once and reused across pipelines [citation:3].
The key must change when dependencies change. The standard pattern is to key by the lockfile hash: key: { files: [package-lock.json] }. When the lockfile changes, a new cache is created; when it doesn't, the existing cache is reused. This is the correct balance between freshness and speed [citation:3].
Use policy: pull on test jobs that only need to read the cache. They don't generate new cache entries, so uploading is wasted time. Only build jobs that install dependencies should use pull-push. This small change can save seconds per test job, which adds up across many jobs [citation:3][citation:12].
# Correct cache key (invalidates on lockfile change)
default:
cache:
key:
files:
- package-lock.json
paths:
- node_modules/
- .npm/
before_script:
- npm ci --cache .npm --prefer-offline
# Build job: generate cache
build:
cache:
policy: pull-push # Download and upload
script:
- npm run build
# Test job: only read cache
test:
cache:
policy: pull # Download only, no upload
script:
- npm test
# Separate cache for different tools
test_python:
cache:
- key:
files: [requirements.txt]
paths: [.cache/pip/]
- key:
files: [poetry.lock]
paths: [.venv/]
# Cache anti-patterns:
# - Static key: cache never invalidates
# - Caching build outputs (use artifacts)
# - No policy on test jobs
# - Skipping install (always run npm ci)
# Cache vs artifacts:
# Cache: dependencies, reusable across pipelines
# Artifacts: build outputs, passed within pipeline
Cache Hit Rate: Monitor your cache hit rate. If it's below 80%, your key is probably too specific (changing too often) or too generic (never invalidating). Aim for 80-95%.
Resource Groups
Resource groups control concurrency for jobs that shouldn't run simultaneously. The classic use case is deployment: you don't want two pipelines deploying to the same environment at the same time, because the second deployment might overwrite the first while it's still in progress. A resource group ensures that only one job with that resource group runs at a time; other jobs wait in a queue [citation:1].
The process_mode determines the queue order. oldest_first runs jobs in pipeline creation order, which is safer for continuous deployment. newest_first runs the newest pipeline first, preventing outdated deployments from overwriting newer ones — but this requires that all deploy jobs are idempotent [citation:1]. unordered runs whichever job is next in the queue, with no guarantee of order, and is the least safe.
Resource groups can also be applied to parent-child pipelines. If a parent pipeline triggers a child deployment pipeline, the parent's trigger job can have a resource_group, ensuring that the entire child pipeline waits for other deployments to finish before starting. This is essential for environments with strict deployment ordering [citation:1].
# Resource group for deployment concurrency control
deploy_production:
stage: deploy
resource_group: production
script:
- ./deploy.sh production
environment:
name: production
# With process mode (API only)
# PUT /projects/:id/resource_groups/:key
# { "process_mode": "oldest_first" }
# Process modes:
# - unordered (default): no order guarantee
# - oldest_first: pipeline creation order (safer)
# - newest_first: reverse pipeline order (prevent outdated deploys)
# - newest_ready_first: newest ready job (faster than newest_first)
# Resource group for parent-child pipeline
trigger_deploy:
trigger:
include: deploy.gitlab-ci.yml
strategy: depend
resource_group: production
# Resource group does NOT block other jobs
# Only jobs with the same resource_group wait
test:
script: npm test # Runs regardless of deploy queue
# When to use resource groups:
# - Production deployments
# - Database migrations
# - Any job that must not run concurrently
# - Child deployment pipelines
Resource Group Best Practices:
- Use
oldest_first for production deploys (safer ordering)
- Use
newest_first only if all deploy jobs are idempotent
- Apply resource groups to parent-child trigger jobs for deployment pipelines
- Resource groups don't block unrelated jobs — only those with the same group
Monitoring and Analytics
You can't optimize what you don't measure. GitLab's CI/CD Analytics page shows pipeline duration trends, failure rates, and pipeline counts over time. The duration chart shows both median (P50) and worst-case (P95) pipeline times, helping you distinguish between "typical" performance and outliers that might indicate a bottleneck [citation:9].
For job-level insight, GitLab's CI/CD Job Performance Metrics (Premium/Ultimate, beta) shows P50 and P95 duration and failure rate for each job. This is the data you need to identify the slowest jobs and prioritize optimization. A job that takes 15 minutes at P50 and 45 minutes at P95 is a candidate for sharding or caching [citation:4].
For deeper observability, GitLab's pipeline telemetry (experimental) exports traces, metrics, and logs to an OpenTelemetry-compatible backend. This gives you span-level visibility into pipeline execution: which jobs are slow, where the critical path is, and how dependencies affect timing. It's the most detailed view available, but it requires setup and an observability backend [citation:14].
# View CI/CD Analytics
# Project > Analyze > CI/CD Analytics
# - Pipeline duration (P50, P95)
# - Failure rate
# - Pipeline count over time
# Job Performance Metrics (Premium/Ultimate, beta)
# Project > Analyze > CI/CD Analytics > Job Performance
# - P50 duration per job
# - P95 duration per job
# - Failure rate per job
# Sort by duration to find bottlenecks
# Pipeline telemetry (experimental)
# Set CI/CD variable:
# GITLAB_OBSERVABILITY_EXPORT=traces,metrics,logs
# Exports to GitLab Observability backend
# Key metrics to monitor:
# - Pipeline duration (P50 and P95)
# - Job duration (P50 and P95)
# - Queue time (runner availability)
# - Cache hit rate
# - Failure rate (flaky tests)
# - Critical path (longest chain of dependencies)
# Tools:
# - GitLab CI/CD Analytics (built-in)
# - Job Performance Metrics (built-in, beta)
# - GitLab CI Pipelines Exporter (Prometheus)
# - OpenTelemetry export (experimental)
# - Custom dashboards (Grafana, etc.)
What to Measure First:
- Pipeline duration P95: The worst-case is what developers remember
- Job duration P95: Find the slowest jobs and optimize them
- Queue time: If jobs wait, you need more runners
- Cache hit rate: If low, your cache key or policy is wrong
Complete Optimization Example
Here's a pipeline that applies the optimizations from this guide: DAG, parallel sharding, immediate-start linting, caching, and resource groups.
# Optimized pipeline example
stages:
- build
- test
- deploy
default:
image: node:18
cache:
key:
files: [package-lock.json]
paths:
- node_modules/
- .npm/
before_script:
- npm ci --cache .npm --prefer-offline
# Build once
build:
stage: build
script: npm run build
cache:
policy: pull-push
artifacts:
paths: [dist/]
expire_in: 1 day
# Lint: starts immediately, no dependencies
lint:
stage: test
needs: []
cache: []
script: npm run lint
# Test: sharded across 4 parallel jobs
test_unit:
stage: test
parallel: 4
needs: [build]
cache:
policy: pull
script: npm test -- --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
artifacts:
when: always
paths: [test-results-$CI_NODE_INDEX/]
expire_in: 1 day
# Integration test: depends on build
test_integration:
stage: test
needs: [build]
cache:
policy: pull
script: npm run test:integration
# Deploy: resource group prevents concurrent deploys
deploy_staging:
stage: deploy
resource_group: staging
needs: [test_unit, test_integration]
script: ./deploy.sh staging
environment:
name: staging
deploy_production:
stage: deploy
resource_group: production
needs: [test_unit, test_integration]
script: ./deploy.sh production
environment:
name: production
when: manual
# Optimizations applied:
# - needs: DAG (test starts after build, not after all builds)
# - parallel: 4 test shards (test suite 4x faster)
# - needs: [] on lint (runs immediately)
# - cache pull-push on build, pull on test
# - resource_group on deploys (serialized)
# - artifacts expire in 1 day (less storage)
Troubleshooting Pipeline Performance
# Pipeline still slow despite needs
# - Check the critical path (longest chain of dependencies)
# - A fast test still waits for a slow build if it needs it
# - Optimize the slowest job in the critical path
# Parallel jobs queueing
# - Check runner availability
# - Scale runners (more instances or larger machines)
# - Reduce parallelism if queue time exceeds job time
# Cache not hitting
# - Check cache key changes on every run
# - Key by lockfile hash (not CI_COMMIT_SHA)
# - Check cache policy (pull-push on build, pull on test)
# needs: "job does not exist"
# - Use optional: true for jobs that may be skipped by rules
# - Check job name spelling
# - Check rules that might skip the needed job
# Resource group queue too long
# - Check process_mode (oldest_first vs newest_first)
# - Check if all deploy jobs need the same resource group
# - Consider separate resource groups for staging/production
# Job performance metrics not available
# - Requires Premium/Ultimate
# - Requires ClickHouse (self-managed)
# - Beta feature, may not be enabled
# Debug: check pipeline duration
# CI/CD Analytics > Duration chart
# Look for P50 vs P95 gap
# Debug: check job duration
# Pipeline > Jobs > (job) > Duration
# Or CI/CD Analytics > Job Performance (if available)
# Debug: check cache
script:
- ls -la node_modules/ 2>/dev/null || echo "No cache"
- echo "Cache key: $CI_COMMIT_REF_SLUG"
Frequently Asked Questions
What's the single most impactful optimization?
Using needs to create a DAG instead of sequential stages. It lets jobs start as soon as their dependencies finish, often cutting pipeline time significantly without any other changes .
How do I find the bottleneck in my pipeline?
Check CI/CD Analytics for pipeline duration (P50/P95), then look at job-level metrics (if available) or job logs to find the slowest jobs. The critical path — the longest chain of dependencies — is where optimization matters most [citation:9].
Should I use parallel:matrix or separate jobs?
Use parallel:matrix when you need the same job with different variable values (versions, platforms). Use separate jobs when the jobs have different scripts or dependencies. Matrix avoids duplication [citation:3].
How do resource groups differ from needs?
needs controls when a job can start (after its dependencies). resource_group controls whether a job can run concurrently with other jobs in the same group. They solve different problems and can be used together [citation:1].
What's a good cache hit rate?
80-95%. Below 80%, your key changes too often (probably using commit SHA instead of lockfile hash). Above 95%, your key might be too static, risking stale dependencies.
Why are my parallel jobs queueing?
Not enough runners. Parallel jobs require simultaneous runner availability. If you don't have enough runners, jobs queue and the parallelism benefit is lost. Scale runners or reduce parallelism [citation:2].
How do I measure pipeline performance?
Use CI/CD Analytics (pipeline duration, failure rate), Job Performance Metrics (P50/P95 per job, Premium/Ultimate), or export telemetry to an observability backend. Start with Analytics — it's built-in and shows the big picture [citation:9].
Should I optimize for P50 or P95?
Both. P50 is typical performance (what developers expect). P95 is worst-case (what developers remember). A pipeline that's fast at P50 but slow at P95 feels slow because the slow runs are the ones that stick in memory.