Matrix & Parallel Jobs
Parallel Execution
Matrix Builds
Dynamic Dependencies
Learn how to use GitLab CI/CD parallel and matrix jobs to run multi-variant builds efficiently, create dynamic job dependencies with matrix expressions, and avoid massive YAML duplication.
Parallel
Matrix
GitLab CI/CD
Why Parallel and Matrix Jobs?
As applications grow, so does the testing matrix. You may need to test against multiple operating systems, language versions, database versions, or architecture targets. Writing a separate job for every combination quickly becomes unmanageable — a matrix of 3 operating systems × 4 language versions × 2 architectures means 24 nearly identical job definitions, each differing only in a few variables. This duplication is error-prone to maintain and makes the .gitlab-ci.yml file bloated and hard to read.
GitLab CI/CD solves this with two related features: parallel and parallel:matrix. The parallel keyword simply runs the same job multiple times, splitting work by index — useful for sharding a large test suite. The parallel:matrix keyword is more powerful: it generates multiple jobs from a single definition, each with a different combination of variable values, allowing you to test across a full matrix of configurations without repeating yourself [citation:7][citation:9].
Matrix jobs also introduce a challenge: dependencies. If your build stage creates 24 image variants, your deploy stage needs to consume the matching artifact from each build, not all 24 at once. GitLab solves this with matrix expressions ($[[ matrix.IDENTIFIER ]]), which create 1:1 mappings between matrix jobs across stages, ensuring each downstream job depends only on its corresponding upstream job [citation:2][citation:4].
Core Principle: Use parallel:matrix when you need the same job logic with different variable combinations. Use matrix expressions in needs to wire up matching dependencies between stages.
parallel
Runs the same job N times. Each instance gets $CI_NODE_INDEX and $CI_NODE_TOTAL for sharding. No variable variation.
parallel:matrix
Generates jobs from a matrix of variable values. Each combination becomes a separate job with its own variable values injected.
Matrix Expressions
$[[ matrix.IDENTIFIER ]] creates 1:1 dependencies between matrix jobs in different stages. GitLab 18.6+ (Beta).
Job Limits
Matrix permutations cannot exceed 200 per job. Total jobs in active pipelines are also limited at the instance level.
Using the parallel Keyword
The parallel keyword is the simpler of the two features. When you set parallel: 5 on a job, GitLab creates five identical instances of that job, named job_name 1/5 through job_name 5/5. Each instance runs in parallel, and GitLab injects two environment variables: $CI_NODE_INDEX (the instance number, starting from 1) and $CI_NODE_TOTAL (the total number of instances) [citation:9].
This is primarily useful for sharding — splitting a large test suite across multiple runners. For example, if you have 1000 tests and 5 parallel jobs, each job can run 200 tests, reducing wall-clock time by a factor of five (assuming enough runners are available). The test framework must support sharding by index, which most modern test runners do.
Note that parallel alone does not vary configuration between instances. All five jobs use the same image, same script, and same variables (except the built-in index variables). If you need different configurations, you need parallel:matrix.
# Basic parallel: run 5 identical jobs
test:
stage: test
parallel: 5
script:
- echo "Running test shard $CI_NODE_INDEX of $CI_NODE_TOTAL"
- npm run test:shard-$CI_NODE_INDEX
# Sharding a test suite
test_sharded:
stage: test
parallel: 4
script:
- echo "Shard $CI_NODE_INDEX/$CI_NODE_TOTAL"
# Test framework splits tests based on node index
- pytest --shard-id=$CI_NODE_INDEX --num-shards=$CI_NODE_TOTAL
# Parallel with artifacts (each instance uploads)
test_parallel:
stage: test
parallel: 3
script:
- npm test -- --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
artifacts:
when: always
paths:
- test-results-$CI_NODE_INDEX/
expire_in: 1 day
# Important: parallel creates identical jobs.
# For different configurations, use parallel:matrix.
Parallel Job Limits:
- Jobs created by
parallel count toward your instance's job limit
- If you don't have enough runners, excess jobs are queued as
pending
- Too many parallel jobs can cause
job_activity_limit_exceeded errors [citation:9]
- Start with a modest number (3-5) and scale up based on runner capacity
Using parallel:matrix
parallel:matrix is where the real power lies. Instead of running identical jobs, it generates a job for each combination of variable values you specify. This is the GitLab equivalent of GitHub Actions' matrix strategy, and it's essential for testing across multiple environments without duplicating YAML [citation:7].
The matrix is defined as a list of dictionaries. Each dictionary can contain one or more variables, each with a list of values. GitLab computes the Cartesian product of all values across all dictionaries in the list. For example, if you define OS: [ubuntu, alpine] and NODE: [16, 18, 20] in a single dictionary, you get 6 jobs (2 × 3). If you define them in separate dictionaries, you also get 6 jobs — the matrix always expands all combinations [citation:4].
Each generated job receives the variable values as environment variables, so your script can reference $OS and $NODE directly. This makes the single job definition reusable across all combinations, which is the core benefit: write once, run everywhere.
# Basic matrix: 2 OS × 3 Node versions = 6 jobs
test:
stage: test
parallel:
matrix:
- OS: [ubuntu, alpine]
NODE: [16, 18, 20]
script:
- echo "Testing on $OS with Node $NODE"
- npm test
# Matrix with single variable (3 jobs)
build:
stage: build
parallel:
matrix:
- ARCH: [amd64, arm64, s390x]
script:
- echo "Building for $ARCH"
- ./build.sh --arch=$ARCH
# Matrix with multiple dictionaries (Cartesian product)
deploy:
stage: deploy
parallel:
matrix:
- ENV: [staging, production]
- REGION: [us-east, eu-west]
script:
- echo "Deploying to $ENV in $REGION"
# Creates 4 jobs: staging/us-east, staging/eu-west,
# production/us-east, production/eu-west
# Using matrix variables in image names
test_versions:
stage: test
image: node:$NODE_VERSION
parallel:
matrix:
- NODE_VERSION: ["16", "18", "20"]
script:
- node --version
- npm test
# Matrix with YAML anchors for reuse
.build_matrix: &build_matrix
parallel:
matrix:
- OS: ["ubuntu", "alpine"]
ARCH: ["amd64", "arm64"]
VARIANT: ["slim", "full"]
compile:
stage: build
<<: *build_matrix
script:
- echo "Compiling for $OS-$ARCH-$VARIANT"
# Creates 8 jobs (2 × 2 × 2)
# Matrix limits:
# - Max 200 permutations per job
# - Variable names: letters, numbers, underscores only
# - Values must be strings or arrays of strings
Matrix Best Practices:
- Use YAML anchors (
&matrix and <<: *matrix) to reuse matrix definitions across jobs
- Keep permutations under 20-30 for reasonable pipeline sizes
- Use descriptive variable names that are meaningful in logs (
$OS, not $V1)
- Combine with
rules to run full matrix only on main/release branches
- Remember: matrix jobs run in parallel, so ensure you have enough runners
Matrix Expressions: 1:1 Dependencies
When you use parallel:matrix across multiple stages, a problem emerges: how do you wire up dependencies correctly? If your build stage creates 8 image variants and your deploy stage also has 8 variants, you want each deploy job to depend on its matching build job — not on all 8 build jobs collectively. Without matrix expressions, needs: [build] would make every deploy job wait for every build job, and artifact downloads would collide [citation:17].
Matrix expressions solve this with the $[[ matrix.IDENTIFIER ]] syntax, introduced in GitLab 18.6 (Beta). Inside a needs:parallel:matrix block, you can reference the current job's matrix values to create a 1:1 mapping. GitLab dynamically resolves the dependency to the job with the same matrix combination, so deploy: [aws, monitoring] depends only on build: [aws, monitoring] [citation:2][citation:4].
This is a compile-time feature: identifiers are resolved when the pipeline is created, not during execution. It only supports string replacement — no complex logic or CI/CD variable references inside the expression. The identifiers must come from the current job's own matrix configuration; you cannot reference another job's matrix values [citation:2].
# Without matrix expressions: deploy depends on ALL builds
build:
stage: build
parallel:
matrix:
- OS: [ubuntu, alpine]
NODE: [16, 18, 20]
script: ./build.sh
deploy_bad:
stage: deploy
needs: [build] # Waits for all 6 build jobs
script: ./deploy.sh
# Problem: deploy has no matrix, but downloads all artifacts
# With matrix expressions: 1:1 mapping
build:
stage: build
parallel:
matrix:
- OS: [ubuntu, alpine]
NODE: [16, 18, 20]
script:
- echo "Building $OS-$NODE"
artifacts:
paths:
- dist/
deploy:
stage: deploy
parallel:
matrix:
- OS: [ubuntu, alpine]
NODE: [16, 18, 20]
needs:
- job: build
parallel:
matrix:
- OS: ['$[[ matrix.OS ]]']
NODE: ['$[[ matrix.NODE ]]']
script:
- echo "Deploying $OS-$NODE"
- ./deploy.sh dist/
# Each deploy job depends on its matching build job only
# YAML anchors to reuse matrix configuration
.matrix: &matrix
parallel:
matrix:
- OS: ["ubuntu", "alpine"]
ARCH: ["amd64", "arm64"]
compile:
stage: build
<<: *matrix
script: echo "Compiling $OS-$ARCH"
test:
stage: test
<<: *matrix
needs:
- job: compile
parallel:
matrix:
- OS: ['$[[ matrix.OS ]]']
ARCH: ['$[[ matrix.ARCH ]]']
script: echo "Testing $OS-$ARCH"
# Selective subset: depend only on specific combinations
prepare:
stage: prepare
parallel:
matrix:
- PLATFORM: [linux, windows, macos]
VERSION: ["16", "18", "20"]
script: echo "Preparing $PLATFORM-$VERSION"
build:
stage: build
parallel:
matrix:
- PLATFORM: [linux, windows, macos]
needs:
- job: prepare
parallel:
matrix:
- PLATFORM: ['$[[ matrix.PLATFORM ]]']
VERSION: ["18"] # Only depend on Node 18
script: echo "Building $PLATFORM"
# Each build depends on the matching prepare job
# but only the VERSION=18 variant (not 16 or 20)
# Matrix expression limitations:
# - GitLab 18.6+ (Beta)
# - Compile-time resolution only
# - String replacement only (no logic)
# - Only current job's matrix identifiers
# - Cannot reference CI/CD variables in expressions
Complete Matrix Pipeline Example
Here's a complete pipeline that builds and tests a Node.js application across multiple operating systems and Node versions, with proper 1:1 dependencies using matrix expressions.
# Complete matrix pipeline example
stages:
- build
- test
- deploy
# Reusable matrix definition
.build_matrix: &build_matrix
parallel:
matrix:
- OS: ["ubuntu", "alpine"]
NODE: ["18", "20"]
# Build: 2 OS × 2 Node versions = 4 jobs
build:
stage: build
image: node:$NODE
<<: *build_matrix
script:
- echo "Building for $OS with Node $NODE"
- npm ci
- npm run build
artifacts:
paths:
- dist/
expire_in: 1 day
name: "build-$OS-$NODE"
# Test: same matrix, 1:1 dependency on build
test:
stage: test
image: node:$NODE
<<: *build_matrix
needs:
- job: build
parallel:
matrix:
- OS: ['$[[ matrix.OS ]]']
NODE: ['$[[ matrix.NODE ]]']
script:
- echo "Testing $OS with Node $NODE"
- npm test
artifacts:
when: always
reports:
junit: test-results.xml
# Deploy: same matrix, 1:1 dependency on test
deploy:
stage: deploy
image: alpine:latest
<<: *build_matrix
needs:
- job: test
parallel:
matrix:
- OS: ['$[[ matrix.OS ]]']
NODE: ['$[[ matrix.NODE ]]']
script:
- echo "Deploying $OS-$NODE variant"
- ./deploy.sh dist/
environment:
name: "$OS-$NODE"
rules:
- if: $CI_COMMIT_BRANCH == "main"
# This creates:
# - 4 build jobs: ubuntu/18, ubuntu/20, alpine/18, alpine/20
# - 4 test jobs: each depends on its matching build
# - 4 deploy jobs: each depends on its matching test
# Total: 12 jobs, defined in ~40 lines of YAML
Troubleshooting Matrix Jobs
# Matrix job fails with "config should be an array of hashes"
# - Check that matrix is a list of dictionaries
# - Each dictionary must have key: [list of values]
# - No nested arrays or complex structures
# Wrong:
parallel:
matrix:
OS: [ubuntu, alpine] # Missing dash
# Correct:
parallel:
matrix:
- OS: [ubuntu, alpine]
# Too many jobs (job_activity_limit_exceeded)
# - Reduce matrix size
# - Split into multiple pipelines (e.g., only full matrix on main)
# - Use rules to limit matrix based on branch
build:
parallel:
matrix:
- OS: [ubuntu, alpine]
NODE: [16, 18, 20]
rules:
- if: $CI_COMMIT_BRANCH == "main"
- if: $CI_COMMIT_BRANCH != "main"
variables:
# Override to smaller matrix on feature branches
# (Not directly possible; use separate jobs or child pipelines)
# Matrix expressions not working (needs:parallel:matrix)
# - Check GitLab version (18.6+ required)
# - Check syntax: $[[ matrix.VAR ]] (double brackets)
# - Ensure variable names match exactly between jobs
# - Ensure matrix definitions are identical
# Artifact collision (multiple matrix jobs upload same path)
# - Use matrix variables in artifact name
artifacts:
name: "build-$OS-$NODE"
paths:
- dist/
# Dependency downloads all artifacts instead of matching one
# - Without matrix expressions, needs: [build] downloads all
# - Use needs:parallel:matrix for 1:1 mapping
needs:
- job: build
parallel:
matrix:
- OS: ['$[[ matrix.OS ]]']
NODE: ['$[[ matrix.NODE ]]']
# Matrix variable not available in script
# - Check that variable is defined in the matrix
# - Check spelling/case (variables are case-sensitive)
# - Debug by echoing: echo "OS=$OS NODE=$NODE"
Frequently Asked Questions
What's the difference between parallel and parallel:matrix?
parallel: 5 runs 5 identical jobs (same configuration). parallel:matrix runs multiple jobs with different variable values from a matrix of combinations.
How many matrix jobs can I create?
A single job's matrix cannot exceed 200 permutations. Total jobs across all active pipelines are also limited at the GitLab instance level [citation:9].
What are matrix expressions and why do I need them?
Matrix expressions ($[[ matrix.X ]]) create 1:1 dependencies between matrix jobs in different stages. Without them, a downstream job depends on all upstream matrix jobs, causing artifact collisions and longer waits.
Is matrix expressions stable?
As of GitLab 18.6, matrix expressions are in Beta. They work but behavior may change. Test thoroughly in non-production pipelines first.
How do I reuse a matrix definition across multiple jobs?
Use YAML anchors. Define the matrix under a hidden job (e.g., .matrix: &matrix) and merge it with <<: *matrix in each job that needs it.
Can I use matrix variables in artifact names?
Yes. Use the variables directly: name: "build-$OS-$NODE". This prevents collisions when multiple matrix jobs upload artifacts.
How do I limit a matrix to only certain branches?
Use rules at the job level. You can't conditionally change the matrix size with rules, but you can skip the entire job or use separate jobs for full vs. reduced matrices.
What happens if a matrix job fails?
The pipeline fails if any matrix job fails (unless allow_failure: true). Downstream jobs that need the failed job will not run. Matrix expressions help isolate failures to the affected combination.