Caching & Artifacts
Pipeline Speed
Data Passing
Storage Management
Learn how to use GitLab CI/CD cache to speed up pipelines and artifacts to pass build outputs between jobs. Understand the difference, configure keys, and manage storage effectively.
Cache
Artifacts
GitLab CI/CD
Cache vs Artifacts: What's the Difference?
One of the most common sources of confusion in GitLab CI/CD is the distinction between cache and artifacts. Both deal with files, both are configured in .gitlab-ci.yml, and both can make your pipelines faster. But they serve fundamentally different purposes and are stored in different places.
Cache is an optimization. It stores dependencies and other files that don't change often — like node_modules, Maven's .m2, or pip's cache directory. Cache is stored on the GitLab Runner (or in S3 if distributed caching is enabled) and is reused across pipeline runs. If the cache is missing, your job should still work — it just takes longer because dependencies need to be downloaded again. Cache is not guaranteed to be available [citation:1][citation:12].
Artifacts are outputs. They are files that a job produces and that later jobs need — compiled binaries, test reports, coverage data, or deployment packages. Artifacts are uploaded to GitLab and are guaranteed to be available to downstream jobs in the same pipeline. They're also downloadable from the GitLab UI, making them useful for debugging and review. Unlike cache, artifacts are part of the pipeline's contract: if a job declares an artifact, downstream jobs can rely on it being there [citation:3][citation:10].
Simple Rule: Use cache for things that speed up jobs but aren't essential (dependencies). Use artifacts for things that later jobs need (build outputs, reports).
Cache
Stored on Runner. Reused across pipelines. Not guaranteed. Best for: node_modules, pip cache, Maven .m2. Makes jobs faster but jobs must work without it.
Artifacts
Stored in GitLab. Passed within same pipeline. Guaranteed. Best for: dist/, test reports, JAR files. Downstream jobs depend on them.
Expiration
Cache: managed by Runner, can be cleared. Artifacts: default 30 days, controlled by expire_in.
Visibility
Cache: hidden from UI. Artifacts: browsable and downloadable from GitLab UI, useful for debugging.
Configuring Cache
Cache is defined using the cache keyword. You can define it globally (applies to all jobs), per-job, or using YAML anchors for reuse. The most important configuration is the key — it determines which cache is used. A well-designed key ensures that different branches or different dependency versions don't collide.
The simplest cache key is the branch name ($CI_COMMIT_REF_SLUG), which gives each branch its own cache. This prevents a feature branch with experimental dependencies from polluting the cache used by main. A more robust approach uses a hash of your lock file (package-lock.json, go.sum, etc.) as the key, so the cache is automatically invalidated when dependencies change [citation:4][citation:18].
The policy option controls whether a job downloads the cache (pull), uploads it (push), or both (pull-push, the default). A common optimization is to set policy: pull on test jobs — they only need to read dependencies, not write them, which speeds up the job and prevents cache thrashing [citation:8][citation:14].
# Basic cache: cache per branch
cache:
key: "$CI_COMMIT_REF_SLUG"
paths:
- node_modules/
- .npm/
# Cache keyed by lock file (invalidates on dependency change)
cache:
key:
files:
- package-lock.json
paths:
- node_modules/
policy: pull-push
# Job-level cache with pull-only policy
test:
stage: test
cache:
key:
files:
- package-lock.json
paths:
- node_modules/
policy: pull
script:
- npm test
# Global cache with fallback keys
cache:
key:
files:
- package-lock.json
prefix: npm
fallback_keys:
- npm-$CI_DEFAULT_BRANCH
- npm-default
paths:
- .npm/
- node_modules/
policy: pull-push
# Cache for a specific tool (npm cache directory)
default:
image: node:18
cache:
key: $CI_COMMIT_REF_SLUG
paths:
- .npm/
before_script:
- npm ci --cache .npm --prefer-offline
# Multiple caches (list syntax)
job:
cache:
- key: node-$CI_COMMIT_REF_SLUG
paths:
- node_modules/
- key: pip-$CI_COMMIT_REF_SLUG
paths:
- .cache/pip/
# Disable cache for a job
no_cache_job:
cache: []
script:
- echo "No cache used"
# Cache policies:
# pull-push (default): Download and upload cache
# pull: Only download cache (fast, good for test jobs)
# push: Only upload cache (good for build jobs that generate cache)
# Cache key variables (available):
# $CI_COMMIT_REF_SLUG - Branch/tag name, slugified
# $CI_COMMIT_REF_NAME - Branch/tag name
# $CI_JOB_NAME - Job name
# $CI_JOB_STAGE - Stage name
# $CI_DEFAULT_BRANCH - Default branch name
# Important: Cache is stored on the Runner.
# If jobs run on different Runners, cache may not be available.
# Use S3 or GCS for distributed caching if needed.
Cache Gotchas:
- Cache is not guaranteed. Always ensure jobs can run without it (e.g., run
npm ci even if node_modules exists)
- Cache is Runner-specific. If your jobs run on different Runners, they won't share cache unless distributed caching is configured
- Cache and artifacts overlap: If both define the same path, cache is restored first, then artifacts overwrite it [citation:3]
- Protected vs unprotected branches use separate caches by default (security feature)
- Cache key collisions: Different jobs with the same key and overlapping paths can overwrite each other's cache
Configuring Artifacts
Artifacts are declared in a job using the artifacts keyword. The most common configuration is paths, which lists files or directories to upload after the job completes. By default, artifacts are only uploaded when the job succeeds (when: on_success), but you can change this to always to capture logs or reports even on failure — extremely useful for debugging [citation:5][citation:10].
The expire_in setting controls how long artifacts are kept. The default is 30 days, but for build outputs that are only needed during the pipeline, you can set it to 1 hour or 1 day to save storage. For test reports and coverage data, a longer retention (like 1 week) is often appropriate [citation:10].
GitLab has special handling for certain artifact types through reports. For example, JUnit XML reports are parsed and displayed in merge requests, showing test failures directly in the MR widget. Coverage reports (Cobertura) are similarly parsed to display coverage percentages. These report artifacts are not just stored — they're ingested by GitLab to provide rich UI features [citation:5].
# Basic artifacts: upload on success
build:
stage: build
script:
- npm run build
artifacts:
paths:
- dist/
expire_in: 1 week
# Artifacts with reports (JUnit, coverage)
test:
stage: test
script:
- npm test
artifacts:
when: always
reports:
junit: test-results.xml
coverage_report:
coverage_format: cobertura
path: coverage/cobertura-coverage.xml
paths:
- test-results.xml
# Artifacts with exclude (exclude source maps)
build:
stage: build
script:
- npm run build
artifacts:
paths:
- dist/
exclude:
- dist/**/*.map
expire_in: 1 day
# Artifacts with custom name
build:
stage: build
script:
- npm run build
artifacts:
name: "build-$CI_COMMIT_SHORT_SHA"
paths:
- dist/
expire_in: 1 week
# Artifacts on failure (for debugging)
test:
stage: test
script:
- npm test
artifacts:
when: on_failure
paths:
- test-logs/
expire_in: 3 days
# Artifacts always (success or failure)
test:
stage: test
script:
- npm test
artifacts:
when: always
paths:
- test-results/
expire_in: 7 days
# Dependencies: control which artifacts are downloaded
deploy:
stage: deploy
dependencies:
- build
script:
- ./deploy.sh
# Prevent downloading artifacts
lint:
stage: test
dependencies: []
script:
- npm run lint
# Artifacts with access control
build:
stage: build
script:
- npm run build
artifacts:
paths:
- dist/
access: maintainer # Only maintainers can download
# Artifact reports types:
# junit: JUnit XML test results
# coverage_report: Cobertura coverage
# sast: SAST findings
# dast: DAST findings
# codequality: Code quality
# license_management: License scanning
# container_scanning: Container vulnerabilities
# Artifacts are stored in GitLab
# Default expiration: 30 days
# Can be downloaded from UI or API
# Passed to downstream jobs automatically (unless dependencies is set)
Artifact Best Practices:
- Use
when: always for test reports so failures are visible
- Set
expire_in aggressively for build outputs (1 day or less)
- Use
reports for JUnit and coverage to get GitLab UI integration
- Use
exclude to avoid uploading unnecessary files (source maps, etc.)
- Use
dependencies to limit which jobs download which artifacts
- Never put secrets in artifacts — they're downloadable by anyone with access
Common Cache & Artifact Patterns
Different languages and toolchains have different caching needs. Here are the most common patterns for Node.js, Python, Java, and Go.
# Node.js: Cache node_modules and .npm
default:
image: node:18
cache:
key:
files:
- package-lock.json
paths:
- node_modules/
- .npm/
before_script:
- npm ci --cache .npm --prefer-offline
build:
stage: build
script:
- npm run build
artifacts:
paths:
- dist/
expire_in: 1 day
test:
stage: test
cache:
policy: pull # Only read cache, don't write
script:
- npm test
artifacts:
when: always
reports:
junit: test-results.xml
# Python: Cache pip and virtualenv
default:
image: python:3.11
cache:
key:
files:
- requirements.txt
paths:
- .cache/pip/
- venv/
before_script:
- python -m venv venv
- source venv/bin/activate
- pip install -r requirements.txt
# Go: Cache Go modules
default:
image: golang:1.21
cache:
key:
files:
- go.sum
paths:
- .go/pkg/mod/
variables:
GOPATH: $CI_PROJECT_DIR/.go
# Java/Maven: Cache .m2 repository
default:
image: maven:3.9-eclipse-temurin-17
cache:
key:
files:
- pom.xml
paths:
- .m2/repository/
variables:
MAVEN_OPTS: "-Dmaven.repo.local=$CI_PROJECT_DIR/.m2/repository"
# Docker-in-Docker: Cache layers
build:
stage: build
image: docker:24
services:
- docker:24-dind
variables:
DOCKER_DRIVER: overlay2
script:
- docker pull $CI_REGISTRY_IMAGE:latest || true
- docker build --cache-from $CI_REGISTRY_IMAGE:latest -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA .
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
# Multi-job artifact passing
build:
stage: build
script:
- npm run build
artifacts:
paths:
- dist/
test_e2e:
stage: test
dependencies:
- build
script:
- npm run test:e2e # Uses dist/ from build
deploy:
stage: deploy
dependencies:
- build
script:
- ./deploy.sh dist/
Complete Pipeline Example
Here's a complete .gitlab-ci.yml that demonstrates effective caching and artifact usage for a Node.js application.
# Complete .gitlab-ci.yml with caching and artifacts
stages:
- build
- test
- deploy
# Global cache: dependencies, keyed by lock file
cache:
key:
files:
- package-lock.json
paths:
- node_modules/
- .npm/
policy: pull-push
# Default image and setup
default:
image: node:18
before_script:
- npm ci --cache .npm --prefer-offline
# Build job: creates artifacts for later jobs
build:
stage: build
script:
- npm run build
artifacts:
paths:
- dist/
expire_in: 1 day
name: "build-$CI_COMMIT_SHORT_SHA"
# Unit test: pull-only cache, JUnit reports
test_unit:
stage: test
cache:
policy: pull
script:
- npm run test:unit
artifacts:
when: always
reports:
junit: test-results/unit.xml
paths:
- test-results/
expire_in: 1 week
# Integration test: pull-only cache, depends on build artifacts
test_integration:
stage: test
cache:
policy: pull
dependencies:
- build
script:
- npm run test:integration
artifacts:
when: always
reports:
junit: test-results/integration.xml
paths:
- test-results/
expire_in: 1 week
# Lint: no cache needed, no artifacts
lint:
stage: test
cache: []
dependencies: []
script:
- npm run lint
# Deploy: only needs build artifacts
deploy_staging:
stage: deploy
dependencies:
- build
script:
- echo "Deploying to staging..."
- ./deploy.sh dist/
environment:
name: staging
url: https://staging.example.com
rules:
- if: $CI_COMMIT_BRANCH == "main"
# Deploy production: manual, only build artifacts
deploy_production:
stage: deploy
dependencies:
- build
script:
- echo "Deploying to production..."
- ./deploy.sh dist/
environment:
name: production
url: https://example.com
rules:
- if: $CI_COMMIT_TAG
when: manual
Troubleshooting Cache & Artifacts
# Cache not working (dependencies re-downloaded every time)
# 1. Check cache key - different keys = different caches
# 2. Check if jobs run on different Runners
# 3. Check cache paths - only paths inside project directory
# 4. Check policy - pull vs pull-push
# 5. Check if protected/unprotected branches use separate caches
# Clear cache manually
# In GitLab UI: CI/CD > Pipelines > Clear runner caches
# Change cache key to force new cache
cache:
key: "$CI_COMMIT_REF_SLUG-v2" # Increment version
# Artifacts not available in downstream job
# 1. Check dependencies - does downstream job depend on the job?
# 2. Check artifacts paths - do they match what was uploaded?
# 3. Check when - is artifact uploaded on failure?
# 4. Check if job failed before artifacts were uploaded
# Artifacts expire too quickly
artifacts:
expire_in: 1 week # Increase retention
# Artifacts taking too much storage
# 1. Use exclude to remove unnecessary files
# 2. Set shorter expire_in
# 3. Only upload necessary paths (not entire directories)
artifacts:
paths:
- dist/
exclude:
- dist/**/*.map
- dist/**/*.test.js
expire_in: 1 day
# Cache is corrupted or stale
# 1. Change cache key (forces new cache)
# 2. Clear runner caches via UI
# 3. Check if job needs to regenerate cache
# Debug: list cached/artifact files in job
script:
- ls -la node_modules/ 2>/dev/null || echo "No node_modules"
- ls -la dist/ 2>/dev/null || echo "No dist"
# Cache not shared between jobs
# Ensure same cache key and same Runner
# If using distributed caching, verify S3/GCS config
# Artifacts too large (pipeline slow)
# 1. Exclude unnecessary files
# 2. Use expire_in
# 3. Consider using cache for dependencies instead of artifacts
Frequently Asked Questions
What's the difference between cache and artifacts?
Cache is for speeding up jobs (dependencies), stored on the Runner, not guaranteed. Artifacts are for passing files between jobs (build outputs), stored in GitLab, guaranteed to be available to downstream jobs.
Should I cache node_modules or use npm ci?
Both. Use cache to store node_modules and .npm, but still run npm ci in your script. If the cache is present, npm ci will be fast; if not, it will still work correctly. Never assume cache is available.
How do I pass files from build to test to deploy?
Use artifacts in the build job (e.g., paths: [dist/]). Later jobs automatically download them. Use dependencies: [build] to limit which jobs download which artifacts.
Why is my cache not being used?
Common causes: different cache keys, jobs running on different Runners, cache paths outside the project directory, or the cache was cleared. Check the job log for "Checking cache" and "Successfully extracted cache" messages.
How do I show test results in merge requests?
Use artifacts:reports:junit with your JUnit XML file. GitLab parses it and displays failures directly in the MR widget. Example: artifacts: { reports: { junit: test-results.xml } }.
How long are artifacts kept?
Default is 30 days. Use expire_in to customize (e.g., 1 day, 1 week). You can also set "Keep latest artifacts" in project settings to prevent important artifacts from expiring.
Can I cache files outside the project directory?
No. Cache paths must be relative to the project directory ($CI_PROJECT_DIR). Tools like npm default to caching in your home directory, so you must redirect them (e.g., npm ci --cache .npm).
What's the difference between dependencies and needs?
dependencies controls which artifacts a job downloads (affects data flow). needs creates a DAG (directed acyclic graph) that lets jobs start before their stage is complete (affects execution order).