CI/CD Best Practices
Pipeline Architecture
Performance
Security
Maintainability
Learn the patterns that separate production-grade GitLab pipelines from fragile prototypes. From pipeline architecture and caching to security hardening and maintainability, build pipelines that are fast, safe, and easy to change.
Optimize
Secure
GitLab CI/CD
Why Best Practices Matter
A GitLab pipeline that works is not the same as a pipeline that's well-designed. The difference shows up over time: pipelines that take 45 minutes to give feedback, configurations that nobody dares to change, secrets that leak into logs, and jobs that rebuild the same artifact five times. These are not failures of GitLab — they're failures of design. The best practices in this guide are the patterns that experienced teams converge on after dealing with these problems firsthand.
The stakes are higher than they appear. A slow pipeline costs developer time every single day. An insecure pipeline can expose credentials or let untrusted code run with production privileges. A monolithic .gitlab-ci.yml becomes a bottleneck where every change requires reviewing a thousand lines of YAML. These costs compound. The time to fix them is before they become entrenched [citation:8].
Best practices are not rigid rules. They're starting points that you adapt to your context. A small team with a simple Node.js app doesn't need parent-child pipelines. A monorepo with twenty services does. The goal is to understand the principles — fast feedback, least privilege, DRY configuration, observable failures — and apply them proportionally. This guide covers the practices that have the highest impact across the most teams.
Core Principle: Optimize for the developer waiting for feedback. Every minute you shave off the pipeline is multiplied by every developer, every day.
Architecture
Choose the right pipeline structure: basic, needs (DAG), parent-child, or multi-project. Match complexity to need.
Performance
Cache dependencies, parallelize jobs, use needs, shard tests, and build once. Target sub-10-minute feedback.
Security
Pin includes to SHAs, scope CI_JOB_TOKEN, use OIDC, protect variables, and isolate runners.
Maintainability
Use default, extends, YAML anchors, and includes to eliminate duplication. Split large files.
Observability
Use interruptible, retry, and allow_failure deliberately. Make failures visible and actionable.
Anti-Patterns
Avoid sequential-only pipelines, rebuilding artifacts, hardcoded secrets, and monolithic YAML.
Pipeline Architecture
GitLab supports several pipeline architectures, and choosing the right one is the first and most important decision. The basic pipeline — stages running sequentially, jobs within a stage running in parallel — is simple and works well for small projects. But as projects grow, the sequential stage model becomes a bottleneck: every job in the build stage must finish before any test job starts, even if only one test needs a specific build artifact [citation:1].
The needs keyword solves this by creating a directed acyclic graph (DAG). Instead of waiting for an entire stage to complete, a job can start as soon as its specific dependencies finish. In a pipeline where build_a is fast and build_b is slow, test_a can start while build_b is still running, and deploy_a can start before build_b even finishes. This is the single biggest performance improvement for most pipelines [citation:1].
For large monorepos or complex multi-service applications, parent-child pipelines and multi-project pipelines provide further decomposition. A parent pipeline can trigger child pipelines for independent components, each with its own stages and jobs, running in parallel. This keeps configuration manageable and allows each team to own their pipeline without editing a shared file [citation:11]. The key is to match the architecture to the project's complexity — don't use parent-child pipelines for a simple app, and don't use a basic pipeline for a monorepo with twenty services.
# Basic pipeline (sequential stages)
stages:
- build
- test
- deploy
# All build jobs finish before any test job starts
# Simple, but can be slow
# Pipeline with needs (DAG)
build_a:
stage: build
script: echo "Fast build"
build_b:
stage: build
script: echo "Slow build"
test_a:
stage: test
needs: [build_a] # Starts as soon as build_a finishes
script: echo "Test A"
test_b:
stage: test
needs: [build_b] # Starts as soon as build_b finishes
script: echo "Test B"
deploy_a:
stage: deploy
needs: [test_a] # Starts as soon as test_a finishes
script: echo "Deploy A"
# Parent-child pipeline
trigger_frontend:
trigger:
include: frontend/.gitlab-ci.yml
strategy: depend
trigger_backend:
trigger:
include: backend/.gitlab-ci.yml
strategy: depend
# Multi-project pipeline
trigger_deploy:
trigger:
project: my-group/deployment
branch: main
strategy: depend
# Choosing an architecture:
# - Simple project: basic pipeline
# - Complex project with dependencies: needs (DAG)
# - Monorepo with independent components: parent-child
# - Microservices across repos: multi-project
Architecture Best Practices:
- Start with basic; add
needs when you have real dependencies to optimize
- Use parent-child for monorepos to give teams ownership of their pipelines
- Use multi-project for microservices to avoid duplicating deploy logic
- Don't over-engineer — a simple pipeline is easier to debug
- Keep child pipelines focused; each should handle one component
Caching and Artifacts
Caching is the most impactful performance optimization for most pipelines, but it's also the most commonly misconfigured. The rule is simple: cache what's expensive to fetch and doesn't change often — node_modules, .m2, .cache/pip, .npm. Key the cache on your lockfile hash so it invalidates when dependencies change. Use pull-push for build jobs that generate cache, and pull for test jobs that only read it [citation:3].
A common mistake is caching too much or keying incorrectly. If the cache key doesn't change when dependencies change, you ship stale packages. If you cache test outputs or build artifacts, you're misusing cache — that's what artifacts are for. Cache is an optimization; the job must work without it. Always run the actual install command (npm ci, pip install) even if the cache is present [citation:8].
Artifacts are for passing outputs between jobs: build artifacts (dist/), test reports (JUnit XML), coverage reports. Use expire_in aggressively — a build artifact that's only needed for the current pipeline should expire in a day, not the default 30 days. Use dependencies to limit which jobs download which artifacts. Downloading every artifact in every job is a hidden performance cost [citation:13].
# Correct caching: keyed by lockfile, pull-only for tests
default:
cache:
key:
files:
- package-lock.json
paths:
- node_modules/
- .npm/
before_script:
- npm ci --cache .npm --prefer-offline
build:
stage: build
script:
- npm run build
cache:
policy: pull-push
artifacts:
paths:
- dist/
expire_in: 1 day
test:
stage: test
script:
- npm test
cache:
policy: pull # Only read cache, don't write
dependencies:
- build # Only download build artifacts
artifacts:
when: always
reports:
junit: test-results.xml
expire_in: 1 week
# Wrong: no lockfile key, cache never invalidates
# cache:
# key: node-cache
# paths:
# - node_modules/
# Wrong: caching build outputs
# cache:
# paths:
# - dist/ # This should be an artifact
# Wrong: downloading all artifacts
# deploy:
# dependencies: [] # Prevents downloading everything
Caching Anti-Patterns:
- Static cache key: Cache never invalidates; you ship stale dependencies
- Caching build outputs: Use artifacts for outputs, cache for dependencies
- No policy on test jobs: Test jobs upload cache unnecessarily, wasting time
- Skipping install: Assuming cache means you can skip
npm ci — always run it
Security Hardening
GitLab CI is a privileged system. Every job gets a CI_JOB_TOKEN, inherits your CI/CD variables, and can pull configuration from other repositories. That's exactly what supply chain attackers exploit. The 2025 tj-actions attack (CVE-2025-30066) demonstrated this: a mutable reference in a GitHub Actions workflow let malicious code reach roughly 23,000 repositories. GitLab pipelines have the same trust boundaries [citation:4].
The most important hardening step is to pin includes and components to a commit SHA, never a branch or tag. If you include: project with ref: main, whoever controls that branch controls your pipeline. Pin to a SHA, and treat the templates repository as production code — require reviews and protect its default branch [citation:4][citation:14]. The second step is to scope the CI_JOB_TOKEN: enable the allowlist and explicitly permit only the projects that need to call in. This turns a cross-project pivot from trivial into blocked [citation:4].
For cloud credentials, stop storing long-lived keys. Use OIDC (id_tokens) to exchange a short-lived JWT for temporary cloud credentials. Scope the trust policy to the specific project, branch, and environment — a permissive trust policy defeats the entire purpose. Combine this with protected variables (only available on protected branches) and protected environments (only authorized users can deploy) for defense in depth [citation:9][citation:14].
# Pin includes to a SHA (not a branch)
include:
- project: 'security/ci-templates'
ref: 'a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0'
file: '/templates/scan.yml'
# Scope CI_JOB_TOKEN access (settings)
# Settings > CI/CD > Token Access
# - Limit access to this project: ✓
# - Allowlist only projects that need access
# OIDC for cloud credentials (no static keys)
deploy:
id_tokens:
AWS_ID_TOKEN:
aud: https://gitlab.example.com
script:
- aws sts assume-role-with-web-identity
--role-arn arn:aws:iam::123456789012:role/gitlab-deploy
--web-identity-token $AWS_ID_TOKEN
# Protected variables (only on protected branches)
# Settings > CI/CD > Variables
# - DATABASE_PASSWORD
# Protected: ✓
# Masked: ✓
# Environment scope: production
# Protected environments
# Settings > CI/CD > Protected environments
# - production: Maintainers can deploy, approvals required
# Isolate runners by trust level
build_image:
tags:
- trusted-docker # Only runners with this tag
script:
- buildah bud -t $IMAGE . # Rootless build
# Security checklist:
# - Pin all includes to SHAs
# - Scope CI_JOB_TOKEN
# - Use OIDC for cloud credentials
# - Protect and mask all sensitive variables
# - Use protected environments for production
# - Isolate runners by trust level
Security Hardening Checklist:
- Pin every include to a commit SHA, never a branch
- Enable CI_JOB_TOKEN allowlist and restrict access
- Replace static cloud keys with OIDC
id_tokens
- Mark all sensitive variables as protected and masked
- Use protected environments for production deployments
- Tag runners by trust level; never run untrusted jobs on privileged runners
Maintainability and DRY Configuration
Duplication is the enemy of maintainable pipelines. If you copy the same before_script or cache configuration into ten jobs, changing it means editing ten places. GitLab provides several mechanisms to eliminate duplication: default for global settings, extends for inheritance, YAML anchors for mapping reuse, and include for splitting files. The key is to use them intentionally, not to over-abstract [citation:10].
The default keyword is the simplest: set image, before_script, cache, and retry globally, and they apply to all jobs. Override per job when needed. For job templates, use hidden jobs (names starting with a dot) and extends. A .test_base job can define the common test configuration, and each test job extends it. This keeps changes localized — update the base job once, and all tests inherit the change [citation:6][citation:10].
For large configurations, split into multiple files and use include: local. A .gitlab/ci/ directory with build.yml, test.yml, deploy.yml, and security.yml is easier to navigate than a 500-line monolith. Each file has a single responsibility, and teams can own their respective files. The root .gitlab-ci.yml becomes a short orchestration file that includes the rest [citation:11].
# Use default for global settings
default:
image: node:18
before_script:
- npm ci
cache:
key:
files:
- package-lock.json
paths:
- node_modules/
retry:
max: 2
when:
- runner_system_failure
- stuck_or_timeout_failure
# Use hidden jobs and extends for shared config
.test_base:
stage: test
artifacts:
when: always
reports:
junit: test-results.xml
cache:
policy: pull
test_unit:
extends: .test_base
script:
- npm run test:unit
test_integration:
extends: .test_base
dependencies:
- build
script:
- npm run test:integration
# Use YAML anchors for mapping reuse
.node_config: &node_config
image: node:18
before_script:
- npm ci
cache:
key:
files:
- package-lock.json
paths:
- node_modules/
build:
<<: *node_config
stage: build
script:
- npm run build
lint:
<<: *node_config
stage: test
script:
- npm run lint
# Split into multiple files
include:
- local: '/ci/build.yml'
- local: '/ci/test.yml'
- local: '/ci/deploy.yml'
- local: '/ci/security.yml'
# Result: root file is small, each file has one responsibility
DRY vs Over-Abstraction:
- DRY: Eliminate duplication of identical configuration
- Over-abstraction: Hiding complexity so deep that debugging is impossible
- Rule: Two levels of inheritance is good. Five levels is a debugging nightmare.
- Clarity beats cleverness: A slightly duplicated pipeline that's easy to understand is better than a DRY pipeline that nobody can trace
Performance Optimization
The target for pull request feedback is under 10 minutes. Beyond that, developers context-switch, and the pipeline becomes a blocker rather than a tool. The biggest wins come from four places: caching dependencies (covered above), parallelizing independent jobs, sharding slow test suites, and using needs instead of sequential stages [citation:8].
Sharding is often overlooked. A test suite that takes 20 minutes can be split across 5 parallel jobs, each running 4 minutes. GitLab's parallel keyword makes this easy: set parallel: 5, and each instance gets $CI_NODE_INDEX and $CI_NODE_TOTAL. The test runner splits tests by index. For matrix builds (testing against multiple language versions or databases), parallel:matrix generates all combinations from a single job definition [citation:3].
Another common waste is rebuilding artifacts. If your build job creates a Docker image or a compiled binary, don't rebuild it in each subsequent stage. Build once, push to the registry, and have downstream jobs pull the same artifact. This eliminates whole stages and ensures that what you test is what you deploy. Use commit SHA tags, not latest, so the artifact is immutable [citation:8].
# Parallelize test shards
test:
parallel: 5
script:
- npm test -- --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
# Matrix builds (multiple versions)
test:
parallel:
matrix:
- NODE_VERSION: [16, 18, 20]
OS: [ubuntu, alpine]
image: node:$NODE_VERSION
script:
- npm test
# Creates 6 jobs
# Build once, promote artifact
build:
stage: build
script:
- docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA .
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
deploy_staging:
stage: deploy
script:
- kubectl set image deployment/app app=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
# Pulls the same image, no rebuild
deploy_production:
stage: deploy
script:
- kubectl set image deployment/app app=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
# Pulls the same image, no rebuild
# Use interruptible for jobs that can be safely cancelled
test:
interruptible: true
script:
- npm test
# Performance checklist:
# - Cache dependencies with lockfile key
# - Use needs for job dependencies
# - Shard slow test suites
# - Use parallel:matrix for version matrices
# - Build once, promote artifact
# - Mark jobs interruptible when safe
Performance Targets:
- PR feedback: Under 10 minutes
- Main branch pipeline: Under 30 minutes
- Cache hit rate: Above 80% for dependencies
- Queue time: Near zero (scale runners if needed)
Common Anti-Patterns
Anti-patterns are practices that seem reasonable but cause problems at scale. The most common is the monolithic .gitlab-ci.yml — a single file that grows to hundreds or thousands of lines. It starts simple and accumulates complexity: every team adds jobs, every feature adds stages, and eventually nobody understands the whole thing. The fix is to split into included files by responsibility, as covered in the maintainability section [citation:11].
Another common anti-pattern is running everything sequentially. The default stage model runs all build jobs, then all test jobs, then all deploy jobs. This is simple but wasteful: a fast test that only needs a fast build waits for a slow build to finish. Use needs to express the actual dependencies and let jobs start as soon as their inputs are ready [citation:1].
Hardcoding secrets is the most dangerous anti-pattern. A secret in .gitlab-ci.yml is committed to version control, visible to anyone with repository access, and persists in git history forever. Even if you delete it later, it's still in the history. Use masked and protected CI/CD variables, or a secrets manager. Never commit a secret, even to a private repository [citation:9].
# Anti-pattern: Monolithic .gitlab-ci.yml
# 1000+ lines, all jobs in one file
# Fix: Split into included files
# Anti-pattern: Sequential-only pipeline
# All build jobs finish before any test job starts
# Fix: Use needs
# Anti-pattern: Hardcoded secrets
# variables:
# DATABASE_PASSWORD: "supersecret123" # NEVER do this
# Fix: Use masked/protected variables
# Anti-pattern: Rebuilding artifacts
# build_image:
# script: docker build -t app:$CI_COMMIT_SHA .
# test:
# script: docker build -t app:$CI_COMMIT_SHA . # Rebuild!
# Fix: Build once, pull artifact
# Anti-pattern: Caching everything
# cache:
# paths:
# - node_modules/
# - dist/
# - .env
# Fix: Cache dependencies only, use artifacts for outputs
# Anti-pattern: No retry on flaky jobs
# Fix: Use retry for transient failures
flaky_test:
retry:
max: 2
when:
- runner_system_failure
- stuck_or_timeout_failure
# Anti-pattern: allow_failure on critical jobs
# deploy_production:
# allow_failure: true # NEVER do this
# Fix: Only allow_failure on experimental jobs
Anti-Pattern Checklist:
- Monolithic YAML that nobody understands
- Sequential-only pipeline that wastes time
- Secrets hardcoded in YAML
- Rebuilding the same artifact in multiple stages
- Caching build outputs instead of using artifacts
- No retry on flaky jobs
allow_failure: true on production deploys
Complete Best-Practices Pipeline
Here's a complete .gitlab-ci.yml that demonstrates the best practices covered in this guide: DAG, caching, DRY configuration, security, and performance.
# Complete best-practices pipeline
stages:
- build
- test
- deploy
# Global defaults
default:
image: node:18
before_script:
- npm ci --cache .npm --prefer-offline
cache:
key:
files:
- package-lock.json
paths:
- node_modules/
- .npm/
retry:
max: 2
when:
- runner_system_failure
- stuck_or_timeout_failure
interruptible: true
# Build once
build:
stage: build
script:
- npm run build
cache:
policy: pull-push
artifacts:
paths:
- dist/
expire_in: 1 day
# Test jobs use needs for DAG
test_unit:
stage: test
script:
- npm run test:unit
needs: [build]
cache:
policy: pull
artifacts:
when: always
reports:
junit: test-results/unit.xml
expire_in: 1 week
test_integration:
stage: test
script:
- npm run test:integration
needs: [build]
cache:
policy: pull
dependencies:
- build
artifacts:
when: always
reports:
junit: test-results/integration.xml
expire_in: 1 week
lint:
stage: test
script:
- npm run lint
needs: []
cache: []
interruptible: true
# Security scan
sast:
stage: test
include:
- template: Security/SAST.gitlab-ci.yml
needs: []
# Deploy staging (automatic)
deploy_staging:
stage: deploy
script:
- ./deploy.sh staging
needs: [test_unit, test_integration]
environment:
name: staging
url: https://staging.example.com
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
# Deploy production (manual, protected)
deploy_production:
stage: deploy
script:
- ./deploy.sh production
needs: [test_unit, test_integration]
environment:
name: production
url: https://example.com
rules:
- if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
when: manual
interruptible: false
# Result:
# - Build once, promote artifact
# - Tests use needs (DAG), start as soon as build finishes
# - Lint runs immediately (no dependencies)
# - SAST runs in parallel
# - Staging auto-deploys, production manual
# - Cache keyed by lockfile
# - Retry on transient failures
# - Interruptible where safe
Frequently Asked Questions
What's the single most impactful best practice?
Use needs instead of sequential stages. It transforms a pipeline from "wait for everything" to "start as soon as your inputs are ready," often cutting pipeline time in half without any other changes [citation:1].
How do I cache correctly?
Cache dependencies (node_modules, .m2, pip cache), key by lockfile hash, use pull-push for build jobs and pull for test jobs. Always run the install command even if cache exists — cache is an optimization, not a guarantee [citation:3].
Should I use parent-child pipelines?
Only if you have a monorepo with independent components. For a single app, a basic pipeline with needs is simpler and sufficient. Parent-child adds complexity that isn't justified for small projects [citation:11].
How do I secure my pipeline?
Pin includes to SHAs, scope CI_JOB_TOKEN, use OIDC instead of static cloud keys, mark variables as protected and masked, and use protected environments for production. Treat .gitlab-ci.yml as privileged code [citation:4][citation:9].
How do I reduce duplication without over-abstracting?
Use default for global settings, hidden jobs and extends for job templates, and split into included files by responsibility. Two levels of inheritance is good. Five levels makes debugging impossible [citation:10].
What's the target pipeline duration?
Under 10 minutes for merge request feedback. Beyond that, developers context-switch and the pipeline becomes a blocker. Profile, cache, parallelize, and use needs to get under 10 minutes [citation:8].
What's the most dangerous anti-pattern?
Hardcoded secrets. A secret in .gitlab-ci.yml is committed to git history forever. Even deleting it later doesn't remove it from history. Use masked/protected variables or a secrets manager [citation:9].
How do I know if my pipeline is slow?
Check the job trace timing view. Look for long queue times (need more runners), long dependency resolution (cache not hitting), and sequential stages that could use needs. Profile before optimizing [citation:8].