CI/CD Anti-Patterns
Common Mistakes
Security Risks
Performance Traps
How to Fix
Learn the anti-patterns that turn GitLab pipelines into liabilities. From hardcoded secrets and monolithic YAML to sequential stages and blanket retries, understand what goes wrong and how to fix it.
Anti-Patterns
Fixes
GitLab CI/CD
Why Anti-Patterns Matter
An anti-pattern is a practice that seems reasonable — even obvious — but causes problems as the project grows. The pipeline works at first, so nobody questions it. Then, six months later, the pipeline takes an hour, secrets are scattered across the configuration, and nobody can safely change a job without breaking three others. The technical debt was accumulated in small decisions that felt fine at the time.
Anti-patterns are insidious because they don't announce themselves. A hardcoded secret in .gitlab-ci.yml works. Sequential stages work. A monolithic file works. The problems are latent — they only manifest when the project scales, when a security incident occurs, or when a new developer tries to understand the pipeline and gives up. By then, the cost of fixing them is much higher than the cost of avoiding them .
The good news is that anti-patterns are well-understood. Each one has a known cause, a clear symptom, and a standard fix. This guide covers the anti-patterns that appear most often in real GitLab pipelines: the ones that security researchers write about, the ones that cost teams hours of debugging, the ones that are easy to avoid if you know about them. Fixing them isn't about perfection — it's about eliminating the patterns that cause the most pain.
Core Principle: An anti-pattern is not a mistake you make once — it's a pattern that repeats. Recognizing the pattern is the first step to fixing it.
Security
Hardcoded secrets, mutable include refs, over-privileged tokens, and unprotected variables.
Performance
Sequential stages, rebuilding artifacts, caching everything, and no test sharding.
Structure
Monolithic YAML, deep extends chains, copy-paste configuration, and undocumented jobs.
Reliability
Blanket retries, allow_failure on critical jobs, and hidden failures.
Security Anti-Patterns
The most dangerous anti-pattern is hardcoding secrets in .gitlab-ci.yml. A secret committed to version control is exposed to everyone with repository access, persists in git history forever, and cannot be deleted by simply removing the line. The 2025 tj-actions supply chain attack (CVE-2025-30066) demonstrated how mutable references in CI configuration let attackers reach thousands of repositories — the same risk applies to GitLab includes pinned to branches or tags .
The second most dangerous is an over-privileged CI_JOB_TOKEN. By default, this token can access other projects the user has access to, creating a pivot path for attackers. The fix is to enable the allowlist and explicitly permit only the projects that need to call in. Without this, a compromised pipeline in one project can compromise others .
The third is unprotected sensitive variables. A variable that isn't marked as protected is available to every pipeline, including feature branch pipelines that might run untrusted code. A variable that isn't masked appears in job logs if accidentally echoed. The fix is to mark every sensitive variable as both protected and masked, and to scope it to the environments where it's needed.
# ANTI-PATTERN: Hardcoded secret
# variables:
# DATABASE_PASSWORD: "supersecret123" # NEVER do this
# Fix: Use protected + masked CI/CD variable
# ANTI-PATTERN: Mutable include ref
# include:
# - project: 'security/ci-templates'
# ref: main # Whoever controls main controls your pipeline
# Fix: Pin to commit SHA
include:
- project: 'security/ci-templates'
ref: 'a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0'
file: '/templates/scan.yml'
# ANTI-PATTERN: Unrestricted CI_JOB_TOKEN
# Default: token can access all projects user can access
# Fix: Enable allowlist in Settings > CI/CD > Token Access
# and explicitly allow only projects that need access
# ANTI-PATTERN: Unprotected, unmasked variable
# DATABASE_PASSWORD (unprotected, unmasked)
# Fix: Mark as protected and masked
# Settings > CI/CD > Variables
# - Protected: ✓
# - Masked: ✓
# - Environment scope: production
# ANTI-PATTERN: Secrets in artifacts
# artifacts:
# paths:
# - .env # Contains secrets, downloadable by anyone
# Fix: Never put secrets in artifacts
# ANTI-PATTERN: Long-lived cloud credentials
# variables:
# AWS_SECRET_ACCESS_KEY: $SECRET # Permanent
# Fix: Use OIDC
deploy:
id_tokens:
AWS_ID_TOKEN:
aud: https://gitlab.example.com
script:
- aws sts assume-role-with-web-identity ...
# Security checklist:
# - Never commit secrets
# - Pin includes to SHA
# - Scope CI_JOB_TOKEN
# - Protect and mask variables
# - Use OIDC for cloud credentials
# - Never put secrets in artifacts
Supply Chain Risk:
- CVE-2025-30066 (tj-actions): Mutable references in CI config let attackers reach ~23,000 repositories
- CVE-2025-33053 (foreshadowing): The same risk applies to any CI system that allows mutable includes
- Fix: Pin every include to a commit SHA. Treat templates as production code.
Performance Anti-Patterns
The sequential stage anti-pattern is the most common cause of slow pipelines. In a basic pipeline, all build jobs must finish before any test job starts. This is simple but wasteful: a fast frontend test that only needs a fast frontend build waits for a slow backend build. The fix is to use needs to create a DAG, letting each job start as soon as its actual dependencies finish .
Rebuilding artifacts is another common waste. If the build stage creates a Docker image or a compiled binary, downstream jobs should pull that artifact, not rebuild it. Rebuilding wastes compute, adds time, and — worse — can produce a different artifact than the one that was tested. Build once, promote the same artifact through the pipeline. Tag images by commit SHA, not latest, so the artifact is immutable .
Over-caching is a subtler problem. Caching node_modules is good. Caching build outputs, test results, or .env files is not — those should be artifacts (for outputs) or secrets (for .env). If your cache includes files that change on every commit, the cache never hits, and you pay the upload cost on every job. Cache only what doesn't change often and is expensive to fetch.
# ANTI-PATTERN: Sequential-only pipeline
# All build jobs finish before any test job starts
# Fix: Use needs
build_a:
stage: build
script: ./build-a.sh
build_b:
stage: build
script: ./build-b.sh
test_a:
stage: test
needs: [build_a] # Starts as soon as build_a finishes
script: ./test-a.sh
# ANTI-PATTERN: Rebuilding artifacts
# build:
# script: docker build -t app:$CI_COMMIT_SHA .
# test:
# script: docker build -t app:$CI_COMMIT_SHA . # Rebuild!
# Fix: Build once, promote artifact
build:
stage: build
script:
- docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA .
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
test:
stage: test
script:
- docker pull $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
- ./test.sh
# ANTI-PATTERN: Over-caching
# cache:
# paths:
# - node_modules/
# - dist/ # Should be artifact
# - .env # Should be secret
# - test-results/ # Should be artifact
# Fix: Cache dependencies only
cache:
key:
files: [package-lock.json]
paths:
- node_modules/
- .npm/
# ANTI-PATTERN: No test sharding
# test:
# script: npm test # 20 minutes
# Fix: Shard tests
test:
parallel: 5
script: npm test -- --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL
# Performance checklist:
# - Use needs for DAG
# - Build once, promote artifact
# - Cache dependencies only
# - Shard slow test suites
# - Use parallel:matrix for version matrices
Performance Fixes:
- Sequential stages: Use
needs to create a DAG
- Rebuilding artifacts: Build once, push to registry, pull downstream
- Over-caching: Cache dependencies only; use artifacts for outputs
- Slow tests: Shard with
parallel; use parallel:matrix for version matrices
Structure Anti-Patterns
The monolithic .gitlab-ci.yml is the structure anti-pattern par excellence. It starts small, grows over time, and eventually nobody understands the whole file. Every team adds jobs, every feature adds stages, and the file becomes a bottleneck where any change requires reviewing hundreds of lines. The fix is to split into included files by responsibility: build.yml, test.yml, deploy.yml, security.yml. Each file has a single purpose, and teams can own their respective files .
Deep extends chains are another structural problem. Extending from a base job is good. Extending from a job that extends from a job that extends from a job is a debugging nightmare — nobody can trace where a configuration value comes from. The rule of thumb is two levels of inheritance is fine; three is the practical limit; beyond that, the abstraction cost exceeds the DRY benefit.
Copy-paste configuration is the flip side: duplicating the same before_script or cache block in ten jobs. The fix is default for global settings and hidden jobs with extends for job templates. A single change to the base job propagates to all inheriting jobs, which is exactly what you want — unless you don't, in which case the abstraction was wrong. The key is to choose abstractions that match the actual variance.
# ANTI-PATTERN: Monolithic .gitlab-ci.yml
# (1000+ lines in one file)
# Fix: Split into included files
include:
- local: '/ci/build.yml'
- local: '/ci/test.yml'
- local: '/ci/deploy.yml'
- local: '/ci/security.yml'
# ANTI-PATTERN: Deep extends chain
# .base -> .base_plus -> .base_plus_more -> job
# Tracing which value comes from where is impossible
# Fix: Limit extends depth (2-3 levels max)
# ANTI-PATTERN: Copy-paste configuration
# Every job has the same before_script:
job1:
before_script: [npm ci]
script: npm run test:unit
job2:
before_script: [npm ci]
script: npm run test:integration
# ... repeated 10 times
# Fix: Use default
default:
before_script:
- npm ci
job1:
script: npm run test:unit
job2:
script: npm run test:integration
# ANTI-PATTERN: Hidden jobs with no documentation
# What does .base do? Why does it exist?
# Fix: Comment hidden jobs with purpose
# Base configuration for Node.js test jobs
# Provides: image, cache, before_script
# Extend this for any job that runs tests
.test_base:
image: node:18
before_script: npm ci
# Structure checklist:
# - Split large files by responsibility
# - Limit extends depth
# - Use default for global settings
# - Use hidden jobs with comments
# - Document non-obvious configuration
Abstraction Rule:
- Use DRY when the duplication is truly identical and likely to change together
- Use duplication when the similarity is coincidental and might diverge
- Prefer clarity over cleverness. A pipeline you can debug is worth more than one that's 20 lines shorter.
Reliability Anti-Patterns
Blanket retries are a common anti-pattern. Setting retry: 2 on every job seems safe — if a job fails, try again. But retries hide problems. A test that fails 10% of the time becomes invisible when it's retried, and the root cause is never fixed. The correct use of retry is for transient failures (runner_system_failure, stuck_or_timeout_failure), not for flaky tests or code bugs. Flaky tests should be fixed, not retried .
allow_failure: true on critical jobs is worse. It tells GitLab to ignore the job's failure and continue the pipeline. If you set this on a test or security scan, you're saying "we don't care if this fails" — which is fine for experimental jobs, but dangerous for anything that gates quality. A production deploy with allow_failure: true means a failed deploy doesn't block the pipeline, which is exactly the opposite of what you want.
The last reliability anti-pattern is hiding failures. A job that catches errors, logs them, and exits 0 hides the failure from the pipeline. The job passes, the pipeline is green, and the problem is invisible until it affects users. Exit codes are the pipeline's language — a failure should exit non-zero, so the pipeline can act on it. If you need to log errors and continue, do it in a separate job that runs after the main job fails, not by swallowing the error.
# ANTI-PATTERN: Blanket retry
# retry: 3 # On every job, hides flaky tests
# Fix: Retry only transient failures
job:
retry:
max: 2
when:
- runner_system_failure
- stuck_or_timeout_failure
# ANTI-PATTERN: allow_failure on critical jobs
# deploy_production:
# allow_failure: true # NEVER do this
# Fix: allow_failure only on experimental jobs
experimental_test:
script: npm run test:experimental
allow_failure: true
deploy_production:
script: ./deploy.sh production
# No allow_failure — this must succeed
# ANTI-PATTERN: Hiding failures
# script:
# - ./build.sh || true # Ignores failure!
# - echo "Build done" # Runs even if build failed
# Fix: Let failures propagate
script:
- ./build.sh # Fails the job if build fails
- echo "Build done"
# ANTI-PATTERN: No timeout
# A job can hang indefinitely, blocking the pipeline
# Fix: Set timeout
job:
timeout: 30m
script: ./long-running.sh
# ANTI-PATTERN: No interruptible
# Pipeline doesn't cancel when a new commit is pushed
# Fix: Mark interruptible when safe
test:
interruptible: true
script: npm test
# Reliability checklist:
# - Retry only transient failures
# - Don't allow_failure on critical jobs
# - Let failures propagate
# - Set timeouts
# - Mark interruptible when safe
Hidden Failure Risk:
- A job that swallows errors and exits 0 is worse than a job that doesn't exist
- It gives false confidence that the step succeeded
- Problems surface later, in production, where they're expensive to debug
- If you need to log errors and continue, use a separate job — not a swallowed error
Complete Anti-Pattern Checklist
Use this checklist to audit an existing pipeline or to avoid anti-patterns in a new one. Each item represents a pattern that causes problems at scale. Fixing all of them isn't always necessary — some pipelines can tolerate some anti-patterns — but knowing which ones are present helps you decide what to prioritize.
# SECURITY
# [ ] No secrets in .gitlab-ci.yml
# [ ] All includes pinned to commit SHA
# [ ] CI_JOB_TOKEN allowlist enabled
# [ ] All sensitive variables masked and protected
# [ ] OIDC used for cloud credentials
# [ ] No secrets in artifacts or logs
# PERFORMANCE
# [ ] needs (DAG) used instead of sequential-only
# [ ] Artifacts built once, promoted downstream
# [ ] Cache keyed by lockfile hash
# [ ] Cache policy set (pull-push vs pull)
# [ ] Slow tests sharded with parallel
# [ ] parallel:matrix used for version matrices
# [ ] Linting and scanning jobs use needs: []
# STRUCTURE
# [ ] .gitlab-ci.yml split by responsibility
# [ ] default used for global settings
# [ ] extends depth is 2-3 levels max
# [ ] No copy-paste configuration
# [ ] Hidden jobs documented
# RELIABILITY
# [ ] retry used only for transient failures
# [ ] allow_failure only on experimental jobs
# [ ] No swallowed errors (|| true)
# [ ] Timeouts set on long-running jobs
# [ ] interruptible set on safe jobs
# [ ] resource_group used for deployments
# OBSERVABILITY
# [ ] Pipeline duration tracked
# [ ] Job duration tracked (P50, P95)
# [ ] Failure rate monitored
# [ ] Cache hit rate monitored
# [ ] Alerts on failure rate spikes
# MAINTENANCE
# [ ] Stale feature flags removed
# [ ] Deprecated only/except migrated to rules
# [ ] Old runner versions updated
# [ ] Pipeline config validated (CI Lint)
Before and After: Fixing Anti-Patterns
Here's the same pipeline before and after fixing the most common anti-patterns.
# BEFORE: Anti-patterns everywhere
# stages: [build, test, deploy]
# variables:
# DATABASE_PASSWORD: "supersecret123" # Hardcoded secret
# API_KEY: "abc123" # Hardcoded secret
# include:
# - project: 'templates/ci'
# ref: main # Mutable ref
# build:
# script:
# - npm install # Not npm ci
# - npm run build
# cache:
# paths:
# - node_modules/ # No key
# - dist/ # Should be artifact
# - .env # Should be secret
# test:
# script:
# - npm install # Reinstalls!
# - npm test || true # Hides failure
# retry: 3 # Blanket retry
# deploy:
# script:
# - docker build -t app . # Rebuilds artifact
# - ./deploy.sh
# allow_failure: true # Hides deploy failure
# AFTER: Fixed pipeline
stages:
- build
- test
- deploy
# Secrets moved to masked/protected variables
# Settings > CI/CD > Variables
# Includes pinned to SHA
include:
- project: 'templates/ci'
ref: 'a1b2c3d4e5f6a7b8c9d0e1f2a3b4c5d6e7f8a9b0'
file: '/templates/scan.yml'
default:
image: node:18
cache:
key:
files:
- package-lock.json
paths:
- node_modules/
- .npm/
before_script:
- npm ci --cache .npm --prefer-offline
build:
stage: build
script:
- npm run build
cache:
policy: pull-push
artifacts:
paths:
- dist/
expire_in: 1 day
test:
stage: test
needs: [build]
cache:
policy: pull
script:
- npm test
artifacts:
when: always
reports:
junit: test-results.xml
deploy:
stage: deploy
needs: [test]
resource_group: production
script:
- docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA .
- docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
- ./deploy.sh production
environment:
name: production
rules:
- if: $CI_COMMIT_BRANCH == "main"
when: manual
# Fixes applied:
# - Secrets moved to protected/masked variables
# - Includes pinned to SHA
# - npm ci instead of npm install
# - Cache keyed by lockfile, no build outputs
# - needs used for DAG
# - Artifacts used for build outputs
# - No blanket retry, no swallowed errors
# - No allow_failure on deploy
# - Build once, promote artifact
# - resource_group on deploy
# - Manual production deploy
Frequently Asked Questions
What's the most dangerous anti-pattern?
Hardcoded secrets. A secret in .gitlab-ci.yml is committed to git history forever, exposed to anyone with repository access. Even deleting it doesn't remove it from history. Use masked/protected variables or a secrets manager .
How do I know if my pipeline has anti-patterns?
Use the checklist in this guide. Check for hardcoded secrets, mutable include refs, sequential-only stages, rebuilding artifacts, over-caching, monolithic YAML, and blanket retries. Most anti-patterns are visible in a quick review of the configuration .
Why is npm install an anti-pattern?
npm install can update package-lock.json and install different versions than the lockfile specifies. npm ci installs exactly what's in the lockfile, giving reproducible builds. Always use npm ci in CI.
Why is retry: 2 on every job bad?
It hides flaky tests. A test that fails 10% of the time becomes invisible when retried, and the root cause is never fixed. Use retry only for transient failures (runner_system_failure, stuck_or_timeout_failure) .
What's wrong with a monolithic .gitlab-ci.yml?
It becomes a bottleneck where any change requires reviewing hundreds of lines. Nobody understands the whole file, and merge conflicts are common. Split by responsibility: build.yml, test.yml, deploy.yml .
Should I ever use allow_failure: true?
Yes, for experimental jobs, optional scans, or non-critical checks. Never for production deploys, required tests, or security scans that gate merge. If a job's failure should block the pipeline, don't set allow_failure.
How do I avoid over-abstracting with extends?
Limit extends depth to 2-3 levels. Prefer clarity over DRY — a slightly duplicated pipeline that's easy to trace is better than a heavily abstracted one that nobody can debug. Document hidden jobs.
What's the risk of unpinned includes?
If you include a template at ref: main, whoever controls that branch controls your pipeline. A compromise of the templates repository can compromise every consuming project. Pin to a commit SHA and treat templates as production code .