Container Scanning

Vulnerability Scanning Image Signing Security Best Practices

A complete guide to container scanning in GitLab covering vulnerability scanning, image signing, security best practices, and integration with CI/CD pipelines.

Vulnerability Scanning Image Signing Security GitLab

Why Container Scanning Matters

Containers bundle an application with its dependencies — including the operating system libraries, language runtime, and third-party packages. Every one of those components can have known vulnerabilities. A base image that was secure when you built it a month ago might have three new CVEs today. Without scanning, you won't know until an attacker exploits them.

Container scanning catches these issues early. It inspects the image's OS packages and application dependencies against known vulnerability databases (such as NVD, GitHub Advisory Database, or vendor feeds), reports findings, and integrates with your pipeline to fail the build or alert your team. Combined with image signing and admission control, it forms a strong supply chain security posture.

GitLab offers container scanning as part of its security scanning suite, along with the ability to sign images using Cosign and verify signatures before deployment. Together these capabilities let you catch vulnerabilities early and prevent tampered images from reaching production.

Core Principle: Container scanning is not a one-time activity. Images must be rescanned periodically — a "clean" scan today doesn't guarantee safety a month from now, because new CVEs are disclosed constantly.

Container Scanning Flow

Container scanning is typically integrated into a CI/CD pipeline as a dedicated job. The flow is: build the image, push it to the registry, scan the pushed image, and report findings. Scanning the pushed image (rather than the local build) ensures you're scanning exactly what will be deployed.

Pipeline Build image Push to registry

Security Stage Container scanning Generate report

Reporting Merge request Approvals

Deploy Pass policy Deploy

                # Pipeline structure
                stages:
                  - build
                  - test
                  - security
                  - deploy

                # Build image
                build:
                  stage: build
                  image: docker:24
                  services:
                    - docker:24-dind
                  before_script:
                    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
                  script:
                    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA .
                    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA

                # Security stage (runs after build)
                container_scanning:
                  stage: security
                  # Scanning runs against the pushed image
            

GitLab Container Scanning

GitLab provides built-in container scanning via a CI/CD template. The template runs the scanner (typically Trivy or Grype) against a Docker image and produces a JSON report that GitLab reads to display vulnerabilities in the merge request UI and security dashboard.

Container scanning is available in GitLab Ultimate (formerly Gold). It produces a gl-container-scanning-report.json artifact that GitLab parses and displays. Vulnerabilities appear in merge requests (allowing you to block merges with critical findings), in the project's security dashboard, and in the group security dashboard for organization-wide visibility.

The scanner can be configured to fail the job on high-severity findings (with allow_failure: false), warn but continue (default), or scan only specific severities. A common production policy is: fail on critical, warn on high, ignore medium and low.

                # .gitlab-ci.yml with container scanning

                # Include the official template
                include:
                  - template: Security/Container-Scanning.gitlab-ci.yml

                # Configure container scanning
                container_scanning:
                  stage: security
                  variables:
                    # The image to scan
                    DOCKER_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA

                    # Image to pull from registry
                    DOCKER_USER: $CI_REGISTRY_USER
                    DOCKER_PASSWORD: $CI_REGISTRY_PASSWORD

                    # Scanner configuration
                    CS_SEVERITY_THRESHOLD: "HIGH"
                    CS_DOCKERFILE_PATH: "Dockerfile"

                    # Timeout (seconds)
                    CS_SCANNER_TIMEOUT: "300"

                    # Ignore unfixed vulnerabilities
                    CS_IGNORE_UNFIXED: "false"

                    # Report format
                    CS_REPORT_FORMAT: "json"
                  artifacts:
                    reports:
                      container_scanning: gl-container-scanning-report.json
                  allow_failure: false    # Fail pipeline on findings

                # Manual scan configuration (no template)
                container_scanning_manual:
                  stage: security
                  image: docker:24
                  services:
                    - docker:24-dind
                  variables:
                    DOCKER_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
                  before_script:
                    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
                    - apk add --no-cache curl
                  script:
                    # Install Trivy
                    - curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin
                    # Scan image and generate report
                    - trivy image --format json --output gl-container-scanning-report.json $DOCKER_IMAGE
                  artifacts:
                    reports:
                      container_scanning: gl-container-scanning-report.json
                    paths:
                      - gl-container-scanning-report.json

                # Scanning with severity filtering
                container_scanning_filtered:
                  extends: container_scanning
                  variables:
                    CS_SEVERITY_THRESHOLD: "CRITICAL"
                  allow_failure: false

                # Available scanner variables:
                # CS_DOCKERFILE_PATH: Path to Dockerfile
                # CS_SEVERITY_THRESHOLD: Minimum severity (LOW, MEDIUM, HIGH, CRITICAL)
                # CS_IGNORE_UNFIXED: Only report fixed vulnerabilities
                # CS_SCANNER_TIMEOUT: Timeout in seconds
                # CS_DOCKER_IMAGE: Image to scan
                # CS_SEVERITY_FILTER: Additional severity filtering
            

Container Scanning Best Practices:

Scan the pushed image (not the local build) to verify what will deploy
Set severity threshold to CRITICAL for fail-fast policies
Enable scanning on every pipeline, not just releases
Rescan existing images periodically — new CVEs emerge
Filter to fixed vulnerabilities only for actionable results
Combine with SAST and dependency scanning for full coverage

Vulnerability Scanning with Trivy

Trivy is the most widely used open-source container vulnerability scanner. It's fast, has a large vulnerability database, supports multiple image formats, and generates reports in many formats including GitLab's expected JSON schema.

Trivy scans image layers, OS packages (Alpine apk, Debian dpkg, RHEL rpm), language dependencies (npm, pip, go, maven), and configuration files. It cross-references findings against vulnerability databases, providing CVE IDs, severity, fixed versions, and remediation guidance.

In a GitLab pipeline, Trivy is usually run as part of the security stage. Its output is a JSON report that GitLab ingests for display. Trivy can also be run locally for pre-commit checks, integrated into other CI systems, or used as part of admission control.

                # Install and use Trivy

                # Install via script (Linux)
                curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin

                # Or download binary
                wget https://github.com/aquasecurity/trivy/releases/latest/download/trivy_0.48.0_Linux-64bit.tar.gz
                tar xf trivy_0.48.0_Linux-64bit.tar.gz
                mv trivy /usr/local/bin/

                # Scan an image
                trivy image nginx:latest

                # Scan with severity filter
                trivy image --severity HIGH,CRITICAL nginx:latest

                # Scan with output format
                trivy image --format json --output report.json nginx:latest
                trivy image --format table --output report.txt nginx:latest
                trivy image --format sarif --output report.sarif nginx:latest

                # Scan local Docker daemon image
                trivy image --input image.tar

                # Ignore unfixed vulnerabilities
                trivy image --ignore-unfixed nginx:latest

                # Use a specific database
                trivy image --db-repository registry.gitlab.com/gitlab-org/security-products/analyzers/trivy-db:latest nginx:latest

                # Complete CI job with Trivy
                trivy_scan:
                  stage: security
                  image: docker:24
                  services:
                    - docker:24-dind
                  before_script:
                    - apk add --no-cache curl
                    - curl -sfL https://raw.githubusercontent.com/aquasecurity/trivy/main/contrib/install.sh | sh -s -- -b /usr/local/bin
                    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
                  script:
                    # Scan image and generate GitLab-compatible report
                    - trivy image --format template --template "@contrib/gitlab.tpl" \
                        --output gl-container-scanning-report.json \
                        --severity HIGH,CRITICAL \
                        $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
                  artifacts:
                    reports:
                      container_scanning: gl-container-scanning-report.json
                    paths:
                      - gl-container-scanning-report.json
                  allow_failure: false

                # Trivy severity levels:
                # CRITICAL: Exploits in the wild, immediate action needed
                # HIGH: Significant impact, action needed soon
                # MEDIUM: Moderate impact, schedule fix
                # LOW: Minor impact, fix when convenient
                # UNKNOWN: No severity assigned

                # Trivy scan types:
                # --scanners vuln    Scan for vulnerabilities (default)
                # --scanners config  Scan for misconfigurations (IaC)
                # --scanners secret  Scan for secrets
                # --scanners license Scan for license compliance
                # Combine with: --scanners vuln,config,secret

                # Trivy configuration file (.trivy.yaml)
                severity:
                  - HIGH
                  - CRITICAL
                ignore-unfixed: true
                exit-code: 1
                format: json
                output: gl-container-scanning-report.json

                # Scan and fail if any critical CVE
                trivy image --severity CRITICAL --exit-code 1 nginx:latest
            

Trivy Advantages:

Fast and efficient (seconds per image)
Supports all major image formats
Scans OS packages, language deps, misconfigs, secrets
Multiple output formats (JSON, table, SARIF, JUnit)
Free and open source
Can be used locally, in CI, and in admission control

Image Signing with Cosign

Container image signing proves that an image was created by a trusted entity and hasn't been tampered with. It's the cryptographic equivalent of a wax seal: a signature attached to the image that can be verified before it's deployed.

Cosign (from the Sigstore project) is the modern standard for image signing. It supports key-based signing (with a key pair you manage) and keyless signing (using OIDC identity — e.g., your GitHub account). It integrates with GitLab CI through pipeline jobs and with Kubernetes through admission controllers.

The workflow is: build the image, push it to the registry, sign it with Cosign, and verify the signature before deploying. Signing is a one-time operation per image version; verification is repeated on every deploy. For Kubernetes, admission controllers (like Kyverno or OPA Gatekeeper) can enforce that only signed images are allowed to run.

                # Install Cosign
                curl -L https://github.com/sigstore/cosign/releases/latest/download/cosign-linux-amd64 \
                  -o /usr/local/bin/cosign
                chmod +x /usr/local/bin/cosign

                # Generate a key pair
                cosign generate-key-pair

                # This creates:
                # - cosign.key (private key, keep secret!)
                # - cosign.pub (public key, share freely)

                # Sign an image
                cosign sign --key cosign.key $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA

                # Verify an image
                cosign verify --key cosign.pub $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA

                # Signature format (stored as OCI artifact):
                # - tag: sha256-.sig
                # - signature: base64-encoded
                # - cert: optional (for keyless)

                # Keyless signing (Sigstore)
                # Uses OIDC identity — no private key management
                cosign sign $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA

                # This uses the OIDC identity of the CI job
                # Requires COSIGN_EXPERIMENTAL=1 and OIDC token

                # In GitLab CI, use the job token for keyless signing
                sign:
                  stage: security
                  image: bitnami/cosign:latest
                  variables:
                    COSIGN_EXPERIMENTAL: "1"
                  script:
                    - cosign sign --identity-token $CI_JOB_JWT_V2 $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
                  only:
                    - main

                # Verify with keyless
                cosign verify --certificate-identity-regexp \
                  ".*@gitlab\\.com" \
                  --certificate-oidc-issuer \
                  "https://gitlab.com" \
                  $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA

                # Attach attestations (SBOM, SLSA provenance)
                cosign attest --key cosign.key \
                  --predicate sbom.json \
                  --type spdxjson \
                  $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA

                # Full pipeline: build, sign, verify
                stages:
                  - build
                  - security
                  - deploy

                build:
                  stage: build
                  image: docker:24
                  services: [docker:24-dind]
                  before_script:
                    - docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
                  script:
                    - docker build -t $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA .
                    - docker push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA

                sign:
                  stage: security
                  image: bitnami/cosign:latest
                  needs: [build]
                  before_script:
                    - cosign login $CI_REGISTRY -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD
                  script:
                    - cosign sign --key $COSIGN_PRIVATE_KEY $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
                  only:
                    - main

                verify:
                  stage: deploy
                  image: bitnami/cosign:latest
                  before_script:
                    - cosign login $CI_REGISTRY -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD
                  script:
                    - cosign verify --key $COSIGN_PUBLIC_KEY $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
                    - kubectl set image deployment/myapp myapp=$CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
                  only:
                    - main

                # Kyverno policy for image verification
                apiVersion: kyverno.io/v1
                kind: ClusterPolicy
                metadata:
                  name: verify-image-signatures
                spec:
                  validationFailureAction: enforce
                  rules:
                    - name: verify-signature
                      match:
                        resources:
                          kinds: [Pod]
                      verifyImages:
                        - imageReferences: ["registry.gitlab.com/mygroup/*"]
                          attestors:
                            - entries:
                                - keys:
                                    publicKeys: |-
                                      -----BEGIN PUBLIC KEY-----
                                      
                                      -----END PUBLIC KEY-----
            

Image Signing Best Practices:

Sign every image you push to the registry
Verify signatures before deploying to Kubernetes
Use keyless signing for CI/CD (no key management)
Use admission controllers (Kyverno, OPA) to enforce signature verification
Store private keys in secret managers (never in code)
Rotate keys periodically
Combine with SBOM attestations for full provenance

Software Bill of Materials (SBOM)

An SBOM is a machine-readable inventory of all components in an image — OS packages, language dependencies, and files. It's essential for supply chain security: when a new CVE is disclosed, you can check your SBOMs to see which images are affected.

SBOMs are typically generated during the build (using tools like Syft, Trivy, or cdxgen) and stored as OCI artifacts in the registry alongside the image. They can also be attached as attestations (using Cosign), making them verifiable.

                # Generate SBOM with Syft
                syft $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA -o spdx-json > sbom.spdx.json

                # Generate SBOM with Trivy
                trivy image --format spdx-json --output sbom.spdx.json $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA

                # Generate SBOM with cdxgen (CycloneDX)
                cdxgen -o sbom.cdx.json .

                # Attach SBOM to image with Cosign
                cosign attach sbom --sbom sbom.spdx.json $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA

                # Verify SBOM
                cosign verify-attestation --key cosign.pub $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA

                # SBOM in CI pipeline
                generate_sbom:
                  stage: security
                  image: aquasec/trivy:latest
                  needs: [build]
                  script:
                    - trivy image --format spdx-json --output sbom.spdx.json $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
                  artifacts:
                    paths:
                      - sbom.spdx.json

                # Scanning the SBOM for vulnerabilities
                # When a new CVE is disclosed, check which images
                # contain the affected package (using SBOMs)
            

Security Scanning Types

Container scanning is one part of a broader security scanning strategy. Understanding how the different scan types complement each other helps you build comprehensive coverage.

Container Scanning

Scans image layers (OS packages, language dependencies) for known CVEs. Runs after image build. Catches vulnerabilities in the image contents.

SAST

Static Application Security Testing. Scans source code for vulnerabilities (SQL injection, XSS, etc.). Runs during the build stage.

Dependency Scanning

Scans package manifests (package.json, go.mod, requirements.txt) for vulnerable dependencies. Runs on every commit.

DAST

Dynamic Application Security Testing. Scans running applications for vulnerabilities. Runs against staging environments.

Secret Detection

Scans code and commit history for secrets (API keys, tokens). Runs on every commit.

IaC Scanning

Scans Terraform, Kubernetes manifests, and other IaC for misconfigurations. Runs on every commit.

                # Complete security scanning pipeline

                include:
                  - template: Security/SAST.gitlab-ci.yml
                  - template: Security/Secret-Detection.gitlab-ci.yml
                  - template: Security/Dependency-Scanning.gitlab-ci.yml
                  - template: Security/Container-Scanning.gitlab-ci.yml
                  - template: Security/DAST.gitlab-ci.yml

                stages:
                  - build
                  - test
                  - security
                  - deploy

                # SAST (static analysis)
                sast:
                  stage: security
                  variables:
                    SAST_EXCLUDED_PATHS: "spec, test, tests, tmp"

                # Secret detection
                secret_detection:
                  stage: security
                  variables:
                    SECRET_DETECTION_HISTORIC_SCAN: "true"

                # Dependency scanning
                dependency_scanning:
                  stage: security
                  variables:
                    DS_EXCLUDED_PATHS: "spec, test, tests, tmp"

                # Container scanning
                container_scanning:
                  stage: security
                  variables:
                    DOCKER_IMAGE: $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
                  needs: [build]

                # DAST (dynamic)
                dast:
                  stage: security
                  variables:
                    DAST_WEBSITE: "https://staging.example.com"
                  only:
                    - main

                # All security jobs run in parallel, produce reports,
                # and are aggregated in the security dashboard
            

Troubleshooting Container Scanning

                # Container scanning job fails
                # 1. Check image exists in registry
                docker pull $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA

                # 2. Check registry credentials
                docker login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY

                # 3. Check scanner configuration
                cat .gitlab-ci.yml | grep -A 10 container_scanning

                # 4. View scanner logs
                # The scanner output is in the job log

                # "No vulnerabilities found" but expected some
                # - Ensure the correct image is being scanned
                # - Check that the vulnerability database is up to date
                # - Check if --ignore-unfixed is filtering results
                # - Check severity threshold

                # Scanner timeout
                # - Increase CS_SCANNER_TIMEOUT
                # - Use a smaller image
                # - Increase runner resources

                # "Unauthorized" errors
                # - Check CI_REGISTRY_USER and CI_REGISTRY_PASSWORD
                # - Check the runner can reach the registry
                # - Check image pull permissions

                # Signature verification fails
                cosign verify --key cosign.pub $IMAGE

                # - Check the image is signed
                cosign verify --key cosign.pub $IMAGE --output json

                # - Check the public key matches the private key used
                # - Check the image digest (signatures are per digest, not tag)

                # SBOM generation fails
                syft $IMAGE -o spdx-json

                # - Check image exists
                # - Check the analyzer has permission to pull

                # Debug with verbose output
                trivy image --debug $IMAGE
                cosign verify --key cosign.pub $IMAGE --verbose

                # Reset scanner cache
                rm -rf ~/.cache/trivy
            

Frequently Asked Questions

What's the difference between container scanning and dependency scanning?

Container scanning scans the built image's layers for CVEs. Dependency scanning scans package manifests (package.json, go.mod) for vulnerable dependencies. Both are needed — container scanning catches OS packages, while dependency scanning catches application packages.

Which scanner should I use — Trivy, Grype, or Clair?

Trivy is the most widely used and fastest. Grype is also excellent (from Anchore). Clair is older but mature. For most teams, Trivy is the best default choice because it's fast, well-maintained, and integrates with GitLab.

Should I fail the pipeline on vulnerabilities?

For critical vulnerabilities, yes. For high, maybe. For medium/low, typically no. A common policy: fail on critical, warn on high, ignore lower. This balances security with pragmatism.

What's keyless signing with Cosign?

Keyless signing uses your OIDC identity (e.g., GitHub or GitLab account) to sign images — no private key to manage. The signature is verified against the OIDC identity. It's simpler and more secure than key-based signing for CI/CD.

How do I enforce image signing in Kubernetes?

Use an admission controller like Kyverno or OPA Gatekeeper to require signed images. Configure the policy to reject unsigned images. This prevents tampered images from being deployed.

How often should I rescan images?

New CVEs are disclosed constantly. Rescan existing images weekly or monthly. Some teams rescan daily. The GitLab Container Registry can be integrated with scheduled scanning jobs.

What's an SBOM and why do I need one?

An SBOM (Software Bill of Materials) is a machine-readable inventory of all components in an image. When a new CVE is disclosed, you can search your SBOMs to find affected images. It's required by many compliance standards.

Does container scanning slow down pipelines?

Yes, but typically by 30-60 seconds per scan. This is usually acceptable. You can run scans in parallel with other jobs, or only scan on main and release branches to reduce overhead.

Container Scanning

Container Scanning

SAST

Dependency Scanning

DAST

Secret Detection

IaC Scanning

GitLab Container Registry

Registry Cleanup

Security Scanning

Secrets Management

CI/CD Best Practices