Helm Security Best Practices

Secrets Management RBAC Chart Signing Provenance

A comprehensive guide to Helm security best practices covering secrets management, RBAC, chart signing, provenance, and securing Helm deployments in production.

Secrets Management RBAC Chart Signing Provenance

Why Helm Security Matters

Helm charts can contain sensitive information and have the power to deploy arbitrary resources in your Kubernetes cluster. Without proper security controls, Helm can become a significant attack vector.

Key security areas for Helm include:

Secrets Management: Handling sensitive data securely
RBAC: Limiting who can deploy and manage releases
Chart Signing: Ensuring chart authenticity
Provenance: Verifying chart integrity
Supply Chain Security: Securing the chart distribution pipeline
Values Security: Protecting configuration data

Key Principle: Treat Helm charts like any other software artifact. Apply the same security rigor to chart development, distribution, and deployment as you would to container images.

Secrets Management

Never store secrets in Helm charts or values files. Use external secret management solutions for production deployments.

                # Bad Practice: Secrets in values.yaml
# NEVER DO THIS
database:
  username: admin
  password: supersecretpassword  # NEVER store secrets in values.yaml
  host: db.example.com

# Bad Practice: Secrets in --set
# Visible in process list and shell history
helm install my-release ./my-chart --set database.password=supersecret

# Good Practice: External Secrets Operator
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-store
spec:
  provider:
    vault:
      server: "https://vault.example.com"
      path: "secret"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "my-app-role"

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  secretStoreRef:
    name: vault-store
    kind: SecretStore
  target:
    name: db-credentials
  data:
  - secretKey: username
    remoteRef:
      key: app/db
      property: username
  - secretKey: password
    remoteRef:
      key: app/db
      property: password

# Good Practice: Sealed Secrets
apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  name: db-credentials
spec:
  encryptedData:
    username: AgBy...  # Encrypted
    password: AgBy...  # Encrypted

# Good Practice: Helm Secrets Plugin
# Install plugin
helm plugin install https://github.com/jkroepke/helm-secrets

# Encrypt values file
helm secrets enc secrets.yaml

# Use encrypted values
helm secrets install my-release ./my-chart -f secrets.yaml

# Decrypt for inspection
helm secrets dec secrets.yaml

# Good Practice: SOPS with Helm
# Install SOPS
helm plugin install https://github.com/jkroepke/helm-secrets

# Encrypt with SOPS
sops --encrypt --in-place secrets.yaml

# Use with Helm
helm secrets install my-release ./my-chart -f secrets.yaml
            

External Secrets Operator

Sync secrets from external providers (Vault, AWS Secrets Manager, GCP Secret Manager) into Kubernetes Secrets.

Sealed Secrets

Encrypt secrets for GitOps. Only the controller in the cluster can decrypt them.

Helm Secrets Plugin

Encrypt values files using SOPS, AWS KMS, GCP KMS, or Azure Key Vault.

Cloud Secret Managers

Use cloud-native secret managers (AWS Secrets Manager, GCP Secret Manager, Azure Key Vault).

Critical: Secrets in values files are stored in plaintext in the Helm release secret. Always use external secret management for production. Even with --set, secrets are visible in process lists and shell history.

RBAC for Helm

Helm uses the Kubernetes RBAC system. Control who can install, upgrade, and delete releases with fine-grained permissions.

                # Tiller (Helm v2) - ClusterAdmin (not recommended)
# Helm v3 uses the user's kubeconfig credentials

# Good Practice: Namespaced RBAC for Helm
apiVersion: v1
kind: ServiceAccount
metadata:
  name: helm-user
  namespace: default

apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: helm-user-role
  namespace: default
rules:
- apiGroups: [""]
  resources: ["pods", "services", "configmaps", "secrets"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["apps"]
  resources: ["deployments", "statefulsets", "daemonsets"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]
- apiGroups: ["batch"]
  resources: ["jobs", "cronjobs"]
  verbs: ["get", "list", "watch", "create", "update", "patch", "delete"]

apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: helm-user-binding
  namespace: default
subjects:
- kind: ServiceAccount
  name: helm-user
  namespace: default
roleRef:
  kind: Role
  name: helm-user-role
  apiGroup: rbac.authorization.k8s.io

# Good Practice: Restrict Helm to specific namespaces
# Users can only deploy to their own namespaces

# Good Practice: Use separate service accounts for CI/CD
apiVersion: v1
kind: ServiceAccount
metadata:
  name: helm-ci
  namespace: default

# Good Practice: Limit secret access
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: helm-secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get"]
  resourceNames: ["my-app-secret"]

# Good Practice: Validate permissions
kubectl auth can-i create deployments --as=system:serviceaccount:default:helm-user
kubectl auth can-i delete secrets --as=system:serviceaccount:default:helm-user
            

RBAC Best Practices:

Use namespaced Roles instead of ClusterRoles
Use separate service accounts for different teams
Apply least privilege principle
Regularly audit RBAC permissions
Use kubectl auth can-i to verify permissions
Implement namespace isolation

Chart Signing and Provenance

Helm supports chart signing using PGP keys. This ensures chart authenticity and integrity.

                # Generate a PGP key pair
gpg --full-generate-key

# List GPG keys
gpg --list-keys

# Export public key
gpg --export --armor "Your Name" > public-key.asc

# Export private key (keep secure!)
gpg --export-secret-keys --armor "Your Name" > private-key.asc

# Sign a chart during packaging
helm package ./my-chart --sign --key "Your Name" --keyring ~/.gnupg/secring.gpg

# This creates:
# - my-chart-1.0.0.tgz (the chart)
# - my-chart-1.0.0.tgz.prov (the provenance file)

# Verify a signed chart
helm verify my-chart-1.0.0.tgz

# Verify with keyring
helm verify --keyring ~/.gnupg/pubring.gpg my-chart-1.0.0.tgz

# Install a signed chart (verifies automatically)
helm install my-release my-chart-1.0.0.tgz

# Pull and verify a chart
helm pull my-repo/my-chart --verify

# View provenance file
cat my-chart-1.0.0.tgz.prov

# Import public key for verification
gpg --import public-key.asc

# Check signature
gpg --verify my-chart-1.0.0.tgz.prov
            

Provenance File Contents:

Chart name and version
Chart SHA256 hash
Signature from the chart author
Timestamp of signing
PGP key information

Chart Signing

Sign charts with PGP keys to prove authorship and integrity.

Provenance

Provenance files contain metadata and signatures for verification.

Verification

Verify chart signatures before installation to ensure authenticity.

Key Management

Securely manage PGP keys for signing and verification.

Signing Best Practices:

Sign all production charts
Store private keys securely (HSM, Vault)
Use strong PGP keys (4096-bit RSA)
Rotate keys regularly
Publish public keys for verification
Verify signatures in CI/CD pipelines

Supply Chain Security

                # 1. Use trusted repositories
helm repo add bitnami https://charts.bitnami.com/bitnami  # Official
helm repo add my-company https://charts.mycompany.com    # Private

# 2. Verify chart signatures
helm pull my-company/my-chart --verify

# 3. Scan charts for vulnerabilities
# Install Trivy plugin
helm plugin install https://github.com/aquasecurity/helm-trivy

# Scan chart
helm trivy scan ./my-chart

# Or use trivy directly
trivy config ./my-chart

# 4. Validate Kubernetes manifests
helm template my-release ./my-chart | kubeconform -strict

# 5. Use OPA/Gatekeeper for policy enforcement
apiVersion: constraints.gatekeeper.sh/v1beta1
kind: K8sAllowedRegistries
metadata:
  name: allowed-registries
spec:
  match:
    kinds:
    - apiGroups: [""]
      kinds: ["Pod"]
  parameters:
    registries:
    - "myregistry.azurecr.io/"
    - "gcr.io/"
    - "docker.io/"

# 6. Use immutable tags for images
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
      - name: app
        image: myregistry.azurecr.io/my-app:v1.2.3  # Specific tag
        imagePullPolicy: Always

# 7. Implement SBOM generation
# Generate SBOM for chart
syft ./my-chart -o spdx-json > sbom.json

# 8. Use dependency scanning
helm dependency list ./my-chart
trivy fs --security-checks vuln ./my-chart
            

Trusted Repositories

Only use charts from trusted, verified sources.

Chart Signing

Verify chart signatures before use.

Vulnerability Scanning

Scan charts for known vulnerabilities.

Policy Enforcement

Use OPA/Gatekeeper to enforce security policies.

SBOM Generation

Generate Software Bill of Materials for charts.

Immutable Tags

Use specific image tags, never latest.

Security Best Practices Summary

                # Helm Security Checklist
# ========================

# Secrets Management
☐ Never store secrets in values.yaml
☐ Use External Secrets Operator or Sealed Secrets
☐ Use helm-secrets plugin for encrypted values
☐ Rotate secrets regularly
☐ Use RBAC for secret access

# RBAC
☐ Use namespaced Roles instead of ClusterRoles
☐ Apply least privilege principle
☐ Use separate service accounts for CI/CD
☐ Regularly audit RBAC permissions
☐ Implement namespace isolation

# Chart Signing
☐ Sign all production charts
☐ Store private keys securely
☐ Use strong PGP keys
☐ Rotate keys regularly
☐ Verify signatures before installation

# Supply Chain Security
☐ Use trusted repositories only
☐ Verify chart signatures
☐ Scan charts for vulnerabilities
☐ Use OPA/Gatekeeper for policy enforcement
☐ Generate SBOMs for charts
☐ Use immutable image tags

# CI/CD Security
☐ Use secrets management in CI/CD
☐ Verify charts in pipelines
☐ Scan charts for vulnerabilities
☐ Use dedicated service accounts
☐ Implement audit logging

# Production Security
☐ Use network policies
☐ Enable pod security standards
☐ Use security contexts
☐ Implement resource quotas
☐ Monitor deployments
            

Frequently Asked Questions

Where should I store secrets for Helm charts?

Use external secret management solutions like HashiCorp Vault, AWS Secrets Manager, or Sealed Secrets. Never store secrets in values.yaml or pass them via --set.

How do I sign a Helm chart?

Generate a PGP key pair with gpg --full-generate-key, then package with helm package --sign --key "Your Name". This creates a .prov file for verification.

How do I verify a signed chart?

Use helm verify my-chart-1.0.0.tgz with the public key imported. Or use helm pull --verify to verify while pulling.

What RBAC permissions does Helm need?

Helm needs permissions to create, update, and delete resources defined in the chart. Use namespaced Roles and limit permissions to specific resource types and names.

How do I use Helm with External Secrets Operator?

Create a SecretStore and ExternalSecret resource in your chart. The operator syncs secrets from external providers (Vault, AWS, GCP) into Kubernetes Secrets.

What is the helm-secrets plugin?

helm-secrets is a plugin that allows you to encrypt values files using SOPS, AWS KMS, GCP KMS, or Azure Key Vault. It decrypts values during installation.

How do I scan Helm charts for vulnerabilities?

Use Trivy with trivy config ./my-chart or install the helm-trivy plugin. Also scan dependencies and generated manifests.

What is the provenance file in Helm?

The provenance file (.prov) contains the chart's metadata, SHA256 hash, and PGP signature. It's used to verify chart authenticity and integrity.

Helm Security Best Practices

External Secrets Operator

Sealed Secrets

Helm Secrets Plugin

Cloud Secret Managers

Chart Signing

Provenance

Verification

Key Management

Trusted Repositories

Chart Signing

Vulnerability Scanning

Policy Enforcement

SBOM Generation

Immutable Tags

What is Helm?

Secrets in Helm

Chart Provenance

Testing Charts

Helm Best Practices