containerd Secrets Management

Kubernetes Secrets HashiCorp Vault Encryption

Secure handling of sensitive data in containerd and Kubernetes. Learn to use Kubernetes Secrets, HashiCorp Vault, encrypted storage, and best practices for secrets management.

Kubernetes Secrets HashiCorp Vault Encryption

Why Secrets Management Matters

Containers need access to sensitive data: API keys, database passwords, TLS certificates, and more. Exposing these secrets in plaintext is a security risk. Secrets management ensures:

Confidentiality - Secrets are encrypted at rest and in transit.
Integrity - Secrets can't be tampered with.
Access Control - Only authorized containers can access secrets.
Auditability - Track who accessed which secrets.

Never hardcode secrets in images or environment variables. Use a dedicated secrets management solution for production workloads.

Secrets Management Options

Kubernetes Secrets

Pros: Built-in, easy to use, RBAC integrated

Cons: Base64 only (not encrypted by default), etcd storage risk

Best for: Simple secrets, non-sensitive data

HashiCorp Vault

Pros: Enterprise-grade, dynamic secrets, auditing, encryption

Cons: Complex setup, operational overhead

Best for: Production, multi-cloud, sensitive data

External Secrets Operator

Pros: Syncs external secrets to Kubernetes, Vault integration

Cons: Additional operator to manage

Best for: Using Vault with Kubernetes

Sealed Secrets

Pros: GitOps-friendly, encrypted in Git

Cons: Requires controller, decryption at runtime

Best for: GitOps workflows

Kubernetes Secrets

Kubernetes Secrets are the simplest way to manage secrets in Kubernetes. They store data as base64-encoded strings.

                # Create a secret
kubectl create secret generic db-credentials \
  --from-literal=username=admin \
  --from-literal=password=supersecret

# Create from file
kubectl create secret generic tls-cert \
  --from-file=cert.pem=/path/to/cert.pem \
  --from-file=key.pem=/path/to/key.pem

# View secret (base64 decoded)
kubectl get secret db-credentials -o yaml

# Decode secret
echo "YWRtaW4=" | base64 -d

# Use secret in pod
apiVersion: v1
kind: Pod
metadata:
  name: nginx
spec:
  containers:
  - name: nginx
    image: nginx
    env:
    - name: DB_USER
      valueFrom:
        secretKeyRef:
          name: db-credentials
          key: username
    - name: DB_PASS
      valueFrom:
        secretKeyRef:
          name: db-credentials
          key: password

# Mount secret as volume
volumes:
- name: secrets
  secret:
    secretName: db-credentials
    mode: 0400
            

Kubernetes Secrets are base64-encoded, not encrypted by default. Enable etcd encryption for better security. Use encryption providers like KMS or aescbc.

Encryption at Rest for Kubernetes Secrets

Enable encryption at rest to protect secrets in etcd. This prevents attackers with etcd access from reading secret data.

                # EncryptionConfiguration file
# /etc/kubernetes/encryption-config.yaml
apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources:
      - secrets
    providers:
    - aescbc:
        keys:
        - name: key1
          secret: 
    - identity: {}

# Generate encryption key
head -c 32 /dev/urandom | base64

# Enable encryption in kube-apiserver
--encryption-provider-config=/etc/kubernetes/encryption-config.yaml

# Rotate encryption key (add new key)
# Update config with new key as first provider
# Restart kube-apiserver
# Re-encrypt all secrets with new key
            

Encryption at rest is a critical security control for production clusters. Always enable it when using Kubernetes Secrets for sensitive data.

HashiCorp Vault Integration

Vault provides enterprise-grade secrets management with dynamic secrets, encryption as a service, and fine-grained access control.

                # Install Vault
helm repo add hashicorp https://helm.releases.hashicorp.com
helm install vault hashicorp/vault

# Unseal Vault (first time)
vault operator init
vault operator unseal

# Enable Kubernetes auth
vault auth enable kubernetes

# Configure Kubernetes auth
vault write auth/kubernetes/config \
  kubernetes_host="https://kubernetes.default.svc"

# Create a policy
vault policy write app-policy - <
            


            
                 Vault Agent Injector automatically injects secrets into pods as files or environment variables. Secrets are refreshed dynamically.



        
        
            
                
                External Secrets Operator (ESO)
            
            ESO syncs secrets from external providers (Vault, AWS Secrets Manager, GCP Secret Manager) into Kubernetes Secrets.
            
                # Install ESO
helm repo add external-secrets https://charts.external-secrets.io
helm install external-secrets external-secrets/external-secrets

# Create SecretStore (Vault)
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: vault-store
spec:
  provider:
    vault:
      server: "https://vault:8200"
      path: "secret"
      version: "v2"
      auth:
        kubernetes:
          mountPath: "kubernetes"
          role: "app-role"

# Create ExternalSecret
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: db-credentials
spec:
  secretStoreRef:
    name: vault-store
    kind: SecretStore
  target:
    name: db-credentials
    creationPolicy: Owner
  data:
  - secretKey: username
    remoteRef:
      key: app/db
      property: username
  - secretKey: password
    remoteRef:
      key: app/db
      property: password
            
        

        
        
            
                
                Sealed Secrets for GitOps
            
            Sealed Secrets encrypts secrets so they can be safely stored in Git. Only the controller in the cluster can decrypt them.
            
                # Install Sealed Secrets
kubectl apply -f https://github.com/bitnami-labs/sealed-secrets/releases/latest/download/controller.yaml

# Install kubeseal CLI
curl -L https://github.com/bitnami-labs/sealed-secrets/releases/latest/download/kubeseal-linux-amd64 -o kubeseal
chmod +x kubeseal

# Create a sealed secret
kubectl create secret generic db-credentials \
  --from-literal=username=admin \
  --dry-run=client -o yaml > secret.yaml

# Seal the secret
kubeseal -f secret.yaml -w sealed-secret.yaml

# Apply sealed secret (GitOps safe)
kubectl apply -f sealed-secret.yaml

# The controller decrypts and creates the secret
            
        

        
        
            
                
                Secrets Management Best Practices
            
            
                Never hardcode secrets - Use environment variables or mounted files from secrets.
                Encrypt secrets at rest - Enable etcd encryption in Kubernetes.
                Use short-lived secrets - Rotate secrets regularly. Use Vault for dynamic secrets.
                Limit secret access - Use RBAC to restrict which pods can access which secrets.
                Audit secret access - Monitor who accesses secrets using audit logs.
                Use GitOps securely - Use Sealed Secrets or SOPS for Git.
                Don't store secrets in images - Images are immutable and exposed.
                Use encryption providers - Use KMS providers (AWS KMS, GCP KMS) for key management.
                Implement secret rotation - Automate rotation for better security.
                Scan for secrets - Use secret scanning tools (GitLeaks, TruffleHog) to detect leaks.
            
            
                 Following these best practices significantly reduces the risk of secret exposure. Start with Kubernetes Secrets and move to Vault for advanced needs.
            
        

        
        
            
                
                Secret Scanning and Prevention
            
            Detect secrets in code and prevent them from being committed to Git.
            
                # GitLeaks (pre-commit scanning)
gitleaks detect --source . --report-format json

# TruffleHog
trufflehog filesystem .
trufflehog github --repo https://github.com/user/repo

# Pre-commit hook
# .pre-commit-config.yaml
repos:
- repo: https://github.com/gitleaks/gitleaks
  rev: v8.16.0
  hooks:
  - id: gitleaks

# GitHub Secret Scanning
# Enable in GitHub settings > Security > Secret scanning

# GitLab Secret Detection
# Enable in Security & Compliance > Secret Detection
            
        

        
        
            
                
                Troubleshooting Secrets
            
            
                # Check if secret exists
kubectl get secret db-credentials

# View secret details (avoid showing values)
kubectl describe secret db-credentials

# Check if pod can access secret
kubectl exec -it nginx -- env | grep DB_

# Check secret mount
kubectl exec -it nginx -- ls -la /etc/secrets

# Verify encryption at rest
kubectl get secret db-credentials -o yaml

# Check Vault health
vault status
vault secrets list

# Test Vault authentication
vault token lookup
            
        

        
        
            
                
                Frequently Asked Questions
            

            
                 Are Kubernetes Secrets secure?
                By default, Kubernetes Secrets are base64-encoded, not encrypted. Enable encryption at rest for better security. For highly sensitive data, use HashiCorp Vault or External Secrets Operator.
            

            
                 What's the difference between Secrets and ConfigMaps?
                Secrets are for sensitive data (passwords, keys). ConfigMaps are for non-sensitive configuration data. Secrets have better access control and can be encrypted.
            

            
                 How do I rotate secrets in Kubernetes?
                Update the secret with new values. Pods using the secret need to be restarted to pick up changes (unless using volume mounts with auto-update). For Vault, use dynamic secrets that expire automatically.
            

            
                 Can I use HashiCorp Vault with containerd directly?
                Not directly. Vault integrates at the Kubernetes level using the Vault Agent Injector or External Secrets Operator. Pods access secrets via volumes or environment variables.
            

            
                 What is the best way to handle secrets in Git?
                Use Sealed Secrets, SOPS (Secrets OPerationS), or external secret management (Vault). Never commit raw secrets to Git. Use pre-commit hooks to detect secrets.
            

            
                 How do I limit which pods can access secrets?
                Use Kubernetes RBAC: create roles that grant access to specific secrets and bind them to service accounts. Use namespace isolation for additional separation.
            

            
                 Should I use environment variables or volumes for secrets?
                Volumes are preferred for security. Environment variables can be exposed in logs or debugging. Volumes also allow automatic updates when secrets change.
            

            
                 What is encryption at rest in Kubernetes?
                Encryption at rest encrypts secret data stored in etcd. Without it, anyone with etcd access can read secrets in plaintext. Enable it with the EncryptionConfiguration API.
            
        

        
        
             Previous: Image Signing & Verification
             Next: Troubleshooting
        

         Secrets management is a critical aspect of container security. Choose the right solution for your needs and follow best practices to protect sensitive data.