containerd Production Deployment Guide

Best Practices Monitoring Security Performance

A comprehensive guide to deploying containerd in production environments. Learn best practices for configuration, monitoring, security, performance optimization, and operational considerations for enterprise-grade container infrastructure.

Production Ready Best Practices Enterprise Grade

Why Production Deployments Require Special Attention

Deploying containerd in production is fundamentally different from development or testing environments. Production workloads demand high availability, performance, security, and operational excellence. A poorly configured containerd deployment can lead to:

Downtime - Failed container starts or image pulls causing application outages
Performance degradation - Slow container startup times affecting user experience
Security breaches - Container escapes or unauthorized access due to misconfiguration
Resource exhaustion - Disk space filling up, causing node failures
Operational overhead - Constant firefighting and manual interventions

Key principle: Production deployments should be predictable, observable, and resilient. Every configuration choice should be intentional and documented. This guide provides a battle-tested approach to achieving these goals.

Production Deployment Steps

1. Capacity Planning and Resource Allocation

Determine storage, memory, and CPU requirements based on workload patterns. Allocate at least 50-100GB for /var/lib/containerd. Consider using SSD storage for better I/O performance. Plan for growth—monitor usage and scale storage proactively.

2. Install containerd from Official Repositories

Use distribution-specific packages (apt, yum) or official releases. Avoid building from source for production. Pin to specific versions and test upgrades in staging first. Document the installed version and upgrade process.

3. Generate and Validate Configuration

Start with the default configuration, then customize for production. Use containerd config default > /etc/containerd/config.toml and modify settings. Always validate with containerd config validate before restarting.

4. Configure Runtime and CRI Settings

Set SystemdCgroup = true for cgroup v2 compatibility. Configure the pause image, snapshotter (overlayfs recommended), and default runtime. Ensure the CRI plugin is enabled for Kubernetes integration.

5. Set Up Registry Authentication and Mirrors

Configure registry mirrors to reduce pull times and avoid rate limits. Set up authentication for private registries using credentials or image pull secrets. Use config_path for managing multiple registry configurations.

6. Enable Metrics and Monitoring

Enable the metrics endpoint and integrate with Prometheus. Set up Grafana dashboards for visualization. Configure alerts for disk usage, operation failures, and performance anomalies.

7. Configure Security Policies

Enable seccomp profiles, AppArmor or SELinux policies. Implement image signing verification with Cosign. Configure network policies to restrict container communication. Enable audit logging for security compliance.

8. Set Up Logging and Log Rotation

Configure container log rotation to prevent disk exhaustion. Use logrotate or configure the CRI plugin's max_container_log_line_size. Set up centralized log shipping for operational visibility.

9. Implement Automated Cleanup

Set up cron jobs or systemd timers for regular image, snapshot, and content pruning. Configure Kubernetes image GC thresholds in the kubelet. Monitor cleanup effectiveness and adjust schedules as needed.

10. Test and Validate in Staging

Before deploying to production, thoroughly test all configurations in a staging environment. Validate performance, security, and operational procedures. Document the rollback process and test it.

11. Deploy and Monitor Gradually

Use a canary or phased rollout strategy. Start with a single node, monitor for issues, then gradually expand. Have a clear rollback plan for each phase. Document the deployment process.

Production Configuration Template

Here's a production-ready config.toml with all essential settings and explanations:

                # /etc/containerd/config.toml - Production Configuration
version = 2

# Storage paths - use SSD for /var/lib/containerd
root = "/var/lib/containerd"
state = "/run/containerd"

# Logging - info level for production, debug only for troubleshooting
log_level = "info"
log_format = "text"  # or "json" for structured logging

# Disable unused plugins to reduce attack surface
disabled_plugins = ["io.containerd.snapshotter.v1.devicemapper"]

# CRI Plugin Configuration
[plugins."io.containerd.grpc.v1.cri"]
  # Sandbox image - use specific version, not latest
  sandbox_image = "registry.k8s.io/pause:3.9"
  
  # Max container log line size - prevents log explosion
  max_container_log_line_size = 16384

  # Cgroup configuration - REQUIRED for cgroup v2
  [plugins."io.containerd.grpc.v1.cri".cgroup]
    path = "/"

  # containerd runtime settings
  [plugins."io.containerd.grpc.v1.cri".containerd]
    default_runtime_name = "runc"
    snapshotter = "overlayfs"  # Best performance
    
    # Pull timeout - adjust based on image sizes
    image_pull_progress_timeout = "120s"
    image_pull_behavior = "missing"  # Only pull when missing

    # Runtime configuration
    [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc]
      runtime_type = "io.containerd.runc.v2"
      [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
        # CRITICAL: SystemdCgroup must be true for cgroup v2
        SystemdCgroup = true

# Registry Configuration
[plugins."io.containerd.grpc.v1.cri".registry]
  config_path = "/etc/containerd/certs.d"  # For separate registry configs
  
  # Registry mirrors for Docker Hub
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
    endpoint = ["https://mirror.gcr.io", "https://registry-1.docker.io"]

# Metrics Configuration
[metrics]
  address = "0.0.0.0:1338"
  enabled = true
  [metrics.prometheus]
    enabled = true

# Garbage Collection Configuration
[plugins."io.containerd.gc.v1.scheduler"]
  # Schedule GC every 5 minutes
  schedule = "*/5 * * * *"
  pause_duration = "1s"
            

Critical: The SystemdCgroup = true setting is mandatory on modern Linux distributions with cgroup v2. Without this, resource limits won't work correctly, and containers may fail to start. Always verify your cgroup version with mount | grep cgroup2.

Production Best Practices

Storage Optimization

Practice: Use SSD storage for /var/lib/containerd and allocate at least 50-100GB per node.

Why: Container operations are I/O intensive. SSD storage significantly improves image pull times, container startup times, and overall performance. Insufficient disk space leads to node pressure, pod evictions, and application failures.

Monitor: Set up alerts at 80% disk usage and configure automatic cleanup.

Registry Optimization

Practice: Configure registry mirrors and use a private registry cache.

Why: Registry mirrors reduce latency, avoid rate limits, and improve reliability. A private registry cache (like Harbor or JFrog) can significantly speed up image pulls in enterprise environments.

Implementation: Use config_path for per-registry configuration. Test mirror endpoints for availability.

Security Hardening

Practice: Enable seccomp, AppArmor/SELinux, and image signing verification.

Why: Defense in depth. seccomp limits syscalls, MAC policies restrict file and network access, and image signing prevents supply chain attacks. These controls reduce the impact of container escapes.

Implementation: Start with default profiles, then customize based on workload requirements. Test thoroughly in staging.

Observability

Practice: Enable metrics, set up structured logging, and implement distributed tracing.

Why: You can't improve what you can't measure. Metrics provide visibility into performance trends, logs help debug issues, and tracing shows request flows through the system.

Implementation: Use Prometheus for metrics, ELK/Loki for logs, and Jaeger for tracing. Set up dashboards and alerts for key metrics.

Automated Cleanup

Practice: Implement regular cleanup of unused images, snapshots, and content.

Why: Accumulated images and snapshots consume disk space and degrade performance. Automated cleanup prevents disk pressure and reduces operational overhead.

Implementation: Use cron jobs for ctr image prune --keep=10, ctr snapshot prune, and ctr content prune. Configure Kubernetes image GC for automatic cleanup.

Version Management

Practice: Pin containerd versions and test upgrades in staging.

Why: Uncontrolled updates can introduce breaking changes or performance regressions. Testing ensures compatibility and prevents surprises in production.

Implementation: Use package managers with version pinning. Document upgrade procedures. Maintain a rollback plan for each upgrade.

Monitoring Strategy

A comprehensive monitoring strategy is essential for production containerd deployments. Here are the key metrics to monitor and how to set them up:

Disk Usage Metrics

                # Key disk metrics to monitor
- /var/lib/containerd usage percentage
- Inode usage (critical for large numbers of small files)
- Snapshot directory size
- Content store size
- Log directory size

# Prometheus queries
node_filesystem_avail_bytes{mountpoint="/var/lib/containerd"}
node_filesystem_size_bytes{mountpoint="/var/lib/containerd"}
node_filesystem_files_avail{mountpoint="/var/lib/containerd"}
            

Performance Metrics

                # Container operation metrics
container_runtime_operations_total
container_runtime_operations_errors_total
container_runtime_operations_duration_seconds

# Image pull metrics
container_image_pull_duration_seconds
container_image_pull_count

# Container resource usage
container_memory_usage_bytes
container_cpu_usage_seconds_total
            

Alerting Rules

                # Critical alerts
- Disk usage > 85%: Immediate action required
- Container operation error rate > 5%: Service degradation
- Image pull duration > 60s: Network or registry issues
- Container restart count > 5 in 10m: Application instability

# Alertmanager configuration example
groups:
- name: containerd_alerts
  rules:
  - alert: ContainerDiskUsageHigh
    expr: (1 - (node_filesystem_avail_bytes{mountpoint="/var/lib/containerd"} / node_filesystem_size_bytes{mountpoint="/var/lib/containerd"})) * 100 > 85
    annotations:
      summary: "containerd disk usage is critical ({{ $value }}%)"
      runbook: "https://wiki.example.com/containerd-disk-cleanup"
            

Monitor proactively: Set up dashboards in Grafana to visualize these metrics. Review them regularly to identify trends before they become problems. Alert fatigue is real—focus on actionable alerts and avoid alerting on every minor fluctuation.

Security Checklist for Production

☑ Enable seccomp profiles
Restrict system calls to reduce attack surface

☑ Configure AppArmor or SELinux
Enforce MAC policies for containers

☑ Implement image signing (Cosign)
Verify image integrity and origin

☑ Drop unnecessary capabilities
Use minimal capability sets

☑ Use rootless mode where appropriate
Reduce impact of container escapes

☑ Enable audit logging
Track who accessed what and when

☑ Restrict registry access
Only allow trusted registries

☑ Configure network policies
Restrict ingress/egress traffic

☑ Regularly update containerd
Apply security patches promptly

☑ Scan images for vulnerabilities
Use Trivy, Clair, or similar tools

Security is layered: No single control is sufficient. Combine multiple controls for defense in depth. Regularly review and update security policies based on new threats and learnings.

Essential Operational Runbooks

Document these procedures for your team to ensure consistent and effective incident response:

Runbook: Disk Space Emergency

Symptoms: Pod evictions, image pull failures, "no space left on device" errors.

Immediate actions:

Check disk usage: df -h /var/lib/containerd
Prune images: ctr image prune --keep=5 -f
Prune snapshots: ctr snapshot prune -f
Prune content: ctr content prune -f
Check logs: journalctl -u containerd -n 50

Prevention: Schedule regular cleanup, monitor disk usage, configure Kubernetes image GC.

Runbook: Image Pull Failures

Symptoms: Pods stuck in "ContainerCreating" state, failed to pull image errors.

Diagnostic steps:

Check containerd logs: journalctl -u containerd | grep -i pull
Test connectivity: curl -v https://registry-1.docker.io/v2/
Check DNS: nslookup registry-1.docker.io
Verify credentials: Check config.toml registry auth
Test with ctr: ctr image pull --debug <image>

Escalation: If issue persists, check network firewall, proxy settings, or registry status.

Runbook: Container Startup Failures

Symptoms: Containers exit immediately or fail to start.

Diagnostic steps:

Check container logs: crictl logs <container-id>
Check containerd logs: journalctl -u containerd -f
Verify runtime: runc --version
Check cgroups: cat /proc/self/cgroup
Check SELinux/AppArmor: sestatus or sudo aa-status

Common fixes: Set SystemdCgroup = true, disable SELinux temporarily for testing, increase memory limits.

Frequently Asked Questions

What's the minimum disk space for production containerd?

For production, allocate at least 50GB for /var/lib/containerd. For large deployments with many images, 100GB or more is recommended. Monitor usage patterns to determine the right size. Always have a cleanup strategy in place.

Should I use systemd or runc as the runtime?

Use runc with SystemdCgroup = true. This gives you the best of both worlds: runc's stability and systemd's cgroup management. For cgroup v2, this is the only supported configuration.

How do I handle containerd upgrades in production?

Follow a phased approach: test in staging, then deploy to a single node, monitor for 24-48 hours, then gradually roll out to all nodes. Always have a rollback plan. Use node draining before upgrades and verify pod stability after.

What's the best practice for containerd logging in production?

Use log_level = "info" (not debug). Configure log_format = "json" for structured logging. Set up log rotation to prevent disk exhaustion. Forward logs to a central logging system (ELK, Loki) for analysis and debugging.

How do I secure containerd in a multi-tenant environment?

Enable seccomp, AppArmor/SELinux, and use namespaces for isolation. Implement image signing verification. Restrict registry access to trusted sources. Use network policies to limit container communication. Consider rootless mode for additional security.

What's the recommended monitoring stack for containerd?

Use Prometheus for metrics collection, Grafana for visualization, and Alertmanager for alerts. Forward logs to ELK or Loki. This combination provides comprehensive observability and helps you detect issues early.

How often should I run automated cleanup?

For high-churn environments (multiple deployments per day), run cleanup daily. For stable environments, weekly may be sufficient. The goal is to keep disk usage below 80%. Adjust frequency based on usage patterns and available disk space.

What should be in a production deployment checklist?

Include: capacity planning, configuration validation, security controls, monitoring setup, cleanup automation, backup and recovery procedures, upgrade documentation, and incident response runbooks. Review and update the checklist regularly based on operational experience.

Previous: Interview Questions Next: Cheatsheet

A well-planned containerd production deployment is the foundation of reliable container infrastructure. Follow these best practices, monitor continuously, and iterate based on operational experience.