containerd Common Issues & Fixes
A comprehensive guide to the most common containerd issues and their solutions. From image pull failures to OCI runtime errors, find the fix you need to get your containers running again.
Common Issues
Solutions
Best Practices
Understanding containerd Issues
containerd is a robust and mature container runtime, but like any complex system, it can encounter issues. Most problems fall into several categories: image pulling problems, runtime execution errors, resource constraints, configuration issues, and network connectivity problems.
This guide covers the most frequently encountered issues and provides step-by-step solutions. Always start by checking the containerd logs (journalctl -u containerd) to identify the specific error message.
Before troubleshooting, ensure you're running the latest stable version of containerd. Many issues are resolved in newer releases.
Quick Reference: Common Issues at a Glance
Image Pull Failures
Cause: Registry auth, network, wrong tag
Check credentials, connectivity, image name
OCI Runtime Errors
Cause: runc issues, cgroup problems, kernel
Verify runc installation, check cgroups
Disk Space Issues
Cause: Full disk, inode exhaustion
Prune images, clean logs, resize volume
Permission Issues
Cause: SELinux, AppArmor, file permissions
Check security policies, fix file ownership
Network Issues
Cause: DNS, proxy, firewall, CNI
Check DNS, configure proxy, verify CNI
Configuration Issues
Cause: Syntax errors, invalid paths
Validate config with `containerd config validate`
1. Image Pull Failures
Error: "failed to pull image: no such image"
Cause: Image doesn't exist, wrong tag, or registry issue.
# Check if image exists
crane manifest /:
# Test with ctr
ctr image pull --debug /:
# Check registry mirrors
containerd config dump | grep -A 5 registry.mirrors
Error: "failed to pull image: unauthorized"
Cause: Missing or invalid registry credentials.
# Set credentials in config.toml
[plugins."io.containerd.grpc.v1.cri".registry.configs."docker.io".auth]
username = "your-user"
password = "your-token"
# Or use environment variables
export REGISTRY_AUTH=
# Login to registry
ctr image pull --user user:password
Error: "failed to pull image: dial tcp: lookup registry-1.docker.io"
Cause: DNS resolution failure or network connectivity.
# Check DNS
nslookup registry-1.docker.io
dig registry-1.docker.io
# Check network connectivity
curl -v https://registry-1.docker.io/v2/
# Configure DNS in /etc/resolv.conf
nameserver 8.8.8.8
nameserver 1.1.1.1
2. OCI Runtime Errors
Error: "failed to create shim: no such file or directory"
Cause: runc or containerd-shim binary missing.
# Check runc installation
which runc
runc --version
# Install runc
apt-get install -y runc
# or
curl -L https://github.com/opencontainers/runc/releases/latest/download/runc.amd64 -o /usr/local/bin/runc
chmod +x /usr/local/bin/runc
# Check shim binary
ls -la /usr/local/bin/containerd-shim-*
Error: "failed to run container: cannot allocate memory"
Cause: System out of memory or cgroup limits.
# Check memory usage
free -h
cat /proc/meminfo
# Check cgroup memory limit
cat /sys/fs/cgroup/memory/memory.limit_in_bytes
# Check container memory limit
crictl inspect | grep -A 5 "memory"
# Increase system swap or add more RAM
Error: "failed to run container: operation not permitted"
Cause: Security policy (SELinux/AppArmor) or capability issues.
# Check SELinux
sestatus
setenforce 0 # Temporarily disable for testing
# Check AppArmor
sudo aa-status
sudo aa-disable
# Check capabilities in config
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
Capabilities = ["NET_BIND_SERVICE", "CHOWN"]
3. Disk Space Issues
Error: "failed to pull image: no space left on device"
Cause: Disk full or inode exhaustion.
# Check disk usage
df -h /var/lib/containerd
df -i /var/lib/containerd
# Prune unused images
ctr image prune
ctr image prune --keep=5 # Keep last 5 images
# Prune snapshots
ctr snapshot prune
# Clean up all unused data
ctr image prune -f
ctr snapshot prune -f
ctr content prune -f
# Check large files
du -sh /var/lib/containerd/* | sort -h
Set up automatic cleanup with a cron job or use Kubernetes image garbage collection. Regular pruning prevents disk space issues.
4. Permission Issues
Error: "failed to create container: permission denied"
Cause: File permission issues or SELinux/AppArmor blocking access.
# Check containerd directory permissions
ls -la /var/lib/containerd
ls -la /run/containerd
# Fix permissions
sudo chown -R root:root /var/lib/containerd
sudo chmod 755 /var/lib/containerd
# Check SELinux context
ls -Z /var/lib/containerd
sudo restorecon -R /var/lib/containerd
# Check if user is in the correct group
groups $USER
Error: "failed to run container: cgroup creation failed"
Cause: Cgroup permissions or configuration.
# Check cgroup version
mount | grep cgroup
# For cgroup v2, set SystemdCgroup = true
[plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options]
SystemdCgroup = true
# Check cgroup permissions
ls -la /sys/fs/cgroup
5. Network Issues
Error: "failed to pull image: dial tcp: i/o timeout"
Cause: Network connectivity, proxy configuration, or firewall.
# Check network connectivity
ping 8.8.8.8
curl -v https://registry-1.docker.io/v2/
# Configure proxy in config.toml
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["http://proxy:5000"]
# Or set environment variables
export HTTP_PROXY=http://proxy:8080
export HTTPS_PROXY=http://proxy:8080
Error: "failed to run container: CNI network not found"
Cause: CNI configuration missing or invalid.
# Check CNI plugins
ls -la /opt/cni/bin/
# Check CNI configuration
ls -la /etc/cni/net.d/
# Test CNI configuration
crictl runp --pod-config pod-config.yaml
6. Configuration Issues
Error: "failed to load config: toml: syntax error"
Cause: TOML syntax error in config.toml.
# Validate configuration
containerd config validate
# Test configuration
containerd config dump
# Check TOML syntax
# Look for:
# - Missing quotes
# - Invalid section names
# - Extra commas
# - Unclosed brackets
# Reset to default config
containerd config default > /etc/containerd/config.toml
Error: "failed to start containerd: address already in use"
Cause: Another process using the containerd socket.
# Check socket
sudo lsof /run/containerd/containerd.sock
# Find process using socket
sudo netstat -lnp | grep containerd
# Stop conflicting process
sudo systemctl stop
# Kill conflicting process
sudo kill -9
Troubleshooting Checklist
- Check containerd status -
systemctl status containerd - View logs -
journalctl -u containerd -n 50 - Check system resources - Memory, CPU, disk space
- Verify runc installation -
runc --version - Check configuration -
containerd config validate - Test image pulling -
ctr image pull --debug nginx:latest - Check network connectivity -
curl -v https://registry-1.docker.io/v2/ - Check SELinux/AppArmor - Security policies may block operations
- Check cgroups -
cat /proc/self/cgroup - Check permissions -
ls -la /var/lib/containerd
Following this checklist systematically will resolve 90% of containerd issues. Document your findings for future reference.
Prevention Best Practices
- Regular updates - Keep containerd and runc updated.
- Monitor disk space - Set up alerts for disk usage.
- Configure log rotation - Prevent log files from filling disk.
- Use image pruning - Automatically remove unused images.
- Test configuration changes - Validate before applying.
- Backup configuration - Keep backups of config.toml.
- Monitor metrics - Use Prometheus for early detection.
- Implement health checks - Monitor containerd health endpoints.
Frequently Asked Questions
Why can't containerd pull images from Docker Hub?
Check network connectivity, DNS resolution, and registry credentials. Also verify Docker Hub is not rate-limiting your IP address. Configure registry mirrors to improve reliability.
What does "failed to create shim" mean?
The containerd shim process couldn't start. Usually caused by missing runc binary, incorrect runtime configuration, or permission issues. Check runc installation and verify runtime_type in config.toml.
How do I recover from disk space issues?
Run `ctr image prune`, `ctr snapshot prune`, and `ctr content prune`. If urgent, manually delete old images from `/var/lib/containerd/io.containerd.content.v1.content/blobs`.
Why do containers fail with "operation not permitted"?
This is often caused by SELinux or AppArmor policies blocking container operations. Temporarily disable them for testing, or create appropriate policies for your workload.
How do I debug image pull failures?
Enable debug logging, use `ctr image pull --debug`, check registry credentials, verify network connectivity, and test with `curl` or `crane` to isolate the issue.
What should I do if containerd won't start?
Check journalctl logs, validate config.toml, ensure runc is installed, check for socket conflicts, and verify system resources. Start with `containerd --log-level debug` for more details.
How do I fix cgroup v2 issues?
Set `SystemdCgroup = true` in the runc options in config.toml. Ensure your system uses cgroup v2 (check with `mount | grep cgroup2`).
Why are images not being garbage collected?
containerd only garbage collects unused images. Images used by running containers or with active references won't be removed. Check GC settings in config.toml and adjust schedule if needed.
Most containerd issues are solvable with the right approach. Use the logs, follow the checklist, and don't hesitate to consult the community for help.