containerd Logs & Debugging
Learn to debug containerd daemon issues effectively. This guide covers log analysis, common error messages, debugging techniques, and troubleshooting container runtime problems.
Log Analysis
Debugging
Troubleshooting
Understanding containerd Logging
containerd generates logs that provide insights into the daemon's operation, errors, and performance. Effective log analysis is the first step in debugging any containerd issue.
Logs are captured by systemd (journald) and can be viewed with `journalctl`. The log level can be configured in `config.toml` to control verbosity.
For production, use `info` or `warn` log levels. Use `debug` only during troubleshooting as it generates significant log volume.
Viewing containerd Logs
# View containerd daemon logs
sudo journalctl -u containerd -f
# View recent logs (last 100 lines)
sudo journalctl -u containerd -n 100
# View logs from the last hour
sudo journalctl -u containerd --since "1 hour ago"
# View logs with timestamps and human-readable format
sudo journalctl -u containerd --output short-full
# View logs in JSON format (for parsing)
sudo journalctl -u containerd --output json
# View specific time range
sudo journalctl -u containerd --since "2025-01-15 10:00:00" --until "2025-01-15 11:00:00"
# View logs with debug level
sudo journalctl -u containerd -p debug
# Follow logs with grep filtering
sudo journalctl -u containerd -f | grep -i error
# Check if containerd is running
sudo systemctl status containerd
Log Levels and Configuration
containerd supports multiple log levels. Configure them in `config.toml` to control verbosity.
# Configure log level in config.toml
# Valid levels: trace, debug, info, warn, error, fatal, panic
log_level = "info"
log_format = "text" # text or json
# Enable debug logging for troubleshooting
log_level = "debug"
# Set log level via command line
containerd --log-level debug
# View logs at specific level
sudo journalctl -u containerd -p info
Trace
Most verbose. All function calls and internal operations.
Debug
Detailed debug information. Use for troubleshooting specific issues.
Info
Default. General operational information.
Warn
Non-critical issues that deserve attention.
Error
Critical issues that may affect functionality.
Fatal
Critical errors causing containerd to exit.
Common Error Messages and Solutions
Error: "failed to start container: context deadline exceeded"
Container startup timed out. Usually caused by slow image pulls or resource constraints.
# Increase timeout in config.toml
[plugins."io.containerd.grpc.v1.cri".containerd]
image_pull_progress_timeout = "120s"
# or
[plugins."io.containerd.grpc.v1.cri".containerd]
image_pull_progress_timeout = "300s"
Error: "failed to pull image: no such image"
Image not found in registry. Check image name, tag, and registry credentials.
# Check image exists
crane manifest
# Verify registry credentials
ctr image pull --user user:password
# Check registry configuration
containerd config dump | grep -A 10 registry
Error: "failed to start container: cannot allocate memory"
System out of memory. Check container memory limits and node capacity.
# Check memory usage
free -h
top
# Check container limits
crictl inspect | grep memory
# Check systemd cgroup
cat /sys/fs/cgroup/memory/memory.limit_in_bytes
Error: "failed to create shim: no such file or directory"
Runc or shim binary missing. Check containerd runtime configuration.
# Check runtime binaries
which runc
ls -la /usr/local/bin/runc
# Verify runtime path in config
containerd config dump | grep runtime_type
# Reinstall runc
apt-get install -y runc
Error: "failed to pull image: TLS handshake timeout"
Network connectivity issues to registry. Check network, proxies, and DNS.
# Test registry connectivity
curl -v https://registry-1.docker.io/v2/
# Check DNS
nslookup registry-1.docker.io
# Configure proxy in containerd
[plugins."io.containerd.grpc.v1.cri".registry]
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
endpoint = ["http://proxy:5000"]
Error: "failed to run container: permission denied"
Permission issues with filesystem or cgroups. Check user permissions and SELinux/AppArmor.
# Check permissions
ls -la /var/lib/containerd
# Check SELinux
sestatus
setenforce 0 # Temporarily disable (testing)
# Check AppArmor
sudo aa-status
Debugging Techniques
1. Enable Debug Logging
# Temporarily enable debug
sudo sed -i 's/log_level = "info"/log_level = "debug"/' /etc/containerd/config.toml
sudo systemctl restart containerd
# View debug logs
sudo journalctl -u containerd -f
2. Use containerd Command-Line Tools
# List containers
ctr container list
crictl ps -a
# Check container logs
crictl logs
# Inspect container
ctr container info
crictl inspect
# Check images
ctr image list
crictl images
# Pull image with verbose output
ctr image pull --debug docker.io/nginx:latest
3. Debug with strace
# Trace containerd daemon
sudo strace -p $(pidof containerd) -o /tmp/containerd-strace.log
# Trace a specific container process
sudo strace -p $(crictl inspect | jq .info.pid) -o /tmp/container-strace.log
4. Debug with crictl
# Get detailed container info
crictl inspect -v
# Get container stats
crictl stats
# Execute a command in container
crictl exec -it /bin/sh
# Check container events
crictl events
5. Check containerd State
# Check containerd status
sudo systemctl status containerd
# Check containerd socket
sudo ls -la /run/containerd/containerd.sock
# Check containerd processes
ps aux | grep containerd
# Check open file descriptors
sudo lsof -p $(pidof containerd)
Troubleshooting Workflow
- Check containerd status - Is the daemon running?
systemctl status containerd - View recent logs -
journalctl -u containerd -n 50 - Check for errors -
journalctl -u containerd | grep -i error - Enable debug logging - Temporarily set log_level to debug
- Check system resources - Memory, CPU, disk space
- Verify network connectivity - Registry access, DNS
- Check runtime configuration -
containerd config dump - Check SELinux/AppArmor - Security policies may block operations
- Check cgroups -
cat /proc/self/cgroup - Check storage -
df -h /var/lib/containerd
Follow this workflow systematically. Don't skip steps—each provides valuable diagnostic information.
Essential Debugging Tools
journalctl
View systemd logs for containerd daemon.
ctr
Native containerd CLI for low-level operations.
crictl
CRI-compatible CLI for Kubernetes pod/container debugging.
strace
Trace system calls for debugging process execution.
top/htop
Monitor system resource usage.
tcpdump
Network packet capture for debugging connectivity issues.
jq
Parse JSON output from containerd commands.
curl
Test registry connectivity and API endpoints.
Performance Debugging
# Check containerd CPU/memory usage
top -p $(pgrep containerd)
# Check container runtime operations
curl http://localhost:1338/metrics | grep container_runtime_operations
# Check image pull latency
curl http://localhost:1338/metrics | grep container_image_pull_duration
# Check containerd goroutines
curl http://localhost:1338/metrics | grep go_goroutines
# Check GC performance
curl http://localhost:1338/metrics | grep containerd_gc
# Profile containerd (pprof)
go tool pprof http://localhost:1338/debug/pprof/heap
Frequently Asked Questions
Where are containerd logs stored?
containerd logs are stored in the system journal (journald). Use `journalctl -u containerd` to view them. They are not stored in files by default.
How do I increase log verbosity?
Set `log_level = "debug"` in `/etc/containerd/config.toml` and restart containerd. For temporary debugging, use `containerd --log-level debug`.
Why is containerd not starting?
Check journalctl logs for errors. Common causes: invalid config.toml, port conflicts, missing runc, permission issues, or cgroup problems.
What does "context deadline exceeded" mean?
An operation timed out. Usually image pull or container startup taking too long. Increase `image_pull_progress_timeout` or check network connectivity.
How do I debug image pull failures?
Enable debug logging, check registry credentials, verify image name and tag, check network connectivity, and test with `ctr image pull --debug`.
What should I check if containers won't start?
Check containerd logs, system resources (memory, disk), cgroup configuration, runtime binaries (runc), and SELinux/AppArmor policies.
How can I debug containerd performance issues?
Enable metrics endpoint, use Prometheus to collect metrics, check container operation latency, and profile containerd with pprof.
What's the difference between ctr and crictl for debugging?
ctr is for low-level containerd debugging without CRI. crictl works with the CRI plugin and is more useful for Kubernetes container debugging.
Effective debugging is a skill. Start with the logs, isolate the problem, and use the right tools for the job. With practice, you'll quickly resolve most containerd issues.