containerd Interview Questions
Ace your container runtime interview with this comprehensive Q&A guide covering containerd architecture, CRI, troubleshooting, performance, and real-world scenarios with detailed explanations for each answer.
These questions test fundamental understanding of containerd and container runtimes. They're typically asked in junior or mid-level interviews.
Answer: containerd is an industry-standard container runtime that manages the complete container lifecycle. It was originally built by Docker and donated to the CNCF, where it became a graduated project. It's responsible for pulling images, creating and starting containers, managing storage, and handling container lifecycle operations.
Answer: CRI is a Kubernetes API that defines how kubelet interacts with container runtimes. containerd implements CRI through the io.containerd.grpc.v1.cri plugin, which translates Kubernetes pod and container requests into containerd operations.
Answer: containerd has a modular architecture with these main components:
- GRPC Server: Handles API requests from clients
- Content Store: Manages OCI image blobs
- Snapshotter: Manages filesystem snapshots (overlayfs, etc.)
- Metadata Store: Tracks image and container metadata
- Runtime: Executes containers (runc by default)
- CRI Plugin: Kubernetes integration
Answer: They operate at different layers of the container stack:
- runc: The low-level OCI runtime that actually creates and runs containers using Linux kernel features (namespaces, cgroups).
- containerd: A high-level runtime that manages the complete container lifecycle, including image pulling, storage, and runtime execution. It uses runc under the hood.
- Docker: A complete container platform that includes containerd as its runtime, plus additional tools for building images, networking, and orchestration.
Answer: The pause image is a minimal container that holds the network namespace for each Kubernetes pod. It's the first container started in a pod and maintains the pod's network stack while other containers are started or restarted.
registry.k8s.io/pause:3.9.
Answer: Use these commands:
# Check status
sudo systemctl status containerd
# View live logs
sudo journalctl -u containerd -f
# View recent logs
sudo journalctl -u containerd -n 50
# Check if socket exists
sudo ls -la /run/containerd/containerd.sock
journalctl -u kubelet to see kubelet errors related to containerd. The combination of both logs is often needed to diagnose issues.
Answer: containerd namespaces provide multi-tenancy isolation for containers, images, and snapshots. They're logical separations that allow different users or applications to use the same containerd instance without interfering with each other.
default (for general use), k8s.io (for Kubernetes CRI plugin), moby (for Docker). The CRI plugin uses k8s.io namespace, which is why ctr -n k8s.io images list shows Kubernetes images.
Answer: config.toml is the main configuration file for containerd, located at /etc/containerd/config.toml. It controls all aspects of containerd's behavior including:
- Logging levels and format
- Storage paths (root and state directories)
- CRI plugin settings (pause image, runtime selection)
- Registry mirrors and authentication
- Snapshotter selection (overlayfs, zfs, etc.)
- GC schedule and behavior
containerd config validate before restarting the service. Invalid config can prevent containerd from starting.
These questions test deeper understanding of containerd operations, troubleshooting, and integration with Kubernetes. Typically asked in senior engineer interviews.
Answer: Follow this systematic approach:
- Check logs:
journalctl -u containerd | grep -i "pull\|error" - Verify network connectivity:
curl -v https://registry-1.docker.io/v2/ - Check DNS resolution:
nslookup registry-1.docker.io - Validate credentials: Check registry auth in config.toml
- Test with ctr:
ctr image pull --debug <image> - Check disk space:
df -h /var/lib/containerd - Verify image exists:
crane manifest <image:tag>
image_pull_progress_timeout), invalid credentials (check registry.auth), rate limiting (use registry mirrors), or DNS issues (check /etc/resolv.conf).
Answer: containerd supports multiple snapshot drivers:
- overlayfs: Best performance, uses Linux kernel overlay filesystem. Recommended for most environments.
- native: Copies entire layers. Poor performance, only for testing.
- devicemapper: Legacy, uses thin provisioning. Not recommended for new deployments.
- zfs: For ZFS filesystems. Good for ZFS users with built-in compression.
- btrfs: For Btrfs filesystems. Good for Btrfs users with CoW features.
- overlay: Legacy overlay, use overlayfs instead.
snapshotter = "overlayfs" in config.toml.
Answer: Each tool serves different purposes:
- ctr: Native containerd CLI. Low-level debugging, direct access to containerd API. Use for troubleshooting without Kubernetes context.
- nerdctl: Docker-compatible CLI for containerd. Use for development when you want Docker-like experience with containerd. Supports compose.
- crictl: CRI-compatible CLI for Kubernetes. Use for debugging pods and containers on Kubernetes nodes. Works with kubelet's CRI.
crictl as it understands pod sandboxes and CRI concepts. For low-level containerd debugging, use ctr. For development, use nerdctl.
Answer: The containerd shim is a lightweight process that acts as a parent for each container. It serves several critical purposes:
- Process reaping: It reaps zombie processes from containers.
- Independence: Allows containerd daemon to restart without affecting running containers.
- Resource management: Maintains container state and handles I/O.
- Communication: Provides a stable API channel between containerd and the container process.
Answer: Configure registry in config.toml:
[plugins."io.containerd.grpc.v1.cri".registry]
# Registry configuration
[plugins."io.containerd.grpc.v1.cri".registry.mirrors."myregistry.io"]
endpoint = ["https://myregistry.io"]
# Auth configuration
[plugins."io.containerd.grpc.v1.cri".registry.configs."myregistry.io".auth]
username = "myuser"
password = "mypassword"
# Or use auth token
[plugins."io.containerd.grpc.v1.cri".registry.configs."myregistry.io".auth]
auth = "base64-encoded-credentials"
# TLS configuration
[plugins."io.containerd.grpc.v1.cri".registry.configs."myregistry.io".tls]
insecure_skip_verify = false
ca_file = "/etc/containerd/certs.d/ca.crt"
auth with base64-encoded credentials or, better yet, use Kubernetes image pull secrets mounted as files. Never hardcode credentials in config.toml for production.
Answer: SystemdCgroup controls how containerd manages cgroups:
- true: Uses systemd to manage cgroups. Required for cgroup v2. Provides better resource management and compatibility with systemd-based systems.
- false: Uses cgroupfs directly. Works with cgroup v1 but causes issues on cgroup v2.
SystemdCgroup = true is required. Without it, resource limits (CPU, memory) won't work correctly, and containers may fail to start. Always check your cgroup version with mount | grep cgroup2.
Answer: Systematic debugging approach:
- Check containerd logs:
journalctl -u containerd -f | grep -i oci - Verify runc installation:
runc --version - Check cgroup configuration:
SystemdCgroup = truefor cgroup v2 - Check SELinux/AppArmor:
sestatusorsudo aa-status - Check system resources: Memory, disk space, inodes
- Test with a simple image:
crictl run hello-world - Enable debug logging: Set
log_level = "debug"
Answer: containerd's garbage collection (GC) automatically removes unused resources:
- What it cleans: Unused snapshots, content blobs, and metadata references.
- How it works: Uses reference counting—resources are kept if referenced by images or containers. Unreferenced resources are marked for deletion.
- When it runs: Can be configured via
plugins."io.containerd.gc.v1.scheduler"in config.toml. - Manual trigger:
ctr image prune,ctr snapshot prune,ctr content prune.
These questions test deep expertise in containerd architecture, performance tuning, and complex troubleshooting. Typically asked in senior/principal or architect interviews.
Answer: The interaction flow is:
- kubelet receives pod specification from Kubernetes API server.
- kubelet sends CRI requests (CreatePodSandbox, CreateContainer, StartContainer) to containerd.
- CRI plugin in containerd translates CRI requests into containerd API calls.
- containerd manages the container lifecycle: pulls images, creates snapshots, starts runc processes.
- runc creates the actual container using Linux kernel features.
- containerd reports status back to kubelet via the CRI plugin.
Answer: Key considerations for production containerd:
- Storage: Use SSD storage for
/var/lib/containerd. Allocate at least 50-100GB per node. - Snapshotter: Use overlayfs for best performance.
- SystemdCgroup: Set to true for cgroup v2 compatibility.
- Logging: Set log_level to "info" (not "debug") in production. Configure log rotation.
- Registry: Configure registry mirrors for faster image pulls and rate limit avoidance.
- GC: Configure aggressive GC schedules or run regular cron jobs for cleanup.
- Monitoring: Enable metrics endpoint and monitor with Prometheus. Set up alerts for disk usage, operation failures.
- Security: Enable seccomp, AppArmor/SELinux, and rootless mode where appropriate.
Answer: This error indicates an operation timed out. Systematic approach:
- Identify the operation: Check logs to see what was timing out (image pull, container start, etc.).
- Image pull timeouts: Increase
image_pull_progress_timeoutin config.toml. Default is 5 minutes. - Network issues: Check registry connectivity, DNS, and proxy settings.
- Resource constraints: Check if system is under memory or CPU pressure.
- Slow storage: Check I/O latency on
/var/lib/containerd. - Large images: Large images may need more time to pull. Use smaller base images.
- GC issues: Slow GC can cause timeouts. Check GC configuration and performance.
strace to trace containerd operations, and use the metrics endpoint to monitor operation durations. Look for patterns—if timeouts occur at specific times, it might indicate resource contention.
Answer: Performance characteristics:
- overlayfs: Best performance (low CPU, low I/O). Uses kernel-level overlay.
- zfs/btrfs: Good performance, but with more CPU overhead for checksums and compression.
- native: Poor performance (copies entire layers). Avoid in production.
Benchmarking method:
# Measure image pull time
time ctr image pull docker.io/nginx:latest
# Measure container start time
time ctr run --rm docker.io/nginx:latest nginx-test
# Measure operation performance using metrics
curl http://localhost:1338/metrics | grep container_runtime_operations
# Use ioping for storage performance
ioping /var/lib/containerd
Answer: containerd uses a content-addressable storage system:
- Layers: Each Dockerfile instruction creates a layer. Layers are stored as blobs in the content store.
- Content addressing: Each blob is identified by its SHA-256 digest, enabling deduplication.
- Sharing: Identical layers across different images are stored once, saving disk space.
- Snapshots: Layers are mounted via the snapshot driver (overlayfs) to create the container root filesystem.
- Lazy pulling: containerd supports lazy pulling where layers are downloaded on-demand, reducing startup time.
Answer: Image signing workflow with Cosign:
- Install Cosign: Download from GitHub releases.
- Generate keys:
cosign generate-key-pair - Sign image:
cosign sign --key cosign.key ghcr.io/user/image:latest - Verification:
cosign verify --key cosign.pub ghcr.io/user/image:latest
containerd integration:
- Use admission controllers (ImagePolicyWebhook) to enforce verification.
- Implement policy-as-code with Gatekeeper/OPA.
- For manual verification, check signatures before pulling.
cosign sign with OIDC identity for GitHub Actions or GitLab CI. This provides better security and auditability.
These questions test practical problem-solving skills in real-world scenarios. They're commonly asked in senior-level interviews to assess hands-on experience.
Answer: Systematic resolution approach:
- Diagnose:
•df -h /var/lib/containerd- Check overall usage
•du -sh /var/lib/containerd/* | sort -h- Find large directories
•crictl images -v- List images with sizes - Immediate actions:
•ctr image prune --keep=5 -f- Remove unused images
•ctr snapshot prune -f- Clean snapshots
•ctr content prune -f- Remove orphaned blobs - Prevent recurrence:
• Configure Kubernetes image GC:--image-gc-high-threshold=85
• Set up cron jobs for daily cleanup
• Monitor disk usage with Prometheus alerts
Answer: This is typically a security policy issue. Diagnostic approach:
- Check SELinux:
•sestatus- Check if enforcing
•setenforce 0- Temporarily disable for testing
• If fixed, create SELinux policy for your container - Check AppArmor:
•sudo aa-status- List loaded profiles
•sudo aa-disable <profile>- Disable for testing - Check file permissions:
•ls -la /var/lib/containerd
•sudo chown -R root:root /var/lib/containerd - Check capabilities:
• Review required capabilities for your container
• Add specific capabilities if needed in config.toml
ausearch to find denied operations and audit2allow to generate SELinux policies. This maintains security while allowing your containers to run.
Answer: Migration plan:
- Preparation:
• Test containerd in staging environment first
• Validate all applications work with containerd
• Update Kubernetes configuration to use containerd
• Prepare rollback plan - Migration steps per node:
• Drain node:kubectl drain <node> --ignore-daemonsets
• Install containerd:apt-get install containerd
• Configure containerd: Generate config.toml
• Update kubelet: Set--container-runtime=remote --container-runtime-endpoint=unix:///run/containerd/containerd.sock
• Restart kubelet and containerd
• Uncordon:kubectl uncordon <node> - Post-migration:
• Verify pods are running
• Monitor logs for errors
• Remove Docker if no longer needed
Answer: Canary deployment strategy:
- Phase 1: Non-production testing
• Test in dev and staging environments
• Run performance benchmarks
• Test all application workloads - Phase 2: Single node canary
• Select one node in production
• Drain and upgrade containerd
• Monitor for 24-48 hours
• Watch metrics: operation latency, error rates - Phase 3: Gradual rollout
• Upgrade nodes in batches (e.g., 10% per day)
• Monitor closely after each batch
• Be ready to rollback if issues appear - Phase 4: Full rollout
• Complete all node upgrades
• Monitor for 1 week before declaring success
Answer: Diagnostic and optimization approach:
- Diagnose:
• Measure startup time:time ctr run --rm nginx
• Check image pull metrics:container_image_pull_duration_seconds
• Check snapshot creation time:ctr snapshot list - Optimize image pulls:
• Use smaller base images (Alpine instead of Ubuntu)
• Configure registry mirrors
• Pre-pull frequently used images on nodes
• Optimize layer order (put infrequently changing layers first) - Optimize container creation:
• Use overlayfs snapshotter (fastest)
• Ensure SystemdCgroup=true for cgroup v2
• Consider using containerd's lazy pulling feature - System-level optimizations:
• Use SSD storage for /var/lib/containerd
• Increase system memory if needed
• Optimize kubelet sync frequency
# Check containerd status
systemctl status containerd
# View logs
journalctl -u containerd -f
# List images
ctr image ls
crictl images
# List containers
ctr container ls
crictl ps -a
# Pull image
ctr image pull docker.io/nginx:latest
# Run container
ctr run --rm docker.io/nginx:latest nginx
# Prune images
ctr image prune --keep=5
# Prune snapshots
ctr snapshot prune
# Prune content
ctr content prune
# Check disk usage
df -h /var/lib/containerd
# Validate config
containerd config validate
# Dump config
containerd config dump
Preparation is key. Practice these questions, set up a test environment, and you'll be well-prepared for your containerd interview. Good luck!