containerd Disk Space Cleanup

Image Pruning Snapshot Cleanup Content Pruning Log Rotation

A complete guide to cleaning up disk space in containerd. Learn how to prune images, snapshots, content, configure log rotation, and implement automated cleanup strategies with detailed explanations for each approach.

Prune Images Clean Snapshots Content Cleanup Log Rotation

Why Disk Space Management Matters

containerd stores container images, snapshots, and content in directories on your host system. Over time, these can consume significant disk space, leading to:

Image pull failures - "no space left on device" errors
Performance degradation - Slower container startup and operations
System instability - Pod evictions and node pressure
Increased costs - For cloud instances with limited storage

Regular cleanup is essential for maintaining healthy container infrastructure. This guide covers everything you need to know about managing containerd's storage footprint.

Understanding containerd Storage: containerd stores data in three main areas:

/var/lib/containerd/io.containerd.content.v1.content/ → Blob storage (image layers)
/var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/ → Snapshot storage
/var/lib/containerd/io.containerd.metadata.v1.db/ → Metadata database

1. Pruning Unused Images

Images consume the most disk space in containerd. Each image pull downloads layers that are stored as blobs. Over time, unused images accumulate, wasting valuable storage.

ctr image prune

What it does: Removes all images that are not currently in use by any running container. This is the safest cleanup operation because it only removes images that have no active references.

When to use: Regular maintenance, typically daily or weekly depending on image churn.

What it doesn't remove: Images that are tagged, images referenced by stopped containers, or images currently being pulled.

ctr image prune --keep=5

What it does: Keeps the 5 most recently used images and removes the rest. The "--keep" flag specifies the number of images to retain based on last access time.

When to use: When you want to retain a cache of frequently used images to avoid repeated pulls.

Best practice: Use this in production to maintain a balance between disk space and performance.

ctr image prune -f

What it does: Forces image pruning without prompting for confirmation. This is useful for automated scripts and cron jobs.

When to use: In CI/CD pipelines or scheduled maintenance scripts where human intervention isn't possible.

Caution: Always test with --dry-run first to understand what will be removed.

                # Check disk space before cleanup
df -h /var/lib/containerd

# List all images and their sizes
ctr image list -q | xargs -I {} ctr image ls -q {}
# or use crictl
crictl images

# Dry run - see what would be removed
ctr image prune --dry-run

# Actual prune
ctr image prune --keep=5

# Check disk space after cleanup
df -h /var/lib/containerd
            

2. Cleaning Up Snapshots

Snapshots are filesystem layers created when images are pulled or containers are started. Each container run creates at least one snapshot. Over time, snapshots from stopped or removed containers can accumulate and consume significant space.

ctr snapshot prune

What it does: Removes snapshots that are no longer referenced by any image or container. This includes snapshots from containers that have been deleted.

When to use: After deleting containers or images, to clean up leftover snapshot layers.

Why it matters: Snapshots can consume as much space as the image itself. Cleaning them up is essential for recovering disk space.

ctr snapshot list

What it does: Lists all existing snapshots with their sizes, helping you identify which snapshots are consuming the most space.

When to use: Before running prune to understand what will be removed.

Output interpretation: Parent snapshots are the base layers, children are derived from them. Removing a parent snapshot will also remove all its children.

                # List all snapshots with sizes
ctr snapshot list

# Dry run - preview what will be pruned
ctr snapshot prune --dry-run

# Actual prune
ctr snapshot prune

# Force prune without confirmation
ctr snapshot prune -f

# Check snapshot directory size
du -sh /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/
            

Important: Never manually delete files from /var/lib/containerd/io.containerd.snapshotter.v1.overlayfs/. Always use the ctr snapshot prune command to ensure metadata consistency.

3. Pruning Content (Blob Storage)

The content store holds the actual blob data for image layers. These are immutable objects identified by their content digest. When images are pulled, the blobs are downloaded and stored here.

ctr content prune

What it does: Removes content blobs that are no longer referenced by any image manifest or snapshot. This is the most aggressive cleanup operation and recovers the most space.

When to use: After image and snapshot pruning, to remove orphaned blobs.

Why it's important: Blobs are immutable, so they never get cleaned up automatically unless content is pruned. This is where the majority of disk space is consumed.

ctr content list

What it does: Lists all content blobs with their digests and sizes, showing you what's currently stored in the content store.

When to use: To audit what content is stored and identify large blobs that might be candidates for removal.

Output format: Each blob is identified by its SHA-256 digest, which corresponds to the image layer content.

                # List content blobs with sizes
ctr content list

# Dry run - preview what will be pruned
ctr content prune --dry-run

# Actual prune
ctr content prune

# Force prune without confirmation
ctr content prune -f

# Check content store size
du -sh /var/lib/containerd/io.containerd.content.v1.content/
            

Recommended order for cleanup:

Prune images (ctr image prune)
Prune snapshots (ctr snapshot prune)
Prune content (ctr content prune)

This ensures all references are properly cleaned up and no orphaned data remains.

4. Cleanup Using crictl (Kubernetes)

If you're using containerd in a Kubernetes cluster, crictl provides Kubernetes-aware cleanup commands that consider pod state.

crictl rmi --prune

What it does: Removes all images that are not referenced by any running pod or container. This is the Kubernetes equivalent of ctr image prune.

When to use: On Kubernetes nodes to clean up images that aren't needed by running workloads.

Advantage: crictl understands Kubernetes pod lifecycle, so it's safer in cluster environments.

crictl rmi --force

What it does: Forcefully removes a specific image even if it's in use. Use with extreme caution.

When to use: When an image is corrupted or you need to free space urgently.

Caution: Force removal can cause running pods to fail if they're using the image.

                # List images with crictl
crictl images

# Remove unused images
crictl rmi --prune

# Remove a specific image
crictl rmi 

# Check disk usage of images
crictl images -v

# List stopped containers and their images
crictl ps -a --state Exited
            

5. Log Rotation and Cleanup

Container logs can consume significant disk space, especially in high-traffic environments. Proper log rotation is essential for preventing disk space exhaustion.

logrotate configuration

What it does: Automatically rotates, compresses, and removes old log files. This prevents logs from growing indefinitely.

Why it matters: A single container logging at high volume can fill a disk in hours. Log rotation is a critical operational practice.

Best practice: Configure logrotate with daily rotation, 7-30 days retention, and compression.

                # /etc/logrotate.d/containerd-logs
/var/log/containers/*.log {
    daily
    rotate 7
    compress
    delaycompress
    missingok
    notifempty
    copytruncate
    maxsize 100M
    create 0640 root root
}

# Apply logrotate manually
sudo logrotate -f /etc/logrotate.d/containerd-logs

# Check log directory size
du -sh /var/log/containers/

# Count log files
ls -la /var/log/containers/ | wc -l

# Remove logs older than 7 days
find /var/log/containers/ -name "*.log" -mtime +7 -delete

# Journald log cleanup (systemd)
sudo journalctl --vacuum-size=500M
sudo journalctl --vacuum-time=7d
            

6. Automated Cleanup Strategies

Manual cleanup is error-prone and easy to forget. Automating disk space cleanup ensures your nodes remain healthy without human intervention.

Cron job for automated cleanup

What it does: Runs cleanup commands on a schedule (daily, weekly) to automatically remove unused data.

Implementation: Create a shell script and schedule it with cron.

Best practice: Run cleanup during off-peak hours to minimize impact on workloads.

                # /usr/local/bin/containerd-cleanup.sh
#!/bin/bash
# Log cleanup operations
echo "Starting containerd cleanup at $(date)" >> /var/log/containerd-cleanup.log

# Prune images (keep last 10)
ctr image prune --keep=10 -f >> /var/log/containerd-cleanup.log 2>&1

# Prune snapshots
ctr snapshot prune -f >> /var/log/containerd-cleanup.log 2>&1

# Prune content
ctr content prune -f >> /var/log/containerd-cleanup.log 2>&1

# Clean logs older than 7 days
find /var/log/containers/ -name "*.log" -mtime +7 -delete

echo "Cleanup completed at $(date)" >> /var/log/containerd-cleanup.log

# Make script executable
chmod +x /usr/local/bin/containerd-cleanup.sh

# Add to crontab (daily at 2 AM)
# 0 2 * * * /usr/local/bin/containerd-cleanup.sh

# Or use systemd timer for more control
# /etc/systemd/system/containerd-cleanup.timer
[Unit]
Description=containerd cleanup timer

[Timer]
OnCalendar=daily
Persistent=true

[Install]
WantedBy=timers.target
            

Kubernetes Image Garbage Collection

What it does: Kubernetes built-in feature that automatically removes unused images from nodes.

Configuration: Set via kubelet arguments or in the kubelet config file.

Advantages: Integrated with Kubernetes, respects pod lifecycle, and runs automatically.

                # kubelet configuration for image GC
# /var/lib/kubelet/config.yaml
apiVersion: kubelet.config.k8s.io/v1beta1
kind: KubeletConfiguration
imageGCHighThresholdPercent: 85   # Start GC when disk usage > 85%
imageGCLowThresholdPercent: 80    # Stop GC when disk usage falls to 80%
imageMinimumGCAge: 2m              # Minimum age for images to be eligible for GC

# Or via command line
--image-gc-high-threshold=85
--image-gc-low-threshold=80
--minimum-image-ttl-duration=2m
            

7. Monitoring Disk Space

Proactive monitoring prevents disk space emergencies. Set up alerts to notify you before disk space becomes critical.

                # Check disk usage of containerd directories
df -h /var/lib/containerd

# Check usage by subdirectory
du -sh /var/lib/containerd/* | sort -h

# Check inode usage (critical for large numbers of small files)
df -i /var/lib/containerd

# Prometheus metrics for disk space
node_filesystem_avail_bytes{device="/var/lib/containerd"}
node_filesystem_size_bytes{device="/var/lib/containerd"}

# Alert when disk usage exceeds 80%
# Alertmanager configuration
- alert: ContainerDiskUsageHigh
  expr: (1 - (node_filesystem_avail_bytes{mountpoint="/var/lib/containerd"} / node_filesystem_size_bytes{mountpoint="/var/lib/containerd"})) * 100 > 80
  annotations:
    summary: "containerd disk usage is high ({{ $value }}%)"
            

Best Practices Summary:

Run ctr image prune --keep=10 daily
Run ctr snapshot prune and ctr content prune weekly
Configure log rotation with 7-14 days retention
Set up monitoring and alerts at 80% disk usage
Use Kubernetes image GC for automatic cleanup
Test cleanup scripts in staging before production

Frequently Asked Questions

What's the difference between image prune and content prune?

Image prune removes images that aren't referenced by any container. Content prune removes the underlying blob data that's no longer referenced by any image. Content prune is more aggressive and recovers more space, but should only be run after image prune.

Will pruning affect running containers?

No. Prune commands only remove data that is not currently in use. Running containers and their images are protected and won't be removed. However, force operations (-f) may remove data that's still referenced if used incorrectly.

How often should I run cleanup?

Depends on your environment. For high-churn clusters (multiple deployments per day), run daily. For stable environments, weekly may suffice. Monitor disk usage to determine the right frequency for your needs.

Can I recover pruned images?

No. Once images are pruned, they must be re-pulled from the registry. Ensure you only prune images that are available in your registry and can be re-pulled if needed.

Why is my disk still full after pruning?

Check other potential culprits: container logs in /var/log/containers/, orphaned content, or filesystem inode exhaustion. Also verify that the prune commands were successful and didn't produce errors.

What's the minimum disk space for containerd?

For production, allocate at least 50GB for /var/lib/containerd. For large deployments with many images, 100GB or more is recommended. Monitor usage and adjust accordingly.

Does containerd support automatic garbage collection?

Yes, containerd has built-in GC for snapshots and content. It runs periodically based on configuration. However, automatic GC may not be aggressive enough for high-volume environments, so manual or scheduled pruning is still recommended.

How do I handle disk space issues in a Kubernetes cluster?

Configure kubelet with image GC thresholds, run crictl rmi --prune on nodes, set up cron jobs for regular cleanup, and monitor disk usage with Prometheus. Consider using node pressure eviction to handle severe disk space issues.

Previous: Common Issues & Fixes Next: Interview Questions

Regular disk space cleanup is essential for maintaining healthy container infrastructure. Implement these practices proactively to prevent disk space emergencies.