Troubleshooting Pods

CrashLoopBackOff ImagePullBackOff OOMKilled Pending Pods

A comprehensive guide to troubleshooting Kubernetes pods covering CrashLoopBackOff, ImagePullBackOff, OOMKilled, pending pods, and common issues with practical diagnostic steps and solutions.

CrashLoopBackOff ImagePullBackOff OOMKilled Pending Pods

Why Pods Fail: Understanding Pod States

Pods can fail for many reasons. Understanding pod states and phases is the first step in troubleshooting. Kubernetes pods go through several phases:

Pending: Pod accepted but not scheduled
Running: Pod bound to a node and containers running
Succeeded: All containers terminated successfully
Failed: All containers terminated with failure
Unknown: Pod state unknown
CrashLoopBackOff: Container repeatedly crashes
ImagePullBackOff: Image cannot be pulled
OOMKilled: Container killed due to memory exhaustion

Systematic Debugging Approach:

Check pod status: kubectl get pods
Describe the pod: kubectl describe pod <pod>
Check pod logs: kubectl logs <pod>
Check previous logs: kubectl logs --previous <pod>
Check events: kubectl get events
Check node status: kubectl describe node <node>

1. CrashLoopBackOff: Container Crashing Repeatedly

CrashLoopBackOff means the container is crashing shortly after startup. Kubernetes will attempt to restart it, but the backoff delay increases with each failure.

Application Errors

Common Cause

Application code throwing exceptions, missing configuration, or failing during initialization. Check the pod logs for error messages.

Fix: Check logs with kubectl logs

Configuration Issues

Common Cause

Missing or invalid environment variables, ConfigMaps, or Secrets. Check configuration sources and ensure they exist.

Fix: Verify ConfigMaps, Secrets, and environment variables

Dependency Failures

Common Cause

Cannot connect to databases, message queues, or other services. Check network connectivity and service availability.

Fix: Verify service endpoints and network policies

Startup Timeout

Common Cause

Application takes too long to start, exceeding readiness or liveness probe thresholds. Adjust probe settings.

Fix: Increase initialDelaySeconds or failureThreshold

                # Diagnostic commands for CrashLoopBackOff
# Check pod status
kubectl get pods

# Describe pod for details
kubectl describe pod <pod-name>

# View current logs
kubectl logs <pod-name>

# View previous container logs (before restart)
kubectl logs --previous <pod-name>

# Check events for backoff reason
kubectl get events --field-selector involvedObject.name=<pod-name>

# Enter debugging container
kubectl debug -it <pod-name> --image=busybox --target=<container-name>

# Common fixes
# 1. Check application code for errors
# 2. Verify environment variables and ConfigMaps
# 3. Check database connectivity
# 4. Adjust liveness/readiness probes
# 5. Increase memory limits if OOMKilled
            

CrashLoopBackOff Solutions:

Check kubectl logs <pod> --previous for the actual error
Use kubectl debug to enter the container
Verify environment variables and ConfigMaps
Check database and service connectivity
Adjust liveness and readiness probe settings

2. ImagePullBackOff: Failed to Pull Image

ImagePullBackOff occurs when Kubernetes cannot pull the container image from the registry. This can happen due to authentication, network, or image name issues.

Authentication Failed

Common Cause

Missing or invalid registry credentials. Check imagePullSecrets or registry authentication configuration.

Fix: Create and reference imagePullSecrets

Network Issues

Common Cause

Cannot reach the container registry. Check network connectivity, DNS, and firewall settings.

Fix: Verify network connectivity and DNS resolution

Image Not Found

Common Cause

The specified image or tag does not exist in the registry. Check the image name and tag spelling.

Fix: Verify image name and tag existence

Registry Timeout

Common Cause

Registry response timeout due to slow connection or registry issues. Increase image pull timeout.

Fix: Increase imagePullProgressTimeout

                # Diagnostic commands for ImagePullBackOff
# Describe pod for image error
kubectl describe pod <pod-name>

# Check image pull secret
kubectl get secrets
kubectl describe secret <secret-name>

# Create image pull secret (Docker Hub)
kubectl create secret docker-registry dockerhub-creds \
  --docker-username=<username> \
  --docker-password=<password> \
  --docker-email=<email>

# Create image pull secret (AWS ECR)
kubectl create secret docker-registry ecr-creds \
  --docker-server=<aws-account>.dkr.ecr.<region>.amazonaws.com \
  --docker-username=AWS \
  --docker-password=$(aws ecr get-login-password)

# Reference secret in Pod
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  imagePullSecrets:
  - name: dockerhub-creds
  containers:
  - name: app
    image: myapp:latest

# Reference secret in Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      imagePullSecrets:
      - name: dockerhub-creds
      containers:
      - name: app
        image: myapp:latest

# Test image pull manually
docker pull myapp:latest
crane pull myapp:latest
            

ImagePullBackOff Considerations:

Always use imagePullSecrets for private registries
Verify image name and tag spelling
Check registry availability and rate limits
Use registry mirrors for Docker Hub to avoid rate limits
Test image pull manually with docker pull or crane pull

3. OOMKilled: Out of Memory

OOMKilled occurs when a container exceeds its memory limit and is terminated by the Linux OOM killer. This can cause pods to restart or fail entirely.

Memory Limit Too Low

Common Cause

Memory limit set lower than application's actual memory usage. Check memory consumption patterns.

Fix: Increase memory limit

Memory Leak

Common Cause

Application gradually consumes more memory over time. Monitor memory usage trends and fix the leak.

Fix: Identify and fix memory leaks

Heavy Workload

Common Cause

Processing large datasets or handling high traffic causing memory spikes. Scale horizontally or optimize code.

Fix: Scale replicas or optimize memory usage

Node Memory Pressure

Common Cause

Node's memory is exhausted, causing pod eviction. Check node memory usage and consider scaling.

Fix: Scale nodes or reduce pod density

                # Diagnostic commands for OOMKilled
# Check pod status
kubectl get pods
# OOMKilled status shown in pod status

# Describe pod for OOMKilled details
kubectl describe pod <pod-name>
# Look for: "State: Terminated" and "Reason: OOMKilled"

# Check container memory usage
kubectl top pods

# Check node memory usage
kubectl top nodes

# View previous logs (if available)
kubectl logs --previous <pod-name>

# Increase memory limit
apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
  - name: app
    image: myapp:latest
    resources:
      requests:
        memory: "256Mi"
      limits:
        memory: "512Mi"  # Increased from 256Mi

# Set memory limit in Deployment
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
      - name: app
        resources:
          limits:
            memory: "1Gi"
          requests:
            memory: "512Mi"
            

OOMKilled Solutions:

Increase memory limits based on actual usage
Optimize application memory usage
Scale replicas horizontally
Monitor memory usage trends
Check for memory leaks
Use VPA for automatic resource recommendations

4. Pending Pods: Not Scheduled

Pending pods are accepted by the API server but cannot be scheduled to a node. This is usually due to resource constraints, scheduling conflicts, or node issues.

Insufficient Resources

Common Cause

No node has enough CPU or memory to schedule the pod. Check node capacity and resource requests.

Fix: Reduce requests, scale nodes, or add nodes

Node Selector Mismatch

Common Cause

No node matches the pod's nodeSelector or node affinity rules. Check node labels and scheduling rules.

Fix: Add labels to nodes or update nodeSelector

Taints and Tolerations

Common Cause

Nodes have taints that the pod does not tolerate. Check node taints and pod tolerations.

Fix: Add tolerations or remove taints

PersistentVolume Issues

Common Cause

PVC cannot be bound to a PV due to storage class or availability issues. Check PVC and PV status.

Fix: Ensure PVs are available and storage class matches

                # Diagnostic commands for Pending pods
# Check pod status
kubectl get pods

# Describe pod for scheduling details
kubectl describe pod <pod-name>
# Check the Events section for scheduling messages

# Check node resources
kubectl describe nodes

# Check PVC status
kubectl get pvc
kubectl describe pvc <pvc-name>

# Check node labels
kubectl get nodes --show-labels

# Check node taints
kubectl describe nodes | grep Taints

# Add node label
kubectl label nodes <node-name> node-type=app

# Add toleration to pod
apiVersion: v1
kind: Pod
metadata:
  name: my-pod
spec:
  tolerations:
  - key: "node-type"
    operator: "Equal"
    value: "app"
    effect: "NoSchedule"

# Reduce resource requests
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
      - name: app
        resources:
          requests:
            cpu: "250m"  # Reduced
            memory: "128Mi"  # Reduced
            

Pending Pods Checklist:

Check node CPU and memory availability
Verify nodeSelector and affinity rules
Check taints and tolerations
Ensure PVCs are bound
Check for resource quotas
Verify cluster autoscaler is working

Advanced Debugging Techniques

                # Ephemeral containers for debugging
kubectl debug -it <pod-name> --image=busybox --target=<container-name>

# Debug with custom image
kubectl debug -it <pod-name> --image=ubuntu -- bash

# Check pod events
kubectl get events --field-selector involvedObject.name=<pod-name>

# Check node logs for pod scheduling issues
kubectl get nodes -o wide
kubectl describe node <node-name>

# Check kubelet logs
journalctl -u kubelet --since "5 minutes ago"

# Debug pod with a sidecar
kubectl run debug-pod --image=busybox -it --rm --restart=Never -- sh

# Check network connectivity from pod
kubectl exec -it <pod-name> -- curl <service>

# Check DNS resolution
kubectl exec -it <pod-name> -- nslookup kubernetes.default.svc.cluster.local

# Get pod YAML for analysis
kubectl get pod <pod-name> -o yaml

# Get all pods with status
kubectl get pods --all-namespaces --field-selector status.phase!=Running
            

Debugging Tips:

Use kubectl debug for ephemeral debugging containers
Check node logs for scheduling issues
Use kubectl get events for cluster-wide events
Test connectivity from inside the pod
Export pod YAML for offline analysis

Frequently Asked Questions

What is the difference between CrashLoopBackOff and Error?

CrashLoopBackOff means the container is crashing repeatedly and Kubernetes is delaying restarts. Error means the container has failed but may not be restarted. CrashLoopBackOff is more specific to restart loops.

Why does a pod stay pending?

Pods stay pending because they cannot be scheduled to a node. Common reasons: insufficient resources, node selector mismatches, taints without tolerations, or PVC binding issues. Check kubectl describe pod for the exact reason.

How do I view previous container logs?

Use kubectl logs <pod> --previous to view the logs from the previous container instance. This is essential for debugging CrashLoopBackOff.

What is OOMKilled and how do I fix it?

OOMKilled means the container exceeded its memory limit and was killed. Fix by increasing memory limits, optimizing application memory usage, or scaling horizontally.

How do I fix ImagePullBackOff?

ImagePullBackOff means Kubernetes cannot pull the image. Check image name/tag spelling, registry credentials, network connectivity, and registry availability. Use imagePullSecrets for private registries.

What should I check for pending pods?

Check node resources, nodeSelector, node affinity, taints/tolerations, PVC status, resource quotas, and cluster autoscaler. Use kubectl describe pod for the specific scheduling error.

How do I debug a container that exits immediately?

Use kubectl logs --previous to see the exit reason. Use kubectl debug to add an ephemeral container for investigation. Check application code and configuration.

What are the best practices for pod troubleshooting?

Follow a systematic approach: check status, describe, view logs, check previous logs, and check events. Use kubectl debug for ephemeral containers. Monitor pod health with probes and implement proper logging.

Previous: Flux CD Next: Troubleshooting Cluster

Troubleshooting pods is a critical skill for Kubernetes administrators. Follow a systematic approach, use the right tools, and always check logs and events for detailed error information.