Common Kubernetes Issues

Resource Exhaustion DNS Issues Certificate Expiration Scheduler Problems

A comprehensive guide to common Kubernetes issues covering resource exhaustion, DNS issues, certificate expiration, scheduler problems, and practical solutions for production clusters.

Resource Exhaustion DNS Issues Certificate Expiration Scheduler Problems

Understanding Common Kubernetes Issues

Kubernetes clusters encounter a range of issues in production. Understanding the most common problems and their solutions is essential for maintaining cluster health.

This guide covers the most frequently encountered issues:

Resource Exhaustion: CPU, memory, disk, and pod limits
DNS Issues: CoreDNS failures, resolution problems
Certificate Expiration: TLS certificate management
Scheduler Problems: Pod scheduling failures
Node Issues: Node not ready, resource pressure
Networking Issues: CNI, network policies, service connectivity

Preventive Approach:

Monitor cluster health proactively
Set up alerts for resource thresholds
Regularly renew certificates
Test DNS resolution regularly
Implement proper resource requests and limits

1. Resource Exhaustion

Resource exhaustion occurs when cluster resources (CPU, memory, disk, pods) are fully utilized, leading to performance degradation, pod evictions, and cluster instability.

Memory Exhaustion

Common Cause

Pods requesting more memory than available, memory leaks, or insufficient node memory. Leads to OOMKilled pods and node pressure.

Fix: Increase memory limits, optimize apps, add nodes

CPU Exhaustion

Common Cause

High CPU usage across pods or nodes. Causes performance degradation, throttling, and scheduler delays.

Fix: Increase CPU limits, optimize apps, add nodes

Disk Exhaustion

Common Cause

Node disk space full due to container images, logs, or storage. Causes pod evictions and node issues.

Fix: Prune images, clean logs, increase disk size

Pod Exhaustion

Common Cause

Reached the pod limit per node (default 110). Prevents new pods from being scheduled.

Fix: Add nodes, reduce pod density, increase node limit

                # Diagnostic commands for Resource Exhaustion
# Check node resource usage
kubectl top nodes

# Check pod resource usage
kubectl top pods --all-namespaces

# Check node conditions
kubectl describe node <node-name> | grep -A 5 "Conditions"

# Check disk usage on node
df -h
du -sh /var/lib/containerd

# Check pod limits per node
kubectl describe node <node-name> | grep -A 10 "Allocated resources"

# Check for pod evictions
kubectl get events --field-selector reason=Evicted

# Check node capacity
kubectl describe node <node-name> | grep -A 10 "Capacity"

# Monitor memory usage over time
kubectl top pods --all-namespaces --sort-by=memory

# Monitor CPU usage over time
kubectl top pods --all-namespaces --sort-by=cpu

# Check resource quotas
kubectl get resourcequotas --all-namespaces

# Clean up unused images
crictl images | grep -v "latest" | awk '{print $3}' | xargs -r crictl rmi
docker system prune -f
            

Resource Exhaustion Solutions:

Set appropriate resource requests and limits
Implement Horizontal Pod Autoscaling (HPA)
Use Cluster Autoscaler for node scaling
Regularly prune unused images and resources
Monitor resource usage trends
Implement resource quotas per namespace

2. DNS Issues

DNS is critical for service discovery in Kubernetes. DNS issues can cause service failures, application errors, and cluster instability.

CoreDNS Failures

Common Cause

CoreDNS pods crashing, not starting, or performance degradation. Check pod status and logs.

Fix: Restart CoreDNS, increase resources, check config

DNS Resolution Issues

Common Cause

Pods cannot resolve service names. Check DNS configuration, network policies, and CoreDNS logs.

Fix: Check CoreDNS config, verify endpoints, test resolution

DNS Timeouts

Common Cause

Slow DNS responses due to overloaded CoreDNS or network issues. Check CoreDNS performance and resources.

Fix: Scale CoreDNS, increase resources, optimize config

DNS Configuration Issues

Common Cause

Incorrect DNS configuration in pods, CoreDNS config, or network policies. Check resolv.conf and CoreDNS ConfigMap.

Fix: Verify CoreDNS ConfigMap, check pod DNS policy

                # Diagnostic commands for DNS Issues
# Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Check CoreDNS logs
kubectl logs -n kube-system -l k8s-app=kube-dns

# Check CoreDNS config
kubectl describe configmap coredns -n kube-system

# Test DNS resolution from pod
kubectl run test-pod --image=busybox -it --rm -- nslookup kubernetes.default.svc.cluster.local

# Check pod's resolv.conf
kubectl exec -it <pod-name> -- cat /etc/resolv.conf

# Check DNS service endpoints
kubectl get endpoints -n kube-system kube-dns

# Check CoreDNS metrics
kubectl port-forward -n kube-system svc/kube-dns 9153:9153
curl http://localhost:9153/metrics | grep coredns

# Restart CoreDNS
kubectl delete pods -n kube-system -l k8s-app=kube-dns

# Scale CoreDNS
kubectl scale deployment coredns -n kube-system --replicas=3
            

DNS Critical Considerations:

Run at least 2 CoreDNS replicas for HA
Set appropriate resource requests/limits for CoreDNS
Monitor CoreDNS metrics and logs
Configure DNS caching for performance
Test DNS resolution after cluster upgrades

3. Certificate Expiration

Kubernetes uses TLS certificates for secure communication between components. Expired certificates can cause cluster failures and API server unavailability.

API Server Certificate Expired

Common Cause

API server certificate expired, preventing client connections. Check certificate expiration dates and renew.

Fix: Renew certificates, restart API server

etcd Certificate Expired

Common Cause

etcd certificates expired, preventing etcd cluster communication. Check etcd certificate expiration.

Fix: Renew etcd certificates, restart etcd

Kubelet Certificate Expired

Common Cause

Node kubelet certificate expired, preventing API server communication. Check kubelet certificate status.

Fix: Renew kubelet certificates, restart kubelet

Service Account Certificate Expired

Common Cause

Service account token certificates expired, preventing pod authentication. Check service account token status.

Fix: Regenerate service account tokens

                # Diagnostic commands for Certificate Issues
# Check certificate expiration (kubeadm)
kubeadm certs check-expiration

# Check API server certificate
openssl x509 -in /etc/kubernetes/pki/apiserver.crt -text -noout

# Check etcd certificate
openssl x509 -in /etc/kubernetes/pki/etcd/server.crt -text -noout

# Check kubelet certificate
openssl x509 -in /var/lib/kubelet/pki/kubelet-client-current.pem -text -noout

# Renew certificates (kubeadm)
kubeadm certs renew all

# Renew specific certificate
kubeadm certs renew apiserver

# Restart control plane components after renewal
kubectl delete pod -n kube-system kube-apiserver-<node>
kubectl delete pod -n kube-system kube-scheduler-<node>
kubectl delete pod -n kube-system kube-controller-manager-<node>

# Check service account token
kubectl describe secret <secret-name>

# Regenerate service account token
kubectl delete secret <secret-name>

# Monitor certificate expiration
kubeadm certs check-expiration --config /etc/kubernetes/kubeadm-config.yaml
            

Certificate Management Best Practices:

Regularly check certificate expiration
Set up monitoring for certificate expiration
Automate certificate renewal (kubeadm auto-renewal)
Test certificate renewal in staging
Keep backup of certificates
Document certificate management procedures

4. Scheduler Problems

The scheduler is responsible for assigning pods to nodes. Scheduler problems can cause pending pods and cluster imbalance.

Insufficient Resources

Common Cause

No node with sufficient CPU or memory to schedule the pod. Check node capacity and resource requests.

Fix: Add nodes, reduce requests, optimize scheduling

Node Affinity/Selector Issues

Common Cause

No node matching nodeSelector or node affinity rules. Check node labels and scheduling rules.

Fix: Update node labels or affinity rules

Taint/Toleration Issues

Common Cause

Nodes with taints that the pod does not tolerate. Check node taints and pod tolerations.

Fix: Add tolerations or remove taints

Pod Limit Reached

Common Cause

Node has reached the maximum number of pods (default 110). Check node pod capacity.

Fix: Add nodes, reduce pod density, increase limit

                # Diagnostic commands for Scheduler Issues
# Check pending pods
kubectl get pods --field-selector status.phase=Pending

# Describe pending pod for reason
kubectl describe pod <pod-name>

# Check node resources
kubectl describe nodes

# Check node labels
kubectl get nodes --show-labels

# Check node taints
kubectl describe nodes | grep Taints

# Check scheduler logs
kubectl logs -n kube-system kube-scheduler-<node>

# Check scheduler events
kubectl get events --field-selector involvedObject.kind=Pod

# Check pod scheduling conditions
kubectl get pods -o wide | grep Pending

# Check node pod count
kubectl get nodes -o json | jq '.items[].status.allocatable.pods'

# Check cluster autoscaler
kubectl get pods -n kube-system -l app=cluster-autoscaler

# Add node label
kubectl label nodes <node-name> node-type=app
            

Scheduler Debugging Tips:

Check pending pod description for scheduling failure reasons
Monitor node resource usage and availability
Review scheduler logs for error messages
Check cluster autoscaler if nodes are insufficient
Verify nodeSelector and affinity rules

Quick Reference: Common Issues and Solutions

                # Resource Exhaustion
kubectl top nodes
kubectl top pods --all-namespaces
kubectl describe node <node>
# Solutions: Add nodes, reduce requests, prune images

# DNS Issues
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system coredns-<pod>
kubectl run test --image=busybox --rm -it -- nslookup kubernetes.default.svc
# Solutions: Restart CoreDNS, scale, check config

# Certificate Expiration
kubeadm certs check-expiration
kubeadm certs renew all
# Solutions: Renew certificates, restart components

# Scheduler Problems
kubectl get pods --field-selector status.phase=Pending
kubectl describe pod <pod>
kubectl describe nodes
# Solutions: Add resources, update affinities, add tolerations

# Node Issues
kubectl get nodes
kubectl describe node <node>
journalctl -u kubelet -f
# Solutions: Restart kubelet, free resources, replace node

# Networking Issues
kubectl get pods -n kube-system -l k8s-app=calico-node
kubectl get endpoints --all-namespaces
kubectl get networkpolicies --all-namespaces
# Solutions: Check CNI, verify policies, restart components
            

Frequently Asked Questions

How do I prevent resource exhaustion in my cluster?

Set appropriate resource requests and limits, implement HPA and Cluster Autoscaler, monitor resource usage regularly, and use resource quotas per namespace. Regular cleanup of unused images and resources is also essential.

What causes DNS resolution failures in Kubernetes?

Common causes: CoreDNS pod failures, network policies blocking DNS traffic, incorrect CoreDNS configuration, pod DNS policy issues, and network connectivity problems between pods and CoreDNS.

How often should I check certificate expiration?

Check certificate expiration monthly, or set up monitoring alerts. Kubernetes certificates typically expire after 1 year. kubeadm auto-renews certificates by default.

Why are my pods stuck in Pending state?

Pending pods indicate scheduling issues: insufficient node resources, node selector mismatches, taints without tolerations, PVC binding issues, or resource quotas. Check kubectl describe pod for the specific reason.

How do I troubleshoot CoreDNS issues?

Check CoreDNS pod status and logs, test DNS resolution from a pod, verify CoreDNS ConfigMap, check service endpoints, and scale CoreDNS if needed. Monitor CoreDNS metrics for performance issues.

What are the signs of certificate expiration?

Signs include: API server connection failures, kubectl command failures, service account authentication issues, and error messages about invalid or expired certificates in logs.

How do I handle out-of-memory (OOM) issues?

Increase memory limits for containers, optimize application memory usage, implement horizontal scaling, and monitor memory usage trends. Use VPA for automatic recommendations.

What should I monitor proactively in production clusters?

Monitor: node resource usage (CPU, memory, disk), pod status and restarts, CoreDNS performance, certificate expiration dates, cluster events, API server latency, and node conditions.

Previous: Troubleshooting Cluster Next: Kubernetes Interview Questions

Understanding common Kubernetes issues and their solutions is essential for maintaining production clusters. Implement proactive monitoring, regular maintenance, and automated responses to minimize downtime.