Troubleshooting Cluster

Control Plane etcd Problems Networking Node Failures

A comprehensive guide to troubleshooting Kubernetes clusters covering control plane issues, etcd problems, networking, node failures, and common cluster-level problems with practical solutions.

Control Plane Issues etcd Problems Networking Issues Node Failures

Understanding Cluster-Level Issues

Cluster-level issues affect the entire Kubernetes cluster and can impact all workloads. These issues are often more complex to diagnose than pod-level issues and require a systematic approach.

Common cluster-level problems include:

Control Plane Issues: API server, scheduler, or controller manager failures
etcd Problems: Database corruption, leader election issues, or performance degradation
Networking Issues: CNI problems, network policies, or service connectivity
Node Failures: Node not ready, resource exhaustion, or hardware issues
Certificate Issues: Expired or invalid certificates
Resource Exhaustion: Node or cluster-level resource pressure

Systematic Debugging Approach:

Check cluster status: kubectl get nodes and kubectl get componentstatuses
Check control plane logs: kubectl logs -n kube-system
Check etcd health: etcdctl endpoint health
Check node status: kubectl describe node <node>
Check cluster events: kubectl get events --all-namespaces

1. Control Plane Issues

The control plane consists of the API server, scheduler, and controller manager. Issues here affect the entire cluster's ability to manage workloads.

API Server Unavailable

Common Cause

API server pod crashes or cannot start. Check API server logs and pod status in kube-system namespace.

Fix: Check logs, verify certificates, increase resources

Scheduler Issues

Common Cause

Pods remain pending because the scheduler cannot assign them to nodes. Check scheduler logs and node status.

Fix: Check scheduler logs, verify node status, check resources

Controller Manager Issues

Common Cause

Controllers fail to reconcile desired state. Deployment updates may not propagate, or autoscaling may not work.

Fix: Check controller manager logs, verify leader election

Certificate Issues

Common Cause

Expired or invalid certificates prevent API server communication. Check certificate expiration dates.

Fix: Renew certificates, regenerate if expired

                # Diagnostic commands for Control Plane
# Check component status
kubectl get componentstatuses

# Check control plane pods
kubectl get pods -n kube-system

# Check API server logs
kubectl logs -n kube-system kube-apiserver-<node>

# Check scheduler logs
kubectl logs -n kube-system kube-scheduler-<node>

# Check controller manager logs
kubectl logs -n kube-system kube-controller-manager-<node>

# Check certificate expiration
kubeadm certs check-expiration

# Renew certificates (kubeadm)
kubeadm certs renew all

# Restart control plane components
kubectl delete pod -n kube-system kube-apiserver-<node>
kubectl delete pod -n kube-system kube-scheduler-<node>
kubectl delete pod -n kube-system kube-controller-manager-<node>

# Check leader election
kubectl get leases -n kube-system
            

Control Plane Solutions:

Check pod logs for errors
Verify certificate expiration with kubeadm certs check-expiration
Renew certificates if expired
Restart control plane components
Increase resources for control plane pods if needed
Check node health for control plane nodes

2. etcd Problems

etcd is the cluster's state store. Issues with etcd can cause cluster instability, data corruption, or complete cluster failure.

Leader Election Issues

Common Cause

etcd cluster cannot elect a leader due to network issues or node failures. Check etcd member status and network connectivity.

Fix: Check network connectivity, restart etcd

Database Corruption

Common Cause

etcd database corruption can occur due to disk issues or sudden power loss. Restore from backup if possible.

Fix: Restore from backup, run etcdctl repair

Performance Issues

Common Cause

Slow etcd responses due to high load, disk I/O, or large data size. Monitor latency and disk performance.

Fix: Increase resources, compact database, optimize queries

Disk Space Exhaustion

Common Cause

etcd data directory fills up, preventing writes. Monitor disk space and configure compaction.

Fix: Compact and defragment database, free disk space

                # Diagnostic commands for etcd
# Check etcd endpoints
ETCDCTL_API=3 etcdctl endpoint health --cluster

# Check etcd member list
ETCDCTL_API=3 etcdctl member list

# Check etcd leader
ETCDCTL_API=3 etcdctl endpoint status --cluster

# Check etcd metrics
curl http://localhost:2379/metrics

# Check etcd data size
ETCDCTL_API=3 etcdctl endpoint status --write-out=table

# Check etcd disk usage
df -h /var/lib/etcd

# Defragment etcd
ETCDCTL_API=3 etcdctl defrag --endpoints=http://localhost:2379

# Compact etcd
ETCDCTL_API=3 etcdctl compaction <revision>

# Backup etcd
ETCDCTL_API=3 etcdctl snapshot save /backups/etcd-$(date +%Y%m%d).db

# Restore etcd
ETCDCTL_API=3 etcdctl snapshot restore /backups/etcd-20250115.db --data-dir /var/lib/etcd-restore

# Check etcd logs
journalctl -u etcd --since "1 hour ago"
            

etcd Critical Considerations:

Always backup etcd regularly
Monitor etcd disk space and performance
Use SSDs for etcd data directory
Run etcd in a cluster of 3 or 5 nodes for HA
Configure compaction and defragmentation
Test restore procedures regularly

3. Cluster Networking Issues

Networking issues can prevent pods from communicating with each other, with services, or with external resources.

CNI Issues

Common Cause

CNI plugin pods failing, network policies misconfigured, or pod CIDR conflicts. Check CNI pod logs and network configuration.

Fix: Restart CNI pods, verify CNI configuration

Network Policy Issues

Common Cause

Network policies blocking traffic between pods or services. Check NetworkPolicy resources and CNI logs.

Fix: Verify network policies, check CNI enforcement

DNS Issues

Common Cause

CoreDNS not resolving service names. Check CoreDNS pods, configuration, and service endpoints.

Fix: Restart CoreDNS, check config, verify endpoints

Ingress Issues

Common Cause

Ingress controller not routing traffic correctly. Check ingress controller logs and Ingress resource configuration.

Fix: Check ingress controller logs, verify Ingress resources

                # Diagnostic commands for Networking
# Check CNI pods
kubectl get pods -n kube-system | grep -E "calico|cilium|flannel|weave|antrea"

# Check CNI logs
kubectl logs -n kube-system calico-node-<id>

# Check NetworkPolicies
kubectl get networkpolicies --all-namespaces

# Check CoreDNS pods
kubectl get pods -n kube-system -l k8s-app=kube-dns

# Check CoreDNS logs
kubectl logs -n kube-system coredns-<pod>

# Test DNS resolution from pod
kubectl run test --image=busybox -it --rm -- nslookup kubernetes.default.svc.cluster.local

# Check kube-proxy
kubectl get pods -n kube-system -l k8s-app=kube-proxy

# Check kube-proxy logs
kubectl logs -n kube-system kube-proxy-<node>

# Check ingress controller
kubectl get pods -n ingress-nginx

# Check ingress resources
kubectl get ingress --all-namespaces

# Check service endpoints
kubectl get endpoints --all-namespaces
            

Networking Best Practices:

Use a reliable CNI plugin (Calico, Cilium)
Monitor CNI pod health
Test network policies in staging
Configure CoreDNS with appropriate resources
Monitor ingress controller performance
Use network policy verification tools

4. Node Failures

Node failures can be caused by hardware issues, resource exhaustion, kernel panics, or configuration problems.

Node Not Ready

Common Cause

Node is not reporting ready status due to network, resource, or kubelet issues. Check node status and kubelet logs.

Fix: Restart kubelet, check network, verify resources

Disk Pressure

Common Cause

Node disk space is low, causing pod eviction. Check disk usage and clean up unnecessary files.

Fix: Clean disk space, increase storage, configure eviction

Memory Pressure

Common Cause

Node memory is exhausted, causing pod eviction. Check memory usage and scale nodes.

Fix: Add nodes, reduce pod density, increase memory

CPU Pressure

Common Cause

Node CPU is overloaded, causing performance degradation. Check CPU usage and scale nodes.

Fix: Add nodes, reduce workload, optimize applications

                # Diagnostic commands for Node Issues
# Check node status
kubectl get nodes

# Describe node for detailed status
kubectl describe node <node-name>

# Check node conditions
kubectl get nodes -o json | jq '.items[].status.conditions'

# Check node resources
kubectl top nodes

# Check kubelet logs
journalctl -u kubelet -f

# Check node disk usage
df -h
sudo du -sh /var/lib/containerd

# Check node memory
free -h
cat /proc/meminfo

# Check node CPU
top
mpstat -P ALL

# Check pods on node
kubectl get pods --field-selector spec.nodeName=<node-name>

# Drain node for maintenance
kubectl drain <node-name> --ignore-daemonsets

# Uncordon node
kubectl uncordon <node-name>

# Check eviction thresholds
kubectl describe node <node-name> | grep -A 5 "Eviction"
            

Node Recovery Steps:

Check node status with kubectl get nodes
Describe node for detailed conditions
Check kubelet logs on the node
Free disk space if needed
Restart kubelet if necessary
Consider draining and replacing the node
Monitor resource usage for trends

Quick Reference: Common Cluster Issues

                # API Server Issues
kubectl get pods -n kube-system -l component=kube-apiserver
kubectl logs -n kube-system kube-apiserver-<pod>

# Scheduler Issues
kubectl get pods -n kube-system -l component=kube-scheduler
kubectl logs -n kube-system kube-scheduler-<pod>

# Controller Manager Issues
kubectl get pods -n kube-system -l component=kube-controller-manager
kubectl logs -n kube-system kube-controller-manager-<pod>

# etcd Issues
kubectl get pods -n kube-system -l component=etcd
ETCDCTL_API=3 etcdctl endpoint health --cluster

# CNI Issues
kubectl get pods -n kube-system -l k8s-app=calico-node
kubectl logs -n kube-system calico-node-<pod>

# CoreDNS Issues
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl logs -n kube-system coredns-<pod>

# Node Issues
kubectl get nodes
kubectl describe node <node>
journalctl -u kubelet --since "1 hour ago"

# Certificate Issues
kubeadm certs check-expiration
kubeadm certs renew all

# Cluster Events
kubectl get events --all-namespaces --sort-by='.lastTimestamp'
            

Frequently Asked Questions

What should I check first when a cluster is unhealthy?

Start with kubectl get nodes to check node status, then kubectl get pods -n kube-system to check control plane pods. Check component statuses and cluster events.

How do I recover a failed etcd cluster?

Stop etcd, restore from backup using etcdctl snapshot restore, start etcd, and verify cluster health. For a multi-node cluster, restore on each node sequentially.

What causes a node to be NotReady?

Common causes: network issues, kubelet not running, disk pressure, memory pressure, or kernel panics. Check kubelet logs and node conditions.

How do I fix certificate expiration?

Use kubeadm certs renew all to renew certificates, then restart control plane components. For managed clusters, use cloud provider tools.

What are the signs of etcd performance issues?

Slow API responses, increased request latency, frequent leader elections, and high disk I/O. Monitor etcd metrics and logs.

How do I troubleshoot DNS issues in the cluster?

Check CoreDNS pods, verify Service endpoints, test DNS resolution from a pod, and check CoreDNS configuration and logs.

What should I do when a node is out of disk space?

Clean up unused images with docker system prune or crictl rmi --prune. Check pod logs for large files. Consider increasing disk size.

How do I check control plane component health?

Use kubectl get componentstatuses to check API server, scheduler, and controller manager health. Also check pod status and logs in kube-system namespace.

Previous: Troubleshooting Pods Next: Common Kubernetes Issues

Cluster-level troubleshooting requires a systematic approach and understanding of all Kubernetes components. Start with status checks, examine logs, and verify each component's health to quickly identify and resolve issues.