Node Maintenance

Cordoning Draining OS Updates Cluster Upgrades Taints/Tolerations

A comprehensive guide to Kubernetes node maintenance covering draining, cordoning, OS updates, cluster upgrades, node taints/tolerations, and practical implementation strategies for production clusters.

Cordoning Taints/Tolerations Cluster Upgrades OS Updates

Why Node Maintenance Matters

Node maintenance is essential for keeping Kubernetes clusters secure, performant, and reliable. Regular maintenance includes:

Security: Apply OS and Kubernetes security patches
Reliability: Replace failing hardware or upgrade components
Performance: Optimize node configurations
Availability: Maintain cluster health without downtime
Compliance: Meet regulatory and security standards
Upgrades: Update Kubernetes version for new features

Maintenance Principles:

Zero Downtime: Maintain application availability during maintenance
Graceful Handling: Allow pods to terminate gracefully
Orderly Process: Follow a structured maintenance workflow
Testing: Always test maintenance procedures in staging
Documentation: Document maintenance procedures and runbooks

Cordoning: Marking Nodes Unschedulable

Cordoning marks a node as unschedulable, preventing new pods from being scheduled on it. Existing pods continue to run.

                # Cordon a node (mark unschedulable)
kubectl cordon node-1

# Verify node status
kubectl get nodes
# NAME     STATUS                     ROLES    AGE   VERSION
# node-1   Ready,SchedulingDisabled      10d   v1.28.0

# Check node details
kubectl describe node node-1

# View nodes with cordon status
kubectl get nodes -o custom-columns=NAME:.metadata.name,STATUS:.status.conditions[-1].type

# Cordon multiple nodes
for node in node-1 node-2 node-3; do
    kubectl cordon $node
done

# Why cordon first?
# 1. Prevents new pods from being scheduled during maintenance
# 2. Allows existing pods to continue running
# 3. Prepares for draining without impacting new workloads
# 4. Provides a controlled maintenance window
            

Cordoning Best Practices:

Always cordon a node before draining
Cordon during maintenance windows or off-peak hours
Monitor cluster capacity during cordoning
Uncordon after maintenance is complete
Document cordoning procedures

Draining: Evicting Pods Gracefully

Draining evicts all pods from a node, allowing them to terminate gracefully and be rescheduled elsewhere. This is essential for node maintenance without downtime.

                # Basic drain
kubectl drain node-1 --ignore-daemonsets

# Drain with grace period
kubectl drain node-1 --ignore-daemonsets --grace-period=60

# Drain with timeout
kubectl drain node-1 --ignore-daemonsets --timeout=300s

# Force drain (ignore PodDisruptionBudget)
kubectl drain node-1 --ignore-daemonsets --force

# Drain without deleting emptyDir data
kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data

# Drain specific pods (with pod-selector)
kubectl drain node-1 --ignore-daemonsets --pod-selector="app=nginx"

# Check drain status
kubectl get pods --all-namespaces --field-selector spec.nodeName=node-1

# Monitor drain progress
kubectl get events --field-selector involvedObject.kind=Node --watch

# Drain all nodes in a node group
for node in node-1 node-2 node-3; do
    kubectl cordon $node
    kubectl drain $node --ignore-daemonsets --delete-emptydir-data
done
            

1. Cordon the Node

kubectl cordon node-1 - Prevent new pods from being scheduled

2. Drain the Node

kubectl drain node-1 --ignore-daemonsets - Evict all pods gracefully

3. Perform Maintenance

Update OS, upgrade Kubernetes, replace hardware, etc.

4. Uncordon the Node

kubectl uncordon node-1 - Allow scheduling again

5. Verify Pods

Check that pods are rescheduled and running correctly

Draining Best Practices:

Always use --ignore-daemonsets (DaemonSets stay on the node)
Use --delete-emptydir-data for pods with emptyDir volumes
Set appropriate grace periods for termination
Monitor cluster capacity to ensure pods can be rescheduled
Test drain procedures in staging before production
Use --timeout to avoid indefinite hanging

OS Updates and Security Patches

Regular OS updates are critical for security and performance. They should be applied during maintenance windows using a controlled process.

                # Node maintenance workflow for OS updates

# 1. Before maintenance: Check cluster health
kubectl get nodes
kubectl get pods --all-namespaces

# 2. Cordon and drain the node
kubectl cordon node-1
kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data

# 3. Perform OS updates (Ubuntu example)
# SSH to the node
ssh node-1
sudo apt-get update
sudo apt-get upgrade -y
sudo reboot

# 4. Wait for node to come back
kubectl get node node-1

# 5. Verify node status and uncordon
kubectl uncordon node-1

# 6. Verify pods are running
kubectl get pods --all-namespaces --field-selector spec.nodeName=node-1

# Automated OS update script
#!/bin/bash
NODES=("node-1" "node-2" "node-3")

for node in "${NODES[@]}"; do
    echo "Maintaining $node"
    kubectl cordon $node
    kubectl drain $node --ignore-daemonsets --delete-emptydir-data --timeout=300s
    # Perform updates via SSH
    ssh $node "sudo apt-get update && sudo apt-get upgrade -y && sudo reboot"
    sleep 60
    kubectl uncordon $node
    echo "Node $node maintenance complete"
done

# Security updates (critical)
sudo apt-get update && sudo apt-get install -y unattended-upgrades
sudo dpkg-reconfigure --priority=low unattended-upgrades

# Kernel updates require reboot
# Check if reboot is needed
if [ -f /var/run/reboot-required ]; then
    sudo reboot
fi
            

OS Update Best Practices:

Apply security patches promptly (critical CVEs)
Schedule regular maintenance windows for updates
Test updates in staging before production
Use automation for consistent updates
Monitor node health after updates
Have rollback procedures for failed updates

Kubernetes Cluster Upgrades

Upgrading Kubernetes versions is essential for new features, security patches, and performance improvements. Always follow a phased approach.

                # Cluster upgrade workflow (kubeadm)

# 1. Check current version
kubectl version --short

# 2. Check available versions
kubeadm upgrade plan

# 3. Upgrade control plane
kubeadm upgrade apply v1.28.0

# 4. Upgrade kubelet and kubectl on control plane nodes
sudo apt-get update && sudo apt-get install -y kubelet=1.28.0-00 kubectl=1.28.0-00
sudo systemctl daemon-reload
sudo systemctl restart kubelet

# 5. Upgrade worker nodes (one at a time)
# Cordon and drain the node
kubectl cordon node-1
kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data

# Upgrade kubelet on worker node
ssh node-1 "sudo apt-get update && sudo apt-get install -y kubelet=1.28.0-00"
ssh node-1 "sudo systemctl daemon-reload && sudo systemctl restart kubelet"

# Uncordon the node
kubectl uncordon node-1

# Repeat for all worker nodes

# Managed cluster upgrades (EKS)
eksctl upgrade cluster --name my-cluster --version 1.28

# Managed cluster upgrades (GKE)
gcloud container clusters upgrade my-cluster --zone us-central1-a --master

# Managed cluster upgrades (AKS)
az aks upgrade --resource-group my-rg --name my-cluster --kubernetes-version 1.28

# Check node versions after upgrade
kubectl get nodes -o wide

# Verify cluster health
kubectl get nodes
kubectl get pods --all-namespaces
            

Upgrade Best Practices:

Always test upgrades in a staging environment
Have a rollback plan
Upgrade one node at a time
Verify API compatibility before upgrading
Schedule upgrades during maintenance windows
Monitor cluster health during and after upgrades

Taints and Tolerations: Node Scheduling Control

Taints and tolerations allow you to control which pods can be scheduled on which nodes. Taints are applied to nodes; tolerations are applied to pods.

                # Add a taint to a node
kubectl taint nodes node-1 key=value:NoSchedule
kubectl taint nodes node-1 dedicated=db:NoSchedule

# Remove a taint
kubectl taint nodes node-1 key=value:NoSchedule-

# View taints on nodes
kubectl describe nodes | grep Taints

# Types of taints:
# - NoSchedule: Pods without toleration won't be scheduled
# - PreferNoSchedule: Cluster tries to avoid scheduling
# - NoExecute: Pods without toleration will be evicted

# Pod toleration for the taint
apiVersion: v1
kind: Pod
metadata:
  name: db-pod
spec:
  containers:
  - name: db
    image: postgres:13
  tolerations:
  - key: dedicated
    operator: Equal
    value: db
    effect: NoSchedule

# Node with NoExecute taint (for maintenance)
kubectl taint nodes node-1 maintenance=upgrade:NoExecute

# Pod toleration with NoExecute (allow pod to stay during maintenance)
apiVersion: v1
kind: Pod
metadata:
  name: critical-pod
spec:
  tolerations:
  - key: maintenance
    operator: Equal
    value: upgrade
    effect: NoExecute
    tolerationSeconds: 3600  # Pod can stay for 1 hour

# Taint with PreferNoSchedule
kubectl taint nodes node-1 node-type=spot:PreferNoSchedule

# Use taints for node groups (dedicated nodes)
kubectl taint nodes node-gpu gpu=true:NoSchedule
kubectl taint nodes node-db db=true:NoSchedule

# Check pods with tolerations
kubectl get pods -o json | jq '.items[] | {name: .metadata.name, tolerations: .spec.tolerations}'
            

Taint/Toleration Best Practices:

Use taints for dedicated node pools (GPU, DB, high-memory)
Use NoExecute taints for maintenance (with tolerationSeconds)
Apply taints before cordoning for controlled maintenance
Document all taints and their purposes
Combine with node selectors for precise scheduling

Maintenance Automation

                # Automated node rotation script
#!/bin/bash
NODE_GROUPS=("node-1" "node-2" "node-3")
MAINTENANCE_WINDOW="2h"

for node in "${NODE_GROUPS[@]}"; do
    echo "Starting maintenance for $node"
    
    # Cordon the node
    kubectl cordon $node
    
    # Drain the node
    kubectl drain $node --ignore-daemonsets --delete-emptydir-data --timeout=300s
    
    # Perform maintenance
    # ... your maintenance tasks ...
    
    # Wait for maintenance to complete
    sleep 60
    
    # Uncordon the node
    kubectl uncordon $node
    
    echo "Maintenance complete for $node"
    echo "Waiting for pods to stabilize..."
    sleep 30
done

# Maintenance with node conditions
apiVersion: v1
kind: Pod
metadata:
  name: maintenance-pod
  annotations:
    cluster-autoscaler.kubernetes.io/safe-to-evict: "true"
spec:
  containers:
  - name: maintenance
    image: alpine
    command: ["sleep", "infinity"]

# Use kubectl plugin for node maintenance
# Install krew
kubectl krew install node-maintenance

# Use node-maintenance plugin
kubectl node-maintenance start node-1 --reason="OS Update"
kubectl node-maintenance status node-1
kubectl node-maintenance stop node-1
            

Automation Best Practices:

Automate node rotation for regular maintenance
Implement monitoring and alerting during maintenance
Use maintenance windows to minimize impact
Test automation in staging before production
Document automation scripts and procedures
Have manual fallback procedures

Frequently Asked Questions

What is the difference between cordoning and draining?

Cordoning marks a node as unschedulable (no new pods). Draining evicts all existing pods from the node. Use cordon first, then drain for maintenance.

What happens to DaemonSets during draining?

DaemonSets are not evicted by default. Use --ignore-daemonsets to allow DaemonSet pods to remain. This is usually the desired behavior.

Can I drain a node with local storage?

Use --delete-emptydir-data to delete emptyDir data. For persistent volumes, pods should be rescheduled. Local storage with hostPath may require additional handling.

What is the recommended maintenance window?

Maintenance windows should be scheduled during low-traffic periods (off-peak hours). Consider business hours, time zones, and traffic patterns. For critical systems, use canary deployments for upgrades.

How do I upgrade a single node?

Cordon the node, drain it, perform the upgrade (OS or Kubernetes), reboot if needed, then uncordon. Check that pods are rescheduled correctly.

What are taints and tolerations used for?

Taints are applied to nodes to repel pods. Tolerations are applied to pods to allow scheduling on tainted nodes. They enable dedicated nodes, maintenance, and workload isolation.

How do I handle maintenance with PodDisruptionBudgets?

PDBs prevent draining if they would violate the disruption budget. Check PDB status before draining. Use --force to bypass PDBs if absolutely necessary.

What should I do if drain hangs?

Check for PDB violations, pods with long termination grace periods, or stuck pods. Use --timeout to set a limit. Consider using --force as a last resort.

Previous: Resource Management Next: Cost Optimization

Regular node maintenance is essential for cluster security, performance, and reliability. Follow these best practices to maintain your nodes without disrupting applications.