Stateful Applications on Kubernetes

StatefulSets Headless Services Ordered Deployment Stable Storage

A comprehensive guide to running stateful applications on Kubernetes covering StatefulSets, headless services, ordered deployment, stable storage, and practical implementation strategies for databases and stateful workloads.

StatefulSets Ordered Deployment Stable Storage Headless Services

What are Stateful Applications?

Stateful applications are applications that maintain state, store data persistently, and require stable identities. Unlike stateless applications, stateful applications cannot be easily replaced or scaled without considering data persistence and ordering.

Examples of stateful applications include databases (PostgreSQL, MySQL, MongoDB), message queues (Kafka, RabbitMQ), distributed systems (Zookeeper, etcd), and any application that stores user data or maintains session state.

Kubernetes provides StatefulSets specifically for managing stateful applications, offering features like stable network identities, ordered deployment, and persistent storage.

Key Stateful Application Requirements:

Stable Identity: Each instance must have a unique, stable hostname
Ordered Deployment: Instances must start and stop in a specific order
Persistent Storage: Data must survive pod restarts and rescheduling
Stable Storage: Each instance should have its own persistent storage
Graceful Termination: Clean shutdown to maintain data integrity

StatefulSets: Managing Stateful Workloads

A StatefulSet is a Kubernetes workload API object used to manage stateful applications. It provides guarantees about the ordering and uniqueness of pods, stable network identities, and persistent storage.

Stable Network Identity

Predictable pod names

Each pod gets a stable, unique name: <statefulset-name>-<ordinal> (e.g., mysql-0, mysql-1). These names persist across rescheduling.

Service discovery, clustering

Ordered Deployment

Controlled startup and shutdown

Pods are created and deleted in order (0 to N-1 for creation, reverse for deletion). Scaling up and down happens sequentially.

Leader election, master-slave

Persistent Storage

Per-pod storage

Each pod gets its own PersistentVolume using volumeClaimTemplates. When a pod is rescheduled, it retains its data and storage.

Databases, stateful services

Headless Service

Direct pod communication

A headless service (clusterIP: None) enables direct DNS resolution of individual pod IPs, essential for stateful application clustering.

Peer-to-peer communication

StatefulSet Example: PostgreSQL

                # Headless Service for StatefulSet
apiVersion: v1
kind: Service
metadata:
  name: postgres
  labels:
    app: postgres
spec:
  clusterIP: None
  selector:
    app: postgres
  ports:
  - port: 5432
    name: postgres

---
# PostgreSQL StatefulSet
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
  labels:
    app: postgres
spec:
  serviceName: postgres
  replicas: 3
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
      - name: postgres
        image: postgres:15
        ports:
        - containerPort: 5432
          name: postgres
        env:
        - name: POSTGRES_PASSWORD
          valueFrom:
            secretKeyRef:
              name: postgres-secret
              key: password
        - name: POSTGRES_USER
          value: postgres
        - name: POSTGRES_DB
          value: mydb
        volumeMounts:
        - name: data
          mountPath: /var/lib/postgresql/data
        resources:
          requests:
            memory: "256Mi"
            cpu: "500m"
          limits:
            memory: "1Gi"
            cpu: "1"
        livenessProbe:
          exec:
            command:
            - pg_isready
            - -U
            - postgres
          initialDelaySeconds: 30
          timeoutSeconds: 5
        readinessProbe:
          exec:
            command:
            - pg_isready
            - -U
            - postgres
          initialDelaySeconds: 5
          timeoutSeconds: 2
      terminationGracePeriodSeconds: 60
  volumeClaimTemplates:
  - metadata:
      name: data
    spec:
      accessModes:
      - ReadWriteOnce
      resources:
        requests:
          storage: 10Gi
      storageClassName: standard

# Secret for database credentials
apiVersion: v1
kind: Secret
metadata:
  name: postgres-secret
type: Opaque
data:
  password: cG9zdGdyZXNwYXNz  # base64 encoded
            

StatefulSet Best Practices:

Always use a headless service for StatefulSets
Define terminationGracePeriodSeconds for graceful shutdown
Use readiness and liveness probes for health checks
Set resource requests and limits for predictable performance
Use volumeClaimTemplates for automatic PVC creation
Consider PodDisruptionBudget for availability

Ordered Deployment and Scaling

StatefulSets enforce strict ordering for pod creation, deletion, and updates. This is critical for applications that require leader election, master-slave replication, or ordered initialization.

                # Ordered creation: 0, 1, 2 (sequential)
# Pods are created in ascending order
kubectl create -f statefulset.yaml
# Watches: postgres-0 created, Running → postgres-1 created, Running → postgres-2 created, Running

# Ordered deletion: 2, 1, 0 (reverse)
kubectl delete statefulset postgres
# postgres-2 deleted → postgres-1 deleted → postgres-0 deleted

# Scaling up: 0, 1, 2, 3
kubectl scale statefulset postgres --replicas=4
# postgres-3 created

# Scaling down: 3, 2, 1, 0
kubectl scale statefulset postgres --replicas=2
# postgres-3 deleted → postgres-2 deleted

# Rolling updates (ordered)
# Updated pods are terminated in reverse order (2, 1, 0)
# New pods are created in order (0, 1, 2)

# Partitioned rolling update (canary)
# Update only pods with index >= partition
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: postgres
spec:
  updateStrategy:
    type: RollingUpdate
    rollingUpdate:
      partition: 2  # Only updates pod-2 and above
            

Ordering Considerations: Ordered deployment means scaling operations take longer for larger StatefulSets. Partitioned rolling updates enable canary deployments and controlled rollouts. Plan for the time required to start/stop pods sequentially.

Headless Services: Direct Pod Access

A headless service is a service with clusterIP: None. Instead of load balancing to a cluster IP, DNS resolves to the individual pod IPs, enabling direct pod-to-pod communication.

                # Headless service definition
apiVersion: v1
kind: Service
metadata:
  name: postgres
spec:
  clusterIP: None
  selector:
    app: postgres
  ports:
  - port: 5432
    targetPort: 5432

# DNS resolution for StatefulSet pods
# Each pod gets a stable DNS entry:
# <pod-name>.<service-name>.<namespace>.svc.cluster.local
# Example:
# postgres-0.postgres.default.svc.cluster.local
# postgres-1.postgres.default.svc.cluster.local
# postgres-2.postgres.default.svc.cluster.local

# Use in application configuration
# PostgreSQL replication configuration
cat <> postgres.conf
primary_conninfo = 'host=postgres-0.postgres.default.svc.cluster.local user=replicator password=replicator'
EOF

# Testing DNS resolution from a pod
kubectl run test --image=busybox -it --rm --restart=Never -- nslookup postgres.default.svc.cluster.local

# For StatefulSets, DNS returns all pod IPs
# For individual pod access, use the pod-specific DNS name
            

Headless Service Benefits:

Direct pod-to-pod communication without load balancing
Stable DNS names for each pod
Essential for clustering and replication
Enables service discovery for stateful applications
Supports custom load balancing strategies

Stable Storage with volumeClaimTemplates

StatefulSets use volumeClaimTemplates to automatically create PersistentVolumeClaims for each pod. Each pod gets its own PVC, ensuring data is retained even if the pod is rescheduled.

                # volumeClaimTemplates in StatefulSet
volumeClaimTemplates:
- metadata:
    name: data
  spec:
    accessModes:
    - ReadWriteOnce
    resources:
      requests:
        storage: 10Gi
    storageClassName: standard

# Each pod gets its own PVC
# postgres-0 → pvc-data-postgres-0
# postgres-1 → pvc-data-postgres-1
# postgres-2 → pvc-data-postgres-2

# View PVCs
kubectl get pvc
# NAME                   STATUS   VOLUME   CAPACITY   ACCESS MODES
# data-postgres-0        Bound    pv-123   10Gi       RWO
# data-postgres-1        Bound    pv-456   10Gi       RWO
# data-postgres-2        Bound    pv-789   10Gi       RWO

# Storage retention after pod deletion
# PVCs are NOT deleted when pods are deleted
# They persist until the StatefulSet or PVC is deleted

# Manual PVC cleanup
kubectl delete pvc data-postgres-0

# Important: When scaling down, PVCs are retained
# This prevents data loss if you scale back up
            

Storage Best Practices:

Use volumeClaimTemplates for automatic provisioning
PVCs persist even when pods are deleted (data safety)
Manual cleanup required for removed PVCs
Consider StorageClasses for different storage tiers
Backup PVC data regularly
Monitor PVC usage and capacity

PodDisruptionBudget: Protecting Availability

PodDisruptionBudget (PDB) limits the number of pods that can be voluntarily evicted during cluster operations like node drains or cluster upgrades.

                # PodDisruptionBudget for StatefulSet
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: postgres-pdb
  labels:
    app: postgres
spec:
  minAvailable: 2  # At least 2 pods must be available
  selector:
    matchLabels:
      app: postgres

# Alternative: maxUnavailable
# spec:
#   maxUnavailable: 1  # Only 1 pod can be unavailable at a time

# Check PDB status
kubectl get pdb
kubectl describe pdb postgres-pdb
            

PDB Considerations: PDBs are essential for maintaining high availability of stateful applications. Without PDBs, cluster operations like node drains can evict all pods simultaneously, causing application downtime.

Common Stateful Applications on Kubernetes

PostgreSQL

Relational database

PostgreSQL StatefulSet with replication, primary-replica architecture using headless services for discovery.

Transaction processing

Apache Kafka

Distributed streaming

Kafka brokers as StatefulSets with persistent storage, ordered deployment for broker IDs, and headless services for peer discovery.

Event streaming, messaging

Elasticsearch

Search and analytics

Elasticsearch cluster with master and data nodes as StatefulSets, using persistent storage for indexes and discovery via headless services.

Logging, search

MySQL

Relational database

MySQL StatefulSet with Group Replication or InnoDB Cluster, using persistent volumes for data and ordered deployment for cluster initialization.

Transaction processing

MongoDB

NoSQL database

MongoDB replica set as a StatefulSet with persistent storage, ordered deployment, and headless services for replica discovery.

Document storage

etcd

Distributed key-value store

etcd cluster as a StatefulSet with persistent storage, ordered deployment for cluster formation, and headless services for peer communication.

Service discovery, configuration

Deployment vs StatefulSet Comparison

Feature	Deployment	StatefulSet
Pod Naming	Random names	Stable, ordered (-0, -1, -2)
Network Identity	Transient	Stable (headless service DNS)
Storage	Ephemeral (unless PVC shared)	Persistent (per-pod PVC)
Deployment Order	Parallel	Ordered (0, 1, 2...)
Scaling	Parallel	Ordered (create/delete in order)
Rolling Update	Parallel	Ordered (reverse order)
Use Case	Stateless applications	Stateful applications

Frequently Asked Questions

When should I use StatefulSet instead of Deployment?

Use StatefulSet when your application requires stable network identities, ordered deployment/scaling, or persistent storage per pod. Examples: databases, message queues, distributed systems. Use Deployment for stateless applications that don't have these requirements.

What is a headless service and why is it needed?

A headless service (clusterIP: None) doesn't provide load balancing. Instead, DNS resolves to individual pod IPs. This is essential for StatefulSets because pods need direct communication with specific peers (e.g., database replication, leader election).

How does ordered deployment work in StatefulSets?

Pods are created in ascending order (0, 1, 2...). Each pod must be Running and Ready before the next one starts. Deleting/scaling down happens in reverse order. This is crucial for applications that require leader election or master-slave setup.

What happens to PVCs when a StatefulSet is deleted?

By default, PVCs are NOT deleted when the StatefulSet is deleted. This prevents accidental data loss. You must manually delete PVCs if you want to remove the storage. Use the --cascade=foreground or --cascade=orphan flags to control behavior.

How do I scale a StatefulSet?

Use kubectl scale statefulset <name> --replicas=<count>. Scaling up creates new pods in ascending order. Scaling down deletes pods in descending order. Scaling operations are sequential, not parallel.

What is a partitioned rolling update?

A partitioned rolling update updates only pods with indices greater than or equal to the partition number. This enables canary testing: update a subset of pods (e.g., partition: 2 updates pod-2 and above), test, then reduce the partition to update the rest.

How do I back up data in a StatefulSet?

Back up PVC data using volume snapshots (CSI), Velero (application-level backup), or database-specific backup tools (pg_dump, mysqldump). Use cron jobs for scheduled backups. Store backups in a different location for disaster recovery.

Can I convert a Deployment to a StatefulSet?

Direct conversion isn't supported. You need to create a new StatefulSet and migrate data. Use a blue-green approach: create the StatefulSet, replicate data, switch traffic, then remove the Deployment. Plan for downtime or use a migration strategy.

Previous: Kubernetes Storage Next: Backup & Restore

Stateful applications are a critical part of many Kubernetes deployments. Master StatefulSets, headless services, and persistent storage to run databases and stateful workloads reliably in production.