Kubernetes Monitoring Stack

Prometheus Grafana AlertManager Node Exporters kube-state-metrics

A comprehensive guide to Kubernetes monitoring stack covering Prometheus, Grafana, AlertManager, node exporters, kube-state-metrics, and practical implementation strategies for production observability.

Prometheus Grafana AlertManager Node Exporters

Why Monitoring Matters

Monitoring is essential for maintaining the health, performance, and reliability of Kubernetes clusters. A comprehensive monitoring stack provides:

Visibility: Understand what's happening in your cluster
Alerting: Get notified when issues occur
Capacity Planning: Track resource usage and trends
Troubleshooting: Quickly identify and resolve issues
Performance Optimization: Identify bottlenecks and optimize
Compliance: Meet regulatory requirements

Key Monitoring Principles:

Metrics: Numerical data points (CPU, memory, latency)
Logs: Event and error messages
Traces: Request flow through distributed systems
Alerts: Notifications based on metric thresholds
Dashboards: Visual representation of monitoring data

Monitoring Stack Components

Prometheus

Metrics collection and storage

Prometheus is the industry-standard monitoring system. It scrapes metrics from exporters, stores them in a time-series database, and provides a powerful query language (PromQL).

Metrics collection

Grafana

Visualization and dashboards

Grafana provides beautiful, customizable dashboards for visualizing metrics from Prometheus and other data sources. It supports alerting and annotations.

Visualization

AlertManager

Alert management and routing

AlertManager handles alerts from Prometheus, deduplicates them, and routes notifications to receivers like email, Slack, PagerDuty, and Opsgenie.

Alerting

Node Exporter

Node-level metrics

Node Exporter collects metrics from Kubernetes nodes: CPU, memory, disk, network, and system-level information. Essential for monitoring node health.

Node monitoring

kube-state-metrics

Kubernetes object metrics

kube-state-metrics generates metrics about Kubernetes objects: pods, deployments, services, nodes, and more. Essential for cluster-level monitoring.

Cluster monitoring

Metric Exporters

Application-specific metrics

Various exporters for applications and services: Blackbox (HTTP/ICMP), MySQL, PostgreSQL, Redis, Nginx, and custom application metrics.

Application monitoring

Installing the Monitoring Stack with Helm

The easiest way to deploy a complete monitoring stack is using the kube-prometheus-stack Helm chart, which bundles Prometheus, Grafana, AlertManager, and exporters.

                # Add Prometheus Community Helm repo
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts
helm repo update

# Install kube-prometheus-stack
helm install monitoring prometheus-community/kube-prometheus-stack \
  --namespace monitoring \
  --create-namespace \
  --set grafana.adminPassword=admin \
  --set prometheus.prometheusSpec.scrapeInterval=15s \
  --set prometheus.prometheusSpec.evaluationInterval=15s \
  --set alertmanager.alertmanagerSpec.retention=120h

# Verify installation
kubectl get pods -n monitoring
kubectl get svc -n monitoring

# Access Prometheus UI (port-forward)
kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-prometheus 9090

# Access Grafana UI (port-forward)
kubectl port-forward -n monitoring svc/monitoring-grafana 3000:80

# Access AlertManager UI (port-forward)
kubectl port-forward -n monitoring svc/monitoring-kube-prometheus-alertmanager 9093

# Get Grafana admin password
kubectl get secret -n monitoring monitoring-grafana -o jsonpath="{.data.admin-password}" | base64 -d

# Alternative: Install individual components
# Prometheus
helm install prometheus prometheus-community/prometheus -n monitoring --create-namespace

# Grafana
helm install grafana grafana/grafana -n monitoring

# AlertManager
helm install alertmanager prometheus-community/alertmanager -n monitoring
            

Storage Considerations: Prometheus requires persistent storage for time-series data. Configure PVCs or use object storage for long-term retention. Default retention is typically 30 days.

Prometheus: Metrics Collection

Prometheus scrapes metrics from configured endpoints, stores them in a time-series database, and provides a powerful query language (PromQL) for analysis.

                # Prometheus Configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-config
  namespace: monitoring
data:
  prometheus.yml: |
    global:
      scrape_interval: 15s
      evaluation_interval: 15s
    scrape_configs:
    - job_name: 'kubernetes-pods'
      kubernetes_sd_configs:
      - role: pod
      relabel_configs:
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
        action: keep
        regex: true
      - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_path]
        action: replace
        target_label: __metrics_path__
        regex: (.+)
      - source_labels: [__address__, __meta_kubernetes_pod_annotation_prometheus_io_port]
        action: replace
        regex: ([^:]+)(?::\d+)?;(\d+)
        replacement: $1:$2
        target_label: __address__

    - job_name: 'kubernetes-nodes'
      scheme: https
      tls_config:
        ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
      bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
      kubernetes_sd_configs:
      - role: node
      relabel_configs:
      - action: labelmap
        regex: __meta_kubernetes_node_label_(.+)

# PromQL Queries
# CPU usage by pod
sum(rate(container_cpu_usage_seconds_total{container!="POD",container!=""}[5m])) by (pod)

# Memory usage by pod
sum(container_memory_usage_bytes{container!="POD",container!=""}) by (pod)

# Node CPU usage
100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100)

# Pod restart count
sum(kube_pod_container_status_restarts_total) by (pod)

# Deployment status
kube_deployment_status_replicas_available / kube_deployment_spec_replicas

# Pod status
kube_pod_status_phase{phase="Running"} / kube_pod_status_phase
            

Prometheus Best Practices:

Use ServiceMonitors for Kubernetes service discovery
Set appropriate scrape intervals (15-30s for production)
Configure retention and storage limits
Use recording rules for expensive queries
Monitor Prometheus itself (self-monitoring)

Grafana: Visualization and Dashboards

Grafana provides beautiful, customizable dashboards for visualizing Prometheus metrics. It supports alerts, annotations, and multiple data sources.

                # Grafana Datasource Configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-datasources
  namespace: monitoring
data:
  datasources.yaml: |
    apiVersion: 1
    datasources:
    - name: Prometheus
      type: prometheus
      access: proxy
      url: http://prometheus-operated:9090
      isDefault: true

# Grafana Dashboard Configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: grafana-dashboards
  namespace: monitoring
data:
  cluster-dashboard.json: |
    {
      "title": "Kubernetes Cluster Overview",
      "panels": [
        {
          "title": "Node CPU Usage",
          "targets": [
            {
              "expr": "100 - (avg(rate(node_cpu_seconds_total{mode=\"idle\"}[5m])) * 100)",
              "legendFormat": "CPU Usage"
            }
          ]
        },
        {
          "title": "Pod Memory Usage",
          "targets": [
            {
              "expr": "sum(container_memory_usage_bytes{container!=\"POD\",container!=\"\"}) by (pod)",
              "legendFormat": "{{pod}}"
            }
          ]
        }
      ]
    }

# Popular Kubernetes Dashboards
# Dashboard IDs (import from grafana.com)
# - 315: Kubernetes Cluster Monitoring
# - 6417: Kubernetes Pod Monitoring
# - 8686: Kubernetes Node Exporter
# - 14285: Kubernetes / Containers
# - 15757: Kubernetes API Server
# - 11871: Kubernetes etcd
# - 9614: Kubernetes CoreDNS

# Import dashboard via CLI
curl -X POST -H "Content-Type: application/json" -d '{"dashboard":{"id":315},"overwrite":true}' \
  http://admin:admin@localhost:3000/api/dashboards/import
            

Grafana Best Practices:

Use templated dashboards for reusability
Set up alerting with Grafana alerts or AlertManager
Use annotations for deployments and events
Implement role-based access control (RBAC)
Store dashboards as code (configmaps or Git)

AlertManager: Alert Routing and Management

AlertManager handles alerts from Prometheus, deduplicates, groups, and routes them to various receivers.

                # AlertManager Configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: alertmanager-config
  namespace: monitoring
data:
  alertmanager.yml: |
    global:
      slack_api_url: 'https://hooks.slack.com/services/XXX/XXX/XXX'
      pagerduty_url: 'https://events.pagerduty.com/v2/enqueue'
    route:
      group_by: ['alertname', 'cluster']
      group_wait: 30s
      group_interval: 5m
      repeat_interval: 4h
      receiver: 'slack-notifications'
      routes:
      - match:
          severity: critical
        receiver: 'pagerduty-critical'
        continue: true
      - match:
          severity: warning
        receiver: 'slack-warning'
    receivers:
    - name: 'slack-notifications'
      slack_configs:
      - channel: '#alerts'
        title: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'
        text: '{{ range .Alerts }}{{ .Annotations.description }}{{ end }}'
    - name: 'pagerduty-critical'
      pagerduty_configs:
      - service_key: 'your-pagerduty-service-key'
    - name: 'slack-warning'
      slack_configs:
      - channel: '#warnings'
    inhibit_rules:
    - source_match:
        severity: 'critical'
      target_match:
        severity: 'warning'
      equal: ['alertname', 'cluster']

# Prometheus Alert Rules
apiVersion: v1
kind: ConfigMap
metadata:
  name: prometheus-alert-rules
  namespace: monitoring
data:
  alert-rules.yaml: |
    groups:
    - name: kubernetes-alerts
      interval: 30s
      rules:
      - alert: HighCPUUsage
        expr: 100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High CPU usage on node {{ $labels.node }}"
          description: "Node {{ $labels.node }} CPU usage is {{ $value }}%"

      - alert: HighMemoryUsage
        expr: (sum(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / sum(node_memory_MemTotal_bytes)) * 100 > 85
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "High memory usage"
          description: "Memory usage is {{ $value }}%"

      - alert: PodCrashLooping
        expr: kube_pod_container_status_restarts_total > 5
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Pod is crash looping"
          description: "Pod {{ $labels.pod }} has restarted {{ $value }} times"

      - alert: NodeNotReady
        expr: kube_node_status_condition{condition="Ready",status="true"} == 0
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "Node is not ready"
          description: "Node {{ $labels.node }} is not ready"
            

Alerting Best Practices:

Define clear alert severity levels (critical, warning, info)
Use meaningful alert names and descriptions
Group alerts to avoid alert fatigue
Set appropriate thresholds to avoid false positives
Implement inhibition rules to suppress related alerts
Test alerts in staging before production

Exporters: Collecting Metrics

                # Node Exporter (DaemonSet)
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-exporter
  template:
    metadata:
      labels:
        app: node-exporter
    spec:
      hostNetwork: true
      hostPID: true
      containers:
      - name: node-exporter
        image: prom/node-exporter:latest
        ports:
        - containerPort: 9100
        args:
        - --path.procfs=/host/proc
        - --path.sysfs=/host/sys
        volumeMounts:
        - name: proc
          mountPath: /host/proc
          readOnly: true
        - name: sys
          mountPath: /host/sys
          readOnly: true
      volumes:
      - name: proc
        hostPath:
          path: /proc
      - name: sys
        hostPath:
          path: /sys

# kube-state-metrics
apiVersion: apps/v1
kind: Deployment
metadata:
  name: kube-state-metrics
  namespace: monitoring
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kube-state-metrics
  template:
    metadata:
      labels:
        app: kube-state-metrics
    spec:
      serviceAccountName: kube-state-metrics
      containers:
      - name: kube-state-metrics
        image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.10.0
        ports:
        - containerPort: 8080

# Metrics Server (for HPA)
apiVersion: v1
kind: Service
metadata:
  name: metrics-server
  namespace: kube-system
spec:
  selector:
    app: metrics-server
  ports:
  - port: 443
    targetPort: 10250

# Application Exporter (custom)
apiVersion: v1
kind: Pod
metadata:
  name: app-exporter
  namespace: monitoring
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8000"
spec:
  containers:
  - name: app-exporter
    image: myapp:latest
    ports:
    - containerPort: 8000
            

Exporter Best Practices:

Deploy Node Exporter as a DaemonSet (one per node)
Use ServiceMonitors to discover exporter endpoints
Add Prometheus annotations to pods for automatic discovery
Secure exporters with TLS and authentication
Monitor exporter health and metrics

ServiceMonitors: Service Discovery

ServiceMonitors are Prometheus Operator CRDs that define how to scrape metrics from services. They provide dynamic service discovery for Prometheus.

                # ServiceMonitor for Node Exporter
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: node-exporter
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: node-exporter
  endpoints:
  - port: metrics
    interval: 15s
    path: /metrics
  jobLabel: node-exporter
  namespaceSelector:
    matchNames:
    - monitoring

# ServiceMonitor for kube-state-metrics
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: kube-state-metrics
  namespace: monitoring
spec:
  selector:
    matchLabels:
      app: kube-state-metrics
  endpoints:
  - port: metrics
    interval: 15s
    path: /metrics

# ServiceMonitor for application
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: app-metrics
  namespace: default
spec:
  selector:
    matchLabels:
      app: my-app
  endpoints:
  - port: metrics
    interval: 30s
    path: /metrics
    bearerTokenSecret:
      name: prometheus-token
      key: token
  namespaceSelector:
    matchNames:
    - default

# Service for Prometheus metrics
apiVersion: v1
kind: Service
metadata:
  name: my-app
  namespace: default
  annotations:
    prometheus.io/scrape: "true"
    prometheus.io/port: "8080"
spec:
  selector:
    app: my-app
  ports:
  - name: metrics
    port: 8080
    targetPort: 8080
            

ServiceMonitor Requirements: ServiceMonitors require the Prometheus Operator to be installed. They provide a declarative way to configure Prometheus scraping and are the recommended approach for large clusters.

Monitoring Tools Comparison

Tool	Purpose	Data Type	Storage	Query Language
Prometheus	Metrics collection	Time-series metrics	Local TSDB	PromQL
Grafana	Visualization	Metrics, logs, traces	N/A	Various
AlertManager	Alert routing	Alerts	In-memory	N/A
Node Exporter	Node metrics	System metrics	N/A	N/A
kube-state-metrics	Kubernetes metrics	K8s object metrics	N/A	N/A
Metrics Server	Resource metrics API	CPU/Memory	In-memory	N/A

Frequently Asked Questions

What is the difference between Prometheus and Grafana?

Prometheus is a metrics collection and storage system. Grafana is a visualization and dashboard tool. They are often used together: Prometheus stores metrics, Grafana visualizes them.

What is kube-state-metrics?

kube-state-metrics is a service that generates metrics about Kubernetes objects like pods, deployments, services, and nodes. It's essential for cluster-level monitoring and alerting.

How do I monitor pod health in Kubernetes?

Use Prometheus metrics (container CPU, memory, restarts), kube-state-metrics (pod status, phase), and kubelet metrics (pod health). Set up alerts for CrashLoopBackOff, OOMKilled, and high restarts.

What are recording rules in Prometheus?

Recording rules pre-calculate frequently used queries and store them as new metrics. They improve performance and reduce query latency, especially for complex aggregations.

How do I configure AlertManager for Slack notifications?

Set slack_api_url in the global config, define a receiver with slack_configs, and route alerts to it. Provide the Slack webhook URL and channel name.

What is the Prometheus Operator?

The Prometheus Operator is a Kubernetes operator that manages Prometheus, AlertManager, and related resources. It provides CRDs like ServiceMonitor and PrometheusRule for declarative configuration.

How much storage does Prometheus need?

Storage depends on the number of metrics, scrape interval, and retention period. A general guideline is 1-2GB per day for a medium-sized cluster with 10-20 nodes. Use remote storage for longer retention.

How do I monitor custom application metrics?

Add Prometheus metrics to your application using client libraries (Go, Java, Python, etc.). Expose metrics on a /metrics endpoint. Use ServiceMonitors or pod annotations to have Prometheus scrape them.

Previous: Network Security Next: Logging with EFK/ELK

A robust monitoring stack is essential for running Kubernetes in production. Start with Prometheus, Grafana, and AlertManager, then expand with additional exporters and custom metrics as needed.