Alerting Best Practices

Alert Rules Receiver Configuration On-Call Management Runbooks

A comprehensive guide to Kubernetes alerting best practices covering alert rules, receiver configuration, on-call management, runbooks, and practical implementation strategies for effective incident response.

Alert Rules On-Call Management Runbooks Kubernetes Native

Why Alerting Matters

Alerting is the bridge between monitoring and incident response. Effective alerting ensures that the right people are notified at the right time about the right issues. Poor alerting leads to:

Alert Fatigue: Too many alerts causing teams to ignore important notifications
Missed Incidents: Important issues being overlooked or delayed
Slow Response: Delays in detecting and responding to problems
Burned Out Teams: Constant alerting leading to team exhaustion

Key Alerting Principles:

Actionable: Every alert should require a specific action
Accurate: Minimize false positives and negatives
Timely: Notify when action is needed, not too early or too late
Contextual: Provide enough information to understand the issue
Throttled: Avoid alert storms and duplicates

Alerting Components

Alert Rules

Define what triggers alerts

Alert rules are Prometheus expressions that evaluate metric conditions. When a condition is met, an alert is fired. Rules include thresholds, time windows, and severity levels.

Condition-based alerting

AlertManager

Alert routing and management

AlertManager handles alerts from Prometheus, groups them, deduplicates, and routes to receivers. It also manages inhibition and silence rules.

Alert management

Receivers

Notification channels

Receivers define where alerts are sent: email, Slack, PagerDuty, Opsgenie, webhooks, and more. Each receiver has specific configuration requirements.

Notification delivery

On-Call Management

Personnel rotation and escalation

On-call management defines who receives alerts based on schedules, rotations, and escalation policies. Integrates with PagerDuty, Opsgenie, and similar tools.

Team coordination

Runbooks

Incident response guides

Runbooks provide step-by-step instructions for responding to alerts. They reduce response time and ensure consistent incident handling.

Incident response

Alert Rules Best Practices

                # Prometheus Alert Rules
groups:
- name: kubernetes-alerts
  interval: 30s
  rules:
  # Good: Specific, actionable, with clear threshold
  - alert: HighCPUUsage
    expr: 100 - (avg(rate(node_cpu_seconds_total{mode="idle"}[5m])) * 100) > 80
    for: 5m  # Prevents flapping
    labels:
      severity: warning
      team: platform
    annotations:
      summary: "High CPU usage on node {{ $labels.node }}"
      description: "Node {{ $labels.node }} CPU usage is {{ $value }}% for 5 minutes"
      runbook: "https://wiki.example.com/high-cpu"

  # Bad: Too broad, no context
  - alert: PodProblems
    expr: kube_pod_container_status_restarts_total > 0
    # No for clause = noise

  # Good: Error rate with threshold
  - alert: HighErrorRate
    expr: sum(rate(nginx_ingress_controller_requests{status=~"5.."}[5m])) / sum(rate(nginx_ingress_controller_requests[5m])) > 0.05
    for: 2m
    labels:
      severity: critical
      team: app
    annotations:
      summary: "High error rate on ingress"
      description: "5xx error rate is {{ $value | humanizePercentage }}"

  # Good: Service down alert
  - alert: ServiceDown
    expr: up{job="kubernetes-service"} == 0
    for: 1m
    labels:
      severity: critical
    annotations:
      summary: "Service {{ $labels.service }} is down"
      description: "Service {{ $labels.service }} has been down for 1 minute"

  # Good: Memory usage with trend
  - alert: MemoryPressure
    expr: (sum(node_memory_MemTotal_bytes - node_memory_MemAvailable_bytes) / sum(node_memory_MemTotal_bytes)) * 100 > 90
    for: 10m
    labels:
      severity: warning
    annotations:
      summary: "Cluster memory pressure"
      description: "Memory usage is {{ $value }}% for 10 minutes"
            

Alert Rule Best Practices:

Use for clauses to prevent flapping (avoid alerts on transient spikes)
Set appropriate severity levels (critical, warning, info)
Include actionable descriptions with context
Add runbook links for easy escalation
Use team labels for routing to the right team
Avoid alerting on every minor issue
Use recording rules for complex queries

Receiver Configuration

                # AlertManager Configuration
route:
  group_by: ['alertname', 'cluster', 'service']
  group_wait: 30s      # Wait to group alerts
  group_interval: 5m   # Wait between groups
  repeat_interval: 4h  # Resend after 4 hours if unresolved
  receiver: 'slack-general'
  routes:
  # Critical alerts go to PagerDuty
  - match:
      severity: critical
    receiver: 'pagerduty-critical'
    continue: true
  # Warning alerts go to Slack
  - match:
      severity: warning
    receiver: 'slack-warning'
  # Team-specific routing
  - match:
      team: platform
    receiver: 'slack-platform'
  - match:
      team: app
    receiver: 'slack-app'

receivers:
- name: 'slack-general'
  slack_configs:
  - api_url: 'https://hooks.slack.com/services/XXX/XXX/XXX'
    channel: '#alerts'
    title: '{{ range .Alerts }}{{ .Annotations.summary }}{{ end }}'
    text: |
      {{ range .Alerts }}
      *Severity:* {{ .Labels.severity }}
      *Summary:* {{ .Annotations.summary }}
      *Description:* {{ .Annotations.description }}
      *Runbook:* {{ .Annotations.runbook }}
      {{ end }}

- name: 'slack-warning'
  slack_configs:
  - channel: '#warnings'
    color: 'warning'

- name: 'slack-platform'
  slack_configs:
  - channel: '#platform-alerts'

- name: 'slack-app'
  slack_configs:
  - channel: '#app-alerts'

- name: 'pagerduty-critical'
  pagerduty_configs:
  - service_key: 'your-pagerduty-key'
    routing_key: 'your-routing-key'
    severity: 'critical'

# Email receiver
- name: 'email-alerts'
  email_configs:
  - to: 'oncall@example.com'
    from: 'alertmanager@example.com'
    smarthost: 'smtp.example.com:587'
    auth_username: 'user'
    auth_password: 'password'

# Webhook receiver
- name: 'webhook'
  webhook_configs:
  - url: 'http://webhook-service:8080/alerts'
    send_resolved: true

# Inhibit rules - suppress related alerts
inhibit_rules:
- source_match:
    severity: 'critical'
  target_match:
    severity: 'warning'
  equal: ['alertname', 'cluster']
            

Receiver Configuration Tips:

Never hardcode API keys or secrets in config files (use environment variables or secrets)
Set appropriate group_wait and group_interval to avoid alert storms
Use repeat_interval to prevent alert fatigue
Route alerts to the right teams using labels
Test webhooks and integrations before deploying

On-Call Management

                # PagerDuty Integration
# Configure PagerDuty service with AlertManager
receivers:
- name: 'pagerduty'
  pagerduty_configs:
  - service_key: 'your-service-key'
    routing_key: 'your-routing-key'
    severity: 'critical'

# PagerDuty escalation policy
# Level 1: Primary on-call (weekday)
# Level 2: Secondary on-call (weekend)
# Level 3: Engineering manager

# Opsgenie Integration
- name: 'opsgenie'
  opsgenie_configs:
  - api_key: 'your-api-key'
    api_url: 'https://api.opsgenie.com'
    message: 'Alert: {{ .Annotations.summary }}'
    description: '{{ .Annotations.description }}'
    responders:
    - name: 'platform-team'
      type: 'team'

# AlertManager with Opsgenie routing
route:
  routes:
  - match:
      team: platform
    receiver: 'opsgenie-platform'
  - match:
      team: app
    receiver: 'opsgenie-app'

# On-call Schedule in Opsgenie
# apiVersion: opsgenie.com/v1
# kind: Schedule
# metadata:
#   name: platform-oncall
# spec:
#   name: "Platform Team On-Call"
#   timezone: "America/New_York"
#   rotations:
#   - name: "Primary"
#     participants:
#     - "user1@example.com"
#     - "user2@example.com"
#     - "user3@example.com"
#     timeRestriction:
#       type: "weekly"
#       restrictions:
#       - startDay: "monday"
#         startTime: "09:00"
#         endDay: "friday"
#         endTime: "17:00"

# On-call Rotation Manager (with Slack)
# Use Slack integrations to notify on-call team
- name: 'slack-oncall'
  slack_configs:
  - channel: '#oncall-platform'
    text: |
      @channel
      *Alert:* {{ .Annotations.summary }}
      *Description:* {{ .Annotations.description }}
      *Severity:* {{ .Labels.severity }}
            

On-Call Best Practices:

Define clear escalation policies with multiple levels
Rotate on-call responsibilities regularly (weekly)
Use primary and secondary on-call for redundancy
Integrate with PagerDuty or Opsgenie for robust management
Notify the team on Slack for visibility
Use Opsgenie/PagerDuty mobile apps for alerts
Document on-call responsibilities and expectations

Runbooks: Incident Response Guides

Runbooks provide step-by-step instructions for responding to alerts. They reduce response time and ensure consistent incident handling.

                # Example Runbook: High CPU Usage
---
name: High CPU Usage
description: Response procedure for high CPU alerts
severity: warning
tags:
  - performance
  - capacity

steps:
  - step: 1
    action: "Check CPU usage across nodes"
    command: "kubectl top nodes"
    expected: "CPU usage should be below 80%"

  - step: 2
    action: "Identify high-CPU pods"
    command: "kubectl top pods --all-namespaces | sort -k3 -rn | head -20"
    expected: "Identify pods with high CPU usage"

  - step: 3
    action: "Check pod logs for issues"
    command: "kubectl logs -n {{ namespace }} {{ pod-name }} --tail=100"
    expected: "Look for errors or high resource consumption"

  - step: 4
    action: "Check resource limits"
    command: "kubectl describe pod {{ pod-name }} -n {{ namespace }}"
    expected: "Verify CPU requests and limits are appropriate"

  - step: 5
    action: "Scale application if needed"
    command: "kubectl scale deployment {{ deployment-name }} -n {{ namespace }} --replicas={{ new-replicas }}"
    expected: "Reduce CPU pressure on individual pods"

  - step: 6
    action: "If unresolved, escalate to team lead"
    contact: "#platform-team on Slack"

# Runbook Template
---
name: {{ alert-name }}
description: {{ description }}
severity: {{ severity }}
tags:
  - {{ tag1 }}
  - {{ tag2 }}

steps:
  - step: 1
    action: "{{ action }}"
    command: "{{ command }}"
    expected: "{{ expected }}"

  - step: 2
    action: "{{ action }}"
    command: "{{ command }}"
    expected: "{{ expected }}"

  - step: 3
    action: "{{ action }}"
    command: "{{ command }}"
    expected: "{{ expected }}"

escalation:
  level1: "Contact primary on-call"
  level2: "Contact team lead"
  level3: "Contact engineering manager"

# Store runbooks in Git
# /runbooks/high-cpu.md
# /runbooks/service-down.md
# /runbooks/memory-pressure.md
            

Runbook Best Practices:

Create runbooks for every critical alert
Keep runbooks concise and actionable
Include commands with exact syntax
Define clear escalation paths
Store runbooks in Git for version control
Review and update runbooks regularly
Test runbooks during incident drills
Include links to dashboards and logs

Common Alert Categories

Infrastructure Alerts

Node and cluster health

CPU, memory, disk, network issues. Node not ready, high load, disk space, network latency.

Infrastructure monitoring

Application Alerts

Application health and errors

Service down, high error rate, slow response, job failures, pod restarts.

Application monitoring

Security Alerts

Security incidents

Unauthorized access, policy violations, vulnerability alerts, suspicious activity.

Security monitoring

Capacity Alerts

Resource usage and trends

High resource usage, approaching limits, storage capacity, pod count limits.

Capacity planning

Alerting Maturity Model

Improving alerting quality is a journey. Use this maturity model to assess and improve your alerting practices:

Level	Characteristics	Key Practices
Level 1: Reactive	Ad-hoc alerts, high noise, frequent false positives	Setup basic monitoring, establish on-call
Level 2: Organized	Defined alert rules, severity levels, basic runbooks	Create runbooks, use AlertManager grouping
Level 3: Proactive	Automated responses, low false positive rate, team routing	Automate remediation, team-specific routing
Level 4: Optimized	Predictive alerts, self-healing, continuous improvement	Machine learning for anomalies, auto-scaling

Improvement Tips: Start with Level 1 and gradually improve. Measure alert quality with metrics like false positive rate, Mean Time To Detect (MTTD), and Mean Time To Resolve (MTTR). Regularly review and refine alerts.

Frequently Asked Questions

How do I prevent alert fatigue?

Reduce alert fatigue by: using grouping and inhibition in AlertManager, setting appropriate thresholds and for-clauses, routing alerts to the right teams, avoiding alerts for non-actionable issues, and regularly reviewing and pruning alerts.

What is the difference between warning and critical alerts?

Critical alerts indicate immediate action required (service down, data loss). Warning alerts indicate potential issues that need attention (high CPU, approaching limits). Critical alerts typically page on-call; warnings go to Slack/email.

How do I set up an on-call rotation?

Use PagerDuty or Opsgenie for on-call management. Define schedules with rotations (weekly), escalation policies (primary → secondary → manager), and integrations with AlertManager. Also configure backup and override procedures.

What is a runbook and why is it important?

A runbook is a guide that provides step-by-step instructions for responding to an alert. It reduces response time, ensures consistency, and helps new team members. Runbooks should include commands, expected outputs, and escalation paths.

How do I test alert rules without triggering them?

Use Prometheus's alertmanager web UI for testing, or use promtool for unit testing. Run promtool check rules alert.rules for syntax validation. Use recording rules for complex queries before alerting.

What is the recommended on-call schedule?

Weekly rotations are common. Have primary and secondary on-call. Ensure coverage during business hours and weekends. Use timezone-aware schedules for global teams. Provide handover time between rotations.

How do I handle alerts during maintenance?

Use AlertManager silences for planned maintenance. Silence by alertname, cluster, or service. Set a duration for the silence. Use the web UI or CLI: amtool silence add --duration=2h.

What are the metrics for measuring alert quality?

Key metrics: false positive rate, Mean Time To Detect (MTTD), Mean Time To Resolve (MTTR), alert volume per day, alerts per team, and on-call paging frequency. Track these to improve alerting.

Previous: Distributed Tracing Next: Cluster Autoscaling

Effective alerting is critical for maintaining reliable Kubernetes clusters. Implement these best practices to reduce alert fatigue, improve incident response, and build a culture of operational excellence.