Home / Blogs / Orchestration / Kubernetes Production / Logging with EFK/ELK

Logging with EFK/ELK

Elasticsearch Fluentd Kibana Log Collection Log Aggregation

A comprehensive guide to Kubernetes logging with EFK/ELK stack covering Elasticsearch, Fluentd, Kibana, log collection, aggregation, analysis, and practical implementation strategies for production observability.

Elasticsearch Fluentd Kibana Log Aggregation
Why Centralized Logging Matters

Centralized logging is essential for operating Kubernetes clusters effectively. It provides:

  • Visibility: See logs from all pods, nodes, and components in one place
  • Troubleshooting: Quickly find error logs and identify root causes
  • Searchability: Powerful search capabilities across millions of log entries
  • Correlation: Connect logs with metrics and traces for complete observability
  • Alerting: Create alerts based on log patterns (errors, security events)
  • Compliance: Meet regulatory requirements for log retention
Logging vs. Monitoring: Monitoring (Prometheus) provides numerical metrics about system health. Logging captures detailed events, errors, and application output. Both are essential for complete observability. The EFK/ELK stack provides the logging component.
EFK vs ELK: What's the Difference?

The EFK stack (Elasticsearch, Fluentd, Kibana) uses Fluentd as the log collector. The ELK stack (Elasticsearch, Logstash, Kibana) uses Logstash. EFK is generally preferred in Kubernetes environments because:

  • Fluentd is lighter and more resource-efficient than Logstash
  • Fluentd integrates better with Kubernetes DaemonSets
  • Fluentd has robust Kubernetes plugins for log discovery
  • Fluentd supports multiple output destinations

Elasticsearch

Storage and search engine
Elasticsearch is a distributed, RESTful search and analytics engine. It stores all logs, indexes them for fast search, and provides powerful query capabilities.
Log storage and indexing

Fluentd

Log collector and forwarder
Fluentd collects logs from various sources, parses and filters them, and forwards them to Elasticsearch. Lightweight and flexible with many plugins.
Log collection and processing

Kibana

Visualization and analysis
Kibana provides a web interface for searching, visualizing, and analyzing logs stored in Elasticsearch. Includes dashboards, charts, and log exploration.
Log visualization and analysis
Installing the EFK Stack
# Using Helm to install EFK stack helm repo add elastic https://helm.elastic.co helm repo update # Install Elasticsearch helm install elasticsearch elastic/elasticsearch \ --namespace logging \ --create-namespace \ --set replicas=3 \ --set resources.requests.memory=2Gi \ --set resources.requests.cpu=1 # Install Kibana helm install kibana elastic/kibana \ --namespace logging \ --set elasticsearchHosts=http://elasticsearch-master:9200 \ --set resources.requests.memory=512Mi \ --set resources.requests.cpu=0.5 # Install Fluentd (using fluentd-kubernetes-daemonset) # Method 1: Using fluent/fluentd-kubernetes-daemonset kubectl apply -f https://raw.githubusercontent.com/fluent/fluentd-kubernetes-daemonset/master/fluentd-daemonset-elasticsearch.yaml # Method 2: Using Helm (fluentd-elasticsearch) helm repo add fluent https://fluent.github.io/helm-charts helm repo update helm install fluentd fluent/fluentd-elasticsearch \ --namespace logging \ --set elasticsearch.host=elasticsearch-master \ --set elasticsearch.port=9200 \ --set elasticsearch.scheme=http # Verify installation kubectl get pods -n logging kubectl get svc -n logging # Access Kibana kubectl port-forward -n logging svc/kibana-kibana 5601:5601
Resource Requirements: Elasticsearch can be resource-intensive. Allocate sufficient memory and storage for your cluster. For production, use a dedicated storage class for Elasticsearch persistent volumes.
Fluentd: Log Collection and Processing

Fluentd runs as a DaemonSet on each node, collecting logs from containers and system components.

# Fluentd DaemonSet Configuration apiVersion: apps/v1 kind: DaemonSet metadata: name: fluentd namespace: logging spec: selector: matchLabels: app: fluentd template: metadata: labels: app: fluentd spec: serviceAccountName: fluentd tolerations: - key: node-role.kubernetes.io/master effect: NoSchedule containers: - name: fluentd image: fluent/fluentd-kubernetes-daemonset:v1-debian-elasticsearch env: - name: FLUENT_ELASTICSEARCH_HOST value: "elasticsearch-master" - name: FLUENT_ELASTICSEARCH_PORT value: "9200" - name: FLUENT_ELASTICSEARCH_SCHEME value: "http" - name: FLUENT_ELASTICSEARCH_LOGSTASH_PREFIX value: "logstash" - name: FLUENT_ELASTICSEARCH_INDEX_NAME value: "logstash-%Y.%m.%d" - name: FLUENT_ELASTICSEARCH_TYPE_NAME value: "fluentd" volumeMounts: - name: varlog mountPath: /var/log - name: varlibdockercontainers mountPath: /var/lib/docker/containers readOnly: true - name: runlog mountPath: /run/log - name: fluentd-config mountPath: /fluentd/etc volumes: - name: varlog hostPath: path: /var/log - name: varlibdockercontainers hostPath: path: /var/lib/docker/containers - name: runlog hostPath: path: /run/log - name: fluentd-config configMap: name: fluentd-config # Fluentd ConfigMap (main configuration) apiVersion: v1 kind: ConfigMap metadata: name: fluentd-config namespace: logging data: fluent.conf: | # System configuration log_level info # Include Kubernetes metadata @type tail @id in_tail_container_logs path /var/log/containers/*.log pos_file /var/log/fluentd-containers.log.pos tag kubernetes.* read_from_head true @type json time_format %Y-%m-%dT%H:%M:%S.%NZ # Filter to add Kubernetes metadata @type kubernetes_metadata @id filter_kube_metadata # Add tag for indexing @type record_transformer @id filter_kube_metadata_transform index_name logstash-${record['time'] ? Time.at(record['time']).to_datetime.strftime("%Y.%m.%d") : Time.now.to_datetime.strftime("%Y.%m.%d")} # Output to Elasticsearch @type elasticsearch @id out_es host elasticsearch-master port 9200 scheme http logstash_format true logstash_prefix logstash logstash_dateformat %Y.%m.%d @type memory flush_thread_count 4 flush_interval 5s chunk_limit_size 2M queue_limit_length 32 retry_max_interval 30 # System logs @type tail @id in_tail_system_logs path /var/log/messages pos_file /var/log/fluentd-system.log.pos tag system @type syslog
Fluentd Best Practices:
  • Use buffer settings to prevent data loss during network issues
  • Use pos_file to track read position and avoid duplicates
  • Add Kubernetes metadata (namespace, pod, container) to each log entry
  • Use logstash_format for date-based indexing
  • Configure retry and flush settings for reliability
Elasticsearch: Log Storage and Indexing

Elasticsearch stores, indexes, and provides search capabilities for all collected logs.

# Elasticsearch Cluster Configuration apiVersion: elasticsearch.k8s.elastic.co/v1 kind: Elasticsearch metadata: name: elasticsearch namespace: logging spec: version: 8.10.0 nodeSets: - name: master count: 3 config: node.roles: ["master"] xpack.security.enabled: false volumeClaimTemplates: - metadata: name: elasticsearch-data spec: accessModes: - ReadWriteOnce resources: requests: storage: 10Gi storageClassName: standard - name: data count: 3 config: node.roles: ["data", "ingest"] volumeClaimTemplates: - metadata: name: elasticsearch-data spec: accessModes: - ReadWriteOnce resources: requests: storage: 50Gi storageClassName: standard # Index Lifecycle Management (ILM) Policy apiVersion: v1 kind: ConfigMap metadata: name: ilm-policy namespace: logging data: policy.json: | { "policy": { "phases": { "hot": { "min_age": "0ms", "actions": { "rollover": { "max_size": "50gb", "max_age": "30d" } } }, "warm": { "min_age": "30d", "actions": { "shrink": { "number_of_shards": 1 }, "forcemerge": { "max_num_segments": 1 } } }, "cold": { "min_age": "60d", "actions": { "freeze": {} } }, "delete": { "min_age": "90d", "actions": { "delete": {} } } } } } # Elasticsearch Index Template apiVersion: v1 kind: ConfigMap metadata: name: index-template namespace: logging data: template.json: | { "index_patterns": ["logstash-*"], "settings": { "number_of_shards": 3, "number_of_replicas": 1, "refresh_interval": "5s" }, "mappings": { "properties": { "time": {"type": "date"}, "kubernetes": { "properties": { "pod_name": {"type": "keyword"}, "namespace_name": {"type": "keyword"}, "container_name": {"type": "keyword"}, "host": {"type": "keyword"} } }, "message": {"type": "text"}, "log": {"type": "text"}, "level": {"type": "keyword"}, "severity": {"type": "keyword"} } } }
Elasticsearch Best Practices:
  • Use index lifecycle management (ILM) for automated index rotation and cleanup
  • Set appropriate number of shards and replicas for performance and redundancy
  • Monitor Elasticsearch health, disk usage, and performance
  • Use dedicated master and data nodes for production
  • Implement backup and restore strategies for long-term retention
Kibana: Visualization and Analysis

Kibana provides a web interface for searching, visualizing, and analyzing logs stored in Elasticsearch.

# Kibana Configuration apiVersion: kibana.k8s.elastic.co/v1 kind: Kibana metadata: name: kibana namespace: logging spec: version: 8.10.0 count: 1 elasticsearchRef: name: elasticsearch config: elasticsearch.hosts: - http://elasticsearch-master:9200 elasticsearch.requestTimeout: 30000 server.publicBaseUrl: https://kibana.example.com xpack.security.enabled: false xpack.spaces.enabled: false podTemplate: spec: containers: - name: kibana resources: requests: memory: 512Mi cpu: 0.5 limits: memory: 1Gi cpu: 1 # Kibana Index Pattern (created via API) # Create index pattern for logstash-* curl -X POST "http://localhost:5601/api/saved_objects/index-pattern" \ -H "Content-Type: application/json" \ -H "kbn-xsrf: true" \ -d '{ "attributes": { "title": "logstash-*", "timeFieldName": "time" } }' # Kibana Dashboard Configuration apiVersion: v1 kind: ConfigMap metadata: name: kibana-dashboards namespace: logging data: dashboard.json: | { "title": "Kubernetes Log Dashboard", "panels": [ { "type": "line", "title": "Log Volume by Namespace", "query": "namespace: *", "x_axis": "time", "y_axis": "count" }, { "type": "bar", "title": "Error Count by Service", "query": "level: error", "x_axis": "kubernetes.namespace_name", "y_axis": "count" }, { "type": "text", "title": "Recent Errors", "query": "level: error", "order": "desc", "size": 50 } ] }
Kibana Best Practices:
  • Create index patterns for each log type
  • Build custom dashboards for different teams and use cases
  • Use Kibana alerts for log-based alerting
  • Implement role-based access control for Kibana
  • Use saved searches and visualizations for common queries
Log Collection Strategies

DaemonSet Method

Node-level collection
Fluentd runs as a DaemonSet on each node, collecting logs from all pods. This is the most common and efficient method for Kubernetes logging.
Standard logging

Sidecar Method

Pod-level collection
Each pod runs a sidecar container that collects and forwards logs. Useful for applications with custom log formats or when you need per-pod control.
Custom log formats

Streaming Method

Direct streaming
Applications stream logs directly to Fluentd via HTTP, TCP, or other protocols. Requires application modification but provides real-time logging.
Real-time logging
# Sidecar Logging Example apiVersion: v1 kind: Pod metadata: name: app-with-sidecar spec: containers: - name: app image: myapp:latest volumeMounts: - name: varlog mountPath: /var/log/myapp - name: log-shipper image: fluent/fluentd:latest volumeMounts: - name: varlog mountPath: /var/log/myapp env: - name: FLUENT_ELASTICSEARCH_HOST value: "elasticsearch-master" - name: FLUENT_ELASTICSEARCH_PORT value: "9200" volumes: - name: varlog emptyDir: {}
Log Processing and Enrichment

Fluentd can parse, filter, and enrich logs before sending them to Elasticsearch.

# Fluentd Config for Log Processing # Parse JSON logs @type tail path /var/log/containers/*.log pos_file /var/log/fluentd-containers.log.pos tag kubernetes.* @type json time_format %Y-%m-%dT%H:%M:%S.%NZ # Add Kubernetes metadata @type kubernetes_metadata @id filter_kube_metadata # Add custom fields @type record_transformer @id filter_transform severity ${record["level"] == "error" ? "ERROR" : "INFO"} environment ${record["kubernetes"]["namespace_name"] == "production" ? "prod" : "dev"} cluster "${ENV['CLUSTER_NAME'] || 'prod'}" # Remove sensitive fields @type record_transformer @id filter_remove_sensitive remove_keys password, token, secret, api_key # Parse structured logs (NGINX, Apache) @type grep key kubernetes.container_name pattern /nginx/ @type parser key_name log @type nginx reserve_data true # Multiline logs (Java stack traces) @type tail path /var/log/containers/*.log @type multiline format_firstline /^[^\s]/ format1 /^(?
Log Processing Best Practices:
  • Add Kubernetes metadata for context (namespace, pod, container)
  • Parse structured logs (JSON, key-value) for better searchability
  • Remove sensitive information before storing
  • Handle multiline logs (stack traces) properly
  • Use filtering to drop noisy or unnecessary logs
Logging Solutions Comparison
Feature EFK (Fluentd) ELK (Logstash) Loki DataDog
Resource Usage Low High Very Low Medium
Full-Text Search Yes Yes Limited Yes
Indexing Speed Fast Medium Fast Fast
Query Language Elasticsearch Elasticsearch LogQL Custom
Retention Management Yes (ILM) Yes (ILM) Limited Yes
Cost Free Free Free Paid
Complexity Medium High Low Low
Best For Production clusters Complex pipelines Cost-sensitive Enterprise
Choosing a Solution: EFK is recommended for most production Kubernetes clusters due to its balance of features, performance, and cost. Loki is excellent for cost-sensitive environments. ELK is better for complex log processing pipelines. DataDog is a paid SaaS solution with integrated observability.
Frequently Asked Questions
What is the difference between EFK and ELK?
EFK uses Fluentd as the log collector, while ELK uses Logstash. Fluentd is lighter and more resource-efficient, making it better suited for Kubernetes environments. Both use Elasticsearch for storage and Kibana for visualization.
Why use Fluentd instead of Logstash in Kubernetes?
Fluentd is lighter, uses less memory, and has better Kubernetes integration with DaemonSets. It also has robust plugins for Kubernetes metadata and log discovery. Logstash is more powerful but heavier and slower.
How do I handle log rotation in Kubernetes?
Configure container log rotation in the container runtime (containerd, Docker) using log rotation settings. Use fluentd's pos_file to track read position and avoid losing logs during rotation.
How do I secure the EFK stack?
Enable TLS for Elasticsearch and Kibana. Use authentication (Basic Auth, API keys). Implement RBAC for Kibana. Use network policies to restrict access. Encrypt data at rest using disk encryption.
What is the recommended log retention period?
Depends on compliance requirements. Typically 30-90 days for production. Use Index Lifecycle Management (ILM) to automate retention: hot (7-30 days), warm (30-60 days), cold (60-90 days), delete (after 90 days).
How do I collect logs from multiple clusters?
Use a central Elasticsearch cluster with dedicated indices per cluster. Configure Fluentd in each cluster to forward logs to the central Elasticsearch. Add cluster name as a field for filtering.
What are alternatives to Elasticsearch for logging?
Loki (lightweight, cost-effective), Splunk (enterprise), DataDog (SaaS), or cloud-native services like AWS OpenSearch, Azure Log Analytics, and GCP Cloud Logging.
How do I troubleshoot Fluentd log collection issues?
Check Fluentd logs (kubectl logs -n logging fluentd-xxxx), verify log path permissions, check pos_file position, test Elasticsearch connectivity, and validate Fluentd configuration syntax.
Previous: Kubernetes Monitoring Next: Distributed Tracing

Centralized logging with EFK/ELK is essential for operating Kubernetes clusters at scale. Start with a basic EFK stack, then expand with custom log parsing, dashboards, and alerting as your needs grow.