Distributed Tracing

Jaeger Zipkin OpenTelemetry Trace Collection Performance Analysis

A comprehensive guide to distributed tracing in Kubernetes covering Jaeger, Zipkin, OpenTelemetry, trace collection, performance analysis, and practical implementation strategies for microservices observability.

Jaeger Zipkin OpenTelemetry Kubernetes Native

What is Distributed Tracing?

Distributed tracing is a method of monitoring and profiling applications built on microservices architecture. It tracks requests as they traverse through different services, providing visibility into the entire request flow, including latency, errors, and dependencies.

Distributed tracing helps answer critical questions:

Where is the latency? Which service is slowing down requests?
What are the dependencies? Which services call which other services?
Where are errors occurring? Which service is failing?
What is the flow? How does a request move through the system?
What is the impact? How does a change affect performance?

Key Tracing Concepts:

Trace: The complete journey of a request through all services
Span: A single unit of work within a trace (e.g., an API call or database query)
Context Propagation: Passing trace context between services via HTTP headers
Tags: Key-value pairs that add metadata to spans
Trace ID: A unique identifier for a trace, passed between services

Distributed Tracing Components

Jaeger

Uber's distributed tracing system

Jaeger is an open-source distributed tracing system. It provides end-to-end distributed tracing, service dependency analysis, and root cause analysis. Features include adaptive sampling, multi-tenancy, and integration with Kubernetes.

Production tracing

Zipkin

Twitter's distributed tracing system

Zipkin is an open-source distributed tracing system that helps gather timing data for microservices. It's lighter and simpler than Jaeger, making it suitable for smaller deployments.

Simple tracing needs

OpenTelemetry

Unified observability standard

OpenTelemetry provides a single API and SDK for generating traces, metrics, and logs. It's the industry standard for observability instrumentation and integrates with Jaeger, Zipkin, Prometheus, and other backends.

Standard instrumentation

OpenTelemetry Collector

Trace collection and export

The OpenTelemetry Collector receives traces from applications, processes them, and exports them to various backends. It supports sampling, filtering, and multiple output destinations.

Trace aggregation

Installing Jaeger

                # Install Jaeger Operator
kubectl create -f https://github.com/jaegertracing/jaeger-operator/releases/latest/download/jaeger-operator.yaml

# Verify installation
kubectl get pods -n observability

# Create Jaeger instance
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger
  namespace: observability
spec:
  strategy: production
  storage:
    type: elasticsearch
    options:
      es:
        server-urls: http://elasticsearch-master:9200
        index-prefix: jaeger
  sampling:
    options:
      default_strategy:
        type: probabilistic
        param: 0.1  # Sample 10% of traces
  ingress:
    enabled: true
    hostname: jaeger.example.com
  agent:
    strategy: DaemonSet

# Apply Jaeger instance
kubectl apply -f jaeger-instance.yaml

# Verify Jaeger deployment
kubectl get pods -n observability -l app=jaeger
kubectl get svc -n observability -l app=jaeger

# Access Jaeger UI
kubectl port-forward -n observability svc/jaeger-query 16686:16686

# Install Jaeger with Helm (alternative)
helm repo add jaegertracing https://jaegertracing.github.io/helm-charts
helm repo update
helm install jaeger jaegertracing/jaeger \
  --namespace observability \
  --create-namespace \
  --set provisionDataStore.cassandra=false \
  --set storage.type=elasticsearch \
  --set storage.elasticsearch.host=elasticsearch-master \
  --set storage.elasticsearch.port=9200 \
  --set agent.enabled=true \
  --set collector.enabled=true \
  --set query.enabled=true
            

Storage Considerations: Jaeger supports multiple storage backends (Elasticsearch, Cassandra, Kafka). Choose based on your scale and requirements. Elasticsearch is recommended for production due to better query capabilities and scalability.

Instrumenting Applications with OpenTelemetry

OpenTelemetry is the industry standard for instrumentation. It provides SDKs for multiple languages to generate and export traces.

                # Java Application with OpenTelemetry
# Add dependencies to pom.xml
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
    <version>1.30.0</version>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-sdk</artifactId>
    <version>1.30.0</version>
</dependency>
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-exporter-jaeger</artifactId>
    <version>1.30.0</version>
</dependency>

# Java code instrumentation
import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.context.Scope;

public class OrderService {
    private static final Tracer tracer = GlobalOpenTelemetry.getTracer("order-service");

    public void createOrder(OrderRequest request) {
        Span span = tracer.spanBuilder("createOrder")
            .setAttribute("order.id", request.getId())
            .setAttribute("customer.id", request.getCustomerId())
            .startSpan();

        try (Scope scope = span.makeCurrent()) {
            // Business logic
            processPayment(request);
            updateInventory(request);
        } catch (Exception e) {
            span.setStatus(StatusCode.ERROR, e.getMessage());
            throw e;
        } finally {
            span.end();
        }
    }
}

# Python Application with OpenTelemetry
from opentelemetry import trace
from opentelemetry.sdk.trace import TracerProvider
from opentelemetry.sdk.trace.export import BatchSpanProcessor
from opentelemetry.exporter.jaeger.thrift import JaegerExporter

# Setup tracer
trace.set_tracer_provider(TracerProvider())
tracer = trace.get_tracer("order-service")

# Create Jaeger exporter
jaeger_exporter = JaegerExporter(
    agent_host_name="jaeger-agent.observability.svc.cluster.local",
    agent_port=6831,
)

# Add span processor
span_processor = BatchSpanProcessor(jaeger_exporter)
trace.get_tracer_provider().add_span_processor(span_processor)

# Instrument a function
with tracer.start_as_current_span("create_order") as span:
    span.set_attribute("order.id", order_id)
    process_payment()
    update_inventory()

# Node.js Application with OpenTelemetry
const { trace, context } = require('@opentelemetry/api');
const { NodeTracerProvider } = require('@opentelemetry/sdk-trace-node');
const { JaegerExporter } = require('@opentelemetry/exporter-jaeger');
const { SimpleSpanProcessor } = require('@opentelemetry/sdk-trace-base');

const provider = new NodeTracerProvider();
const exporter = new JaegerExporter({
    agentHost: 'jaeger-agent.observability.svc.cluster.local',
    agentPort: 6832,
});
provider.addSpanProcessor(new SimpleSpanProcessor(exporter));
provider.register();

const tracer = trace.getTracer('order-service');
tracer.startActiveSpan('createOrder', (span) => {
    span.setAttribute('order.id', orderId);
    processPayment();
    updateInventory();
    span.end();
});
            

Instrumentation Best Practices:

Use OpenTelemetry SDKs for consistent instrumentation across languages
Add meaningful span names and attributes
Use context propagation to maintain trace continuity
Set span status for errors and exceptions
Use sampling to control trace volume
Add baggage for cross-service context

OpenTelemetry Collector

The OpenTelemetry Collector provides a vendor-agnostic way to receive, process, and export telemetry data.

                # OpenTelemetry Collector Configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: otel-collector-config
  namespace: observability
data:
  config.yaml: |
    receivers:
      otlp:
        protocols:
          grpc:
            endpoint: 0.0.0.0:4317
          http:
            endpoint: 0.0.0.0:4318
      jaeger:
        protocols:
          thrift_http:
            endpoint: 0.0.0.0:14268
    processors:
      batch:
        timeout: 1s
        send_batch_size: 1024
      memory_limiter:
        check_interval: 1s
        limit_mib: 512
      attributes:
        actions:
        - key: environment
          value: production
          action: upsert
    exporters:
      jaeger:
        endpoint: jaeger-collector.observability.svc.cluster.local:14250
        tls:
          insecure: true
    service:
      pipelines:
        traces:
          receivers: [otlp, jaeger]
          processors: [memory_limiter, batch, attributes]
          exporters: [jaeger]

# Deploy OpenTelemetry Collector
apiVersion: apps/v1
kind: Deployment
metadata:
  name: otel-collector
  namespace: observability
spec:
  replicas: 2
  selector:
    matchLabels:
      app: otel-collector
  template:
    metadata:
      labels:
        app: otel-collector
    spec:
      containers:
      - name: collector
        image: otel/opentelemetry-collector-contrib:latest
        args:
        - --config=/etc/otel/config.yaml
        ports:
        - containerPort: 4317
          name: otlp-grpc
        - containerPort: 4318
          name: otlp-http
        - containerPort: 14268
          name: jaeger-thrift
        volumeMounts:
        - name: config
          mountPath: /etc/otel
      volumes:
      - name: config
        configMap:
          name: otel-collector-config
            

Collector Best Practices:

Use the Collector as a central trace aggregation point
Configure batching to reduce network overhead
Use memory limiting to prevent OOM issues
Add global attributes (environment, cluster) to all traces
Use tail-based sampling for critical traces

Context Propagation

Context propagation is the mechanism by which trace context is passed between services. It's essential for creating a complete distributed trace.

                # HTTP Headers (W3C Trace Context)
# Standard headers for context propagation:
# - traceparent: version-traceid-parentspanid-flags
# - tracestate: vendor-specific trace state

# Example: traceparent
# 00-4bf92f3577b34da6a3ce929d0e0e4736-00f067aa0ba902b7-01
# Format: version-trace_id-parent_id-flags

# HTTP Propagation Example (Client)
import requests
from opentelemetry import trace
from opentelemetry.propagate import inject

tracer = trace.get_tracer("client")
with tracer.start_as_current_span("make_request") as span:
    headers = {}
    inject(headers)  # Injects trace context into headers
    response = requests.get("http://backend-service", headers=headers)

# HTTP Propagation Example (Server)
from opentelemetry.propagate import extract
from opentelemetry.trace.propagation.tracecontext import TraceContextTextMapPropagator

def handle_request(request):
    # Extract trace context from headers
    ctx = extract(request.headers)
    with trace.start_as_current_span("handle_request", context=ctx) as span:
        span.set_attribute("http.method", request.method)
        # Process request

# gRPC Context Propagation
import grpc
from opentelemetry import trace
from opentelemetry.instrumentation.grpc import GrpcInstrumentorClient

# Client-side
with tracer.start_as_current_span("grpc_call") as span:
    # Metadata automatically carries trace context
    response = stub.Call(request)

# Server-side
class ServiceServicer(ServiceServicer):
    def Call(self, request, context):
        # Trace context automatically extracted
        with tracer.start_as_current_span("handle_call") as span:
            return response
            

Propagation Considerations: Ensure all services use the same propagation format (W3C Trace Context is the standard). Consistent propagation is essential for trace continuity. Use OpenTelemetry's automatic instrumentation for language-specific propagation.

Analyzing Traces with Jaeger

Jaeger provides a powerful UI for analyzing traces, identifying performance issues, and understanding service dependencies.

                # Jaeger UI Features

# 1. Service Dependencies Graph
# Shows how services are connected and traffic flow
# Identifies bottleneck services and dependencies

# 2. Trace Search
# Search by service, operation, tags, and time range
# Find specific traces for analysis
jaeger_query:
  search:
    service: order-service
    operation: createOrder
    minDuration: 1s
    maxDuration: 5s

# 3. Trace Flame Graph
# Visual representation of spans with timing
# Identify where latency is occurring
# Zoom into specific spans for details

# 4. Trace Detail
# View span details, tags, and logs
# Analyze performance per operation
# View trace context and propagation

# 5. Performance Analytics
# Span statistics per service
# Percentile latency tracking
# Error rate analysis

# Jaeger Query API
# Get traces for a service
curl "http://jaeger-query:16686/api/traces?service=order-service&limit=20"

# Get specific trace
curl "http://jaeger-query:16686/api/traces/{trace_id}"

# Get services
curl "http://jaeger-query:16686/api/services"

# Get operations
curl "http://jaeger-query:16686/api/services/{service_name}/operations
            

Tracing Analysis Best Practices:

Search for slow traces to identify latency issues
Look for error traces to find failures
Analyze service dependencies for optimization opportunities
Use trace comparisons for performance tuning
Set up automated analysis for common patterns
Create custom dashboards for key metrics

Sampling Strategies

Sampling controls how many traces are collected. It's essential for managing trace volume and storage costs.

Probabilistic Sampling

Random percentage sampling

Samples a random percentage of traces (e.g., 10%). Simple to implement but may miss rare errors. Good for high-volume environments.

High volume, general observability

Rate-Limiting Sampling

Fixed rate sampling

Limits the number of traces per second. Prevents overwhelming the tracing system. Good for environments with variable traffic.

Variable traffic

Head-Based Sampling

Decision at the start

Decision is made at the beginning of the trace. All spans in the trace are sampled based on the initial decision. Simple but may miss important traces.

Simple environments

Tail-Based Sampling

Decision at the end

Decision is made after the trace is complete. Can prioritize slow or error traces. More complex but ensures important traces are captured.

Critical environments

                # Jaeger Sampling Configuration
apiVersion: jaegertracing.io/v1
kind: Jaeger
metadata:
  name: jaeger
spec:
  sampling:
    options:
      default_strategy:
        type: probabilistic
        param: 0.1  # Sample 10%
      service_strategies:
      - service: order-service
        type: rate_limiting
        param: 5  # 5 traces per second
      - service: payment-service
        type: probabilistic
        param: 0.5  # Sample 50%

# OpenTelemetry Collector Sampling
processors:
  tail_sampling:
    decision_wait: 10s
    num_traces: 100
    expected_new_traces_per_sec: 10
    policies:
      - name: errors-policy
        type: status_code
        status_code: ERROR
      - name: slow-policy
        type: latency
        latency_threshold: 5s
      - name: random-policy
        type: probabilistic
        sampling_rate: 0.1

# Sampling in application
from opentelemetry.sdk.trace import sampling

class MySampler(sampling.Sampler):
    def should_sample(self, parent_context, trace_id, name, kind, attributes, links):
        # Sample based on custom logic
        if name == "critical_operation":
            return sampling.Decision.RECORD_AND_SAMPLE
        return sampling.Decision.RECORD_ONLY
            

Sampling Best Practices:

Start with 10-20% probabilistic sampling for production
Use tail-based sampling to capture slow and error traces
Adjust sampling rates based on trace volume and storage capacity
Use service-specific sampling strategies
Monitor trace collection volume to prevent backpressure

Tracing Solutions Comparison

Feature	Jaeger	Zipkin	OpenTelemetry
Architecture	Distributed	Centralized	Unified standard
Storage	Elasticsearch, Cassandra	MySQL, Cassandra, Elasticsearch	Various
UI	Rich, modern	Simple, clean	Varies
Sampling	Adaptive	Static	Configurable
Service Dependencies	Yes	Limited	Yes
Performance	High	Good	Good
Complexity	Medium	Low	Medium
Best For	Production, large scale	Small to medium	Standard instrumentation

Choosing a Solution: Jaeger is recommended for production due to its scalability, features, and Kubernetes integration. Zipkin is a good choice for smaller deployments. OpenTelemetry is the standard for instrumentation and can export to both Jaeger and Zipkin.

Frequently Asked Questions

What is the difference between distributed tracing and logging?

Logging records discrete events with messages and timestamps. Distributed tracing tracks the flow of requests through services, showing how each component contributes to the overall request latency. Tracing provides context about how services interact, while logs provide detail about individual events.

What is OpenTelemetry and why is it important?

OpenTelemetry is the industry standard for generating telemetry data (traces, metrics, logs). It provides a single, vendor-agnostic API and SDK for instrumentation, eliminating vendor lock-in. It's the future of observability instrumentation.

How do I propagate trace context between services?

Use W3C Trace Context headers (traceparent, tracestate) for HTTP requests. OpenTelemetry's context propagation automatically injects and extracts these headers. For asynchronous communication, use message headers or context propagation with message brokers.

What is sampling and why is it important?

Sampling controls the number of traces collected to manage storage costs and system load. Without sampling, high-volume systems can generate millions of traces per second, overwhelming storage and processing. Sampling captures a representative sample while reducing costs.

How do I instrument existing applications for tracing?

Use OpenTelemetry SDKs or auto-instrumentation agents (Java agent, Python instrumentation). For many languages, you can add the OpenTelemetry agent without code changes. For custom instrumentation, add spans for specific operations.

What is the performance impact of distributed tracing?

The impact depends on instrumentation overhead. OpenTelemetry SDKs are optimized for low overhead (typically 1-5% CPU impact). Sampling reduces the impact further. In production, the benefits of visibility typically outweigh the performance costs.

How do I store and manage trace data?

Use Elasticsearch for Jaeger or Zipkin storage. Configure index lifecycle management (ILM) for automatic rollover and deletion. Monitor storage usage and plan for capacity growth. Consider using object storage for long-term archival.

What are the alternatives to Jaeger and Zipkin?

Alternatives include: Tempo (Grafana), Honeycomb (SaaS), Lightstep (SaaS), Datadog APM (SaaS), AWS X-Ray (AWS). Choose based on your scale, budget, and existing observability stack.

Previous: Logging with EFK/ELK Next: Alerting Best Practices

Distributed tracing is essential for understanding complex microservices interactions. Start with Jaeger and OpenTelemetry, instrument your critical services, and use tracing data to optimize performance and troubleshoot issues.