Kubernetes Networking

CNI Services Ingress NetworkPolicy DNS

A comprehensive guide to Kubernetes networking covering CNI plugins, Services, Ingress controllers, NetworkPolicy, DNS, service discovery, and troubleshooting with detailed explanations and practical examples.

CNI Services NetworkPolicy DNS & Discovery

Understanding Kubernetes Networking

Kubernetes networking is a complex but fundamental aspect of container orchestration. It enables communication between pods, services, and external clients. Kubernetes follows a set of core networking principles that all implementations must satisfy:

Pod-to-Pod: Every pod must be able to communicate with every other pod on any node without NAT
Pod-to-Service: Every pod must be able to communicate with every service
Service-to-External: External clients must be able to access services via Ingress or LoadBalancer
Pod-to-Node: Each pod should have a unique IP address within the cluster network

Pod → CNI → Cluster Network → Service → Ingress → External

Key Networking Principles:

Every pod gets its own IP address (no port conflicts)
Pod IPs are routable within the cluster
Services provide stable endpoints with load balancing
Network policies enable micro-segmentation

CNI: Container Network Interface

The Container Network Interface (CNI) is a standard for configuring network interfaces in Linux containers. CNI plugins implement the network connectivity between pods and provide features like IPAM, network policies, and service mesh integration.

Calico

Cilium

eBPF-based networking

Uses eBPF for high-performance networking and security. Provides deep observability, service mesh, and network policy enforcement. Supports identity-based security.

High performance, security

Flannel

Simple overlay network

One of the simplest CNI plugins. Uses VXLAN or host-gw for pod networking. Easy to set up but lacks advanced features like network policies.

Simple clusters, testing

Weave Net

DNS and encryption built-in

Provides automatic service discovery and encryption. Simple to install but may have performance overhead compared to Calico or Cilium.

Small to medium clusters

                # Install Calico
kubectl apply -f https://raw.githubusercontent.com/projectcalico/calico/v3.27/manifests/calico.yaml

# Install Cilium (with Helm)
helm repo add cilium https://helm.cilium.io/
helm install cilium cilium/cilium --namespace kube-system \
  --set ipam.mode=kubernetes \
  --set routingMode=native \
  --set hubble.enabled=true

# Install Flannel
kubectl apply -f https://github.com/flannel-io/flannel/releases/latest/download/kube-flannel.yml

# Check CNI configuration
ls -la /etc/cni/net.d/
cat /etc/cni/net.d/10-calico.conflist

# Check CNI plugin status
kubectl get pods -n kube-system | grep -E "calico|cilium|flannel|weave"

# Network interface in pod
kubectl exec -it <pod-name> -- ip addr show
kubectl exec -it <pod-name> -- ip route show
            

Choosing a CNI: Calico is the most widely used CNI for production due to its feature set and performance. Cilium is excellent for security and observability with eBPF. Flannel is simple but lacks network policies. Choose based on your specific requirements.

Services: Stable Network Endpoints

Services provide stable network endpoints for accessing pods. They abstract the dynamic nature of pod IPs and provide load balancing, service discovery, and external exposure.

ClusterIP

Internal-only access

The default service type. Exposes the service on an internal IP in the cluster. Only reachable from within the cluster. Used for internal service communication.

Internal microservices

NodePort

External access via node IP

Exposes the service on the same port on each node's IP. Accessible from outside the cluster via <node-ip>:<node-port>. Limited load balancing capabilities.

Development, simple external access

LoadBalancer

Cloud provider load balancer

Provisions a cloud load balancer (AWS ELB, GCP LB, Azure LB) to expose the service externally. Provides external IP and full load balancing.

Production external access

Headless Service

No cluster IP, direct pod DNS

clusterIP set to None. Used for stateful applications where each pod needs direct access. DNS resolves to pod IPs instead of a single service IP.

StatefulSets, databases

                # ClusterIP Service
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  type: ClusterIP
  selector:
    app: my-app
  ports:
  - port: 80
    targetPort: 8080
    protocol: TCP

# NodePort Service
apiVersion: v1
kind: Service
metadata:
  name: my-nodeport
spec:
  type: NodePort
  selector:
    app: my-app
  ports:
  - port: 80
    targetPort: 8080
    nodePort: 30080  # Optional, Kubernetes will assign if not set

# LoadBalancer Service
apiVersion: v1
kind: Service
metadata:
  name: my-lb
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: "nlb"
    service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
spec:
  type: LoadBalancer
  selector:
    app: my-app
  ports:
  - port: 80
    targetPort: 8080

# Headless Service (for StatefulSet)
apiVersion: v1
kind: Service
metadata:
  name: my-statefulset
spec:
  clusterIP: None
  selector:
    app: my-statefulset
  ports:
  - port: 3306
    targetPort: 3306

# Service with session affinity
apiVersion: v1
kind: Service
metadata:
  name: my-service
spec:
  selector:
    app: my-app
  ports:
  - port: 80
  sessionAffinity: ClientIP
  sessionAffinityConfig:
    clientIP:
      timeoutSeconds: 10800

# Check services
kubectl get svc
kubectl describe svc my-service

# Check endpoints (pods behind service)
kubectl get endpoints my-service
            

Service Considerations: LoadBalancer services incur cloud provider costs. NodePort services use ports in the 30000-32767 range. Headless services bypass service-level load balancing and should only be used when direct pod access is required.

Ingress: HTTP/HTTPS Routing

Ingress manages external access to services in a cluster, typically HTTP and HTTPS. It provides load balancing, SSL/TLS termination, and name-based virtual hosting.

NGINX Ingress

AWS ALB Ingress

AWS Application Load Balancer

Uses AWS ALB for ingress. Provides native AWS integration, path-based routing, and SSL termination. Requires AWS Load Balancer Controller.

AWS environments

Traefik

Modern ingress controller

Modern, dynamic ingress controller with support for multiple protocols. Integrates with Let's Encrypt, provides dashboard, and supports service mesh.

Dynamic environments

Istio Ingress Gateway

Service mesh ingress

Provides ingress capabilities as part of the Istio service mesh. Supports advanced traffic management, mTLS, and observability.

Service mesh environments

                # Install NGINX Ingress Controller
helm repo add ingress-nginx https://kubernetes.github.io/ingress-nginx
helm install ingress-nginx ingress-nginx/ingress-nginx --namespace ingress-nginx --create-namespace

# Basic Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: my-ingress
  annotations:
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  ingressClassName: nginx
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-service
            port:
              number: 80

# TLS/SSL Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: tls-ingress
spec:
  ingressClassName: nginx
  tls:
  - hosts:
    - app.example.com
    secretName: tls-secret
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: my-service
            port:
              number: 80

# Multi-service Ingress
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: multi-service-ingress
spec:
  ingressClassName: nginx
  rules:
  - host: api.example.com
    http:
      paths:
      - path: /api/v1
        pathType: Prefix
        backend:
          service:
            name: api-service
            port:
              number: 8080
      - path: /web
        pathType: Prefix
        backend:
          service:
            name: web-service
            port:
              number: 80

# Canary Ingress (NGINX annotations)
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
  name: canary-ingress
  annotations:
    nginx.ingress.kubernetes.io/canary: "true"
    nginx.ingress.kubernetes.io/canary-weight: "10"
spec:
  ingressClassName: nginx
  rules:
  - host: app.example.com
    http:
      paths:
      - path: /
        pathType: Prefix
        backend:
          service:
            name: canary-service
            port:
              number: 80
            

Ingress Best Practices:

Use TLS for all production ingress
Implement rate limiting at the ingress level
Use ingress class to manage multiple ingress controllers
Implement health checks for backend services
Use annotations for advanced routing (canary, A/B testing)

NetworkPolicy: Micro-Segmentation

NetworkPolicy enables fine-grained control over pod-to-pod communication. It's like a firewall for your Kubernetes cluster, allowing you to enforce zero-trust networking principles.

                # Default deny all ingress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-ingress
spec:
  podSelector: {}  # Applies to all pods
  policyTypes:
  - Ingress

# Default deny all egress
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: deny-all-egress
spec:
  podSelector: {}
  policyTypes:
  - Egress

# Allow ingress from same namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-namespace-ingress
spec:
  podSelector:
    matchLabels:
      app: my-app
  ingress:
  - from:
    - podSelector: {}  # All pods in same namespace
  policyTypes:
  - Ingress

# Allow from specific namespace
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-monitoring
spec:
  podSelector:
    matchLabels:
      app: my-app
  ingress:
  - from:
    - namespaceSelector:
        matchLabels:
          name: monitoring
  policyTypes:
  - Ingress

# Allow from specific pod with label
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-from-frontend
spec:
  podSelector:
    matchLabels:
      app: backend
  ingress:
  - from:
    - podSelector:
        matchLabels:
          app: frontend
    ports:
    - port: 8080
      protocol: TCP
  policyTypes:
  - Ingress

# Allow egress to specific external IP
apiVersion: networking.k8s.io/v1
kind: NetworkPolicy
metadata:
  name: allow-egress-external
spec:
  podSelector:
    matchLabels:
      app: my-app
  egress:
  - to:
    - ipBlock:
        cidr: 192.168.1.0/24
    ports:
    - port: 5432
      protocol: TCP
  policyTypes:
  - Egress

# Apply network policies
kubectl apply -f network-policy.yaml

# Check network policies
kubectl get networkpolicies
kubectl describe networkpolicy deny-all-ingress
            

NetworkPolicy Requirements: NetworkPolicy enforcement requires a CNI plugin that supports network policies (Calico, Cilium, Weave, etc.). Flannel does not support NetworkPolicy by default. Always test policies in a non-production environment first.

DNS and Service Discovery

Kubernetes provides built-in DNS-based service discovery through CoreDNS (or kube-dns). Services are automatically registered with DNS, enabling pods to discover services by name.

                # CoreDNS configuration
# Check CoreDNS deployment
kubectl get pods -n kube-system -l k8s-app=kube-dns
kubectl get configmap coredns -n kube-system

# Service DNS format
# <service-name>.<namespace>.svc.cluster.local
# Example: my-service.default.svc.cluster.local

# Pod DNS format (via headless service or statefulset)
# <pod-name>.<service-name>.<namespace>.svc.cluster.local
# Example: nginx-0.nginx-svc.default.svc.cluster.local

# Test DNS resolution from a pod
kubectl run test-pod --image=busybox -it --rm --restart=Never -- nslookup kubernetes.default.svc.cluster.local

# Check DNS configuration inside a pod
kubectl exec -it <pod-name> -- cat /etc/resolv.conf

# CoreDNS configuration
apiVersion: v1
kind: ConfigMap
metadata:
  name: coredns
  namespace: kube-system
data:
  Corefile: |
    .:53 {
        errors
        health {
            lameduck 5s
        }
        ready
        kubernetes cluster.local in-addr.arpa ip6.arpa {
            pods insecure
            fallthrough in-addr.arpa ip6.arpa
            ttl 30
        }
        prometheus :9153
        forward . /etc/resolv.conf {
            max_concurrent 1000
        }
        cache 30
        loop
        reload
        loadbalance
    }

# Custom DNS entries (hostAliases)
apiVersion: v1
kind: Pod
metadata:
  name: custom-dns-pod
spec:
  hostAliases:
  - ip: "192.168.1.100"
    hostnames:
    - "internal.example.com"
    - "db.internal"
  containers:
  - name: app
    image: nginx
            

DNS Best Practices:

Use fully qualified domain names for service discovery
Monitor CoreDNS performance and scale accordingly
Use headless services for stateful applications requiring direct pod DNS
Consider using ExternalDNS for automated DNS record management

Network Troubleshooting

Troubleshooting Kubernetes networking issues requires a systematic approach. Here are common techniques and tools.

                # Check network connectivity between pods
kubectl run busybox --image=busybox -it --rm --restart=Never -- wget -O- http://<pod-ip>:<port>

# Check service resolution
kubectl run busybox --image=busybox -it --rm --restart=Never -- nslookup <service-name>.<namespace>.svc.cluster.local

# Check pod network interface
kubectl exec -it <pod-name> -- ip addr show

# Check network policies
kubectl get networkpolicies --all-namespaces
kubectl describe networkpolicy <policy-name> -n <namespace>

# Check CNI logs
kubectl logs -n kube-system -l k8s-app=calico-node
kubectl logs -n kube-system -l k8s-app=cilium

# Check kube-proxy
kubectl logs -n kube-system -l k8s-app=kube-proxy

# Debug with ephemeral container
kubectl debug -it <pod-name> --image=busybox --target=<container-name>

# Check network connectivity with netcat
kubectl run test-pod --image=busybox -it --rm --restart=Never -- sh -c "nc -zv <host> <port>"

# Check IPVS rules (if using IPVS mode)
kubectl exec -n kube-system <kube-proxy-pod> -- ipvsadm -L -n

# Check iptables rules (if using iptables mode)
kubectl exec -n kube-system <kube-proxy-pod> -- iptables -t nat -L -n
            

Troubleshooting Checklist:

Check if CNI pods are running (kubectl get pods -n kube-system)
Verify service endpoints (kubectl get endpoints)
Test DNS resolution from within pods
Check network policy enforcement
Review kube-proxy logs
Check node network configuration

Frequently Asked Questions

What is the difference between a Service and an Ingress?

A Service provides stable networking and load balancing for pods within the cluster. An Ingress provides HTTP/HTTPS routing from outside the cluster to Services. Ingress is built on top of Services and adds features like SSL termination, path-based routing, and virtual hosting.

Which CNI should I use for production?

Calico is the most widely used and recommended for production due to its rich feature set, performance, and network policy support. Cilium is excellent for eBPF-based networking and security. Choose based on your specific requirements (performance, security, simplicity).

How do I expose my Kubernetes application externally?

Use Ingress for HTTP/HTTPS routing (recommended for web applications) or a LoadBalancer Service for cloud environments. For development, you can use NodePort or port-forwarding. Ingress provides the most flexibility and features.

What is a headless service and when should I use it?

A headless service has clusterIP set to None. It does not provide load balancing; instead, DNS resolves to individual pod IPs. Use headless services for StatefulSets, databases, and any case where direct pod discovery is required.

How does CoreDNS work in Kubernetes?

CoreDNS is the default DNS server in Kubernetes. It watches the Kubernetes API for Services and Endpoints, and automatically updates DNS records. Service names resolve to cluster IPs, and pods can discover services by name.

What are NetworkPolicies used for?

NetworkPolicies control traffic flow between pods and external endpoints. They enforce micro-segmentation, enabling zero-trust networking. You can define which pods can talk to which other pods based on labels and namespaces.

How do I debug pod-to-pod connectivity issues?

Check CNI pod status, test connectivity with tools like ping/wget from a debug pod, verify network policies, check service endpoints, and inspect kube-proxy logs. Use ephemeral containers for advanced debugging.

What's the difference between ClusterIP, NodePort, and LoadBalancer?

ClusterIP is internal-only. NodePort exposes the service on each node's IP at a high port. LoadBalancer provisions a cloud load balancer to expose the service externally. LoadBalancer provides the most external exposure and is the recommended production approach for external access.

Previous: Helm Charts Next: Ingress Controllers

Kubernetes networking is the backbone of container orchestration. Master these concepts to build resilient, secure, and scalable applications on Kubernetes.