Beyond Iptables: Solving the Observability Gap in eBPF-based Kubernetes Networking

For years, the standard way to handle networking in Kubernetes was a settled matter. We relied on kube-proxy to manage the flow of traffic between pods and services, and kube-proxy, in turn, relied on iptables. It was a reliable, well-understood architecture that served the community through the early growth of container orchestration. However, as clusters scaled from dozens of nodes to thousands, and from hundreds of services to tens of thousands, the limitations of this legacy stack began to surface.

The industry is currently in the midst of a massive architectural shift toward eBPF (Extended Berkeley Packet Filter) as the primary engine for Kubernetes networking. By bypassing the traditional Linux networking stack, eBPF offers dramatic performance improvements and lower CPU overhead. But this performance gain comes with a significant, often overlooked cost: the “observability gap.” When you remove iptables and netfilter from the data path, you also bypass the very hooks that traditional monitoring, security, and troubleshooting tools have relied on for decades. Transitioning to an eBPF-based CNI (Container Network Interface) like Cilium isn’t just a networking upgrade; it’s a fundamental change in how we must observe and debug our systems.

The Legacy: How Kube-Proxy and Iptables Work

To understand why the shift to eBPF is so disruptive, we first need to look at the “traditional” way Kubernetes handles traffic. In a standard cluster, kube-proxy runs on every node. Its job is to watch the Kubernetes API server for changes to Service and Endpoint objects and then translate those objects into networking rules that the Linux kernel can understand.

For most of Kubernetes’ history, those rules were implemented using iptables, a utility for configuring the IP packet filter rules of the Linux kernel’s netfilter framework.

The Role of Netfilter and Conntrack

When a packet arrives at a node or is sent from a pod, it enters the netfilter stack. iptables uses a series of “chains” (PREROUTING, INPUT, FORWARD, OUTPUT, POSTROUTING) to decide what to do with that packet. kube-proxy populates these chains with thousands of rules to handle LoadBalancing (Service IPs) and Network Address Translation (NAT).

Crucially, iptables is stateful. It relies on a kernel subsystem called nf_conntrack (connection tracking). Conntrack keeps a table of all active connections, allowing the kernel to remember that a specific return packet belongs to a specific outgoing request. This table is the “source of truth” for almost every legacy monitoring tool. If you’ve ever used tcpdump, conntrack-tools, or looked at /proc/net/nf_conntrack, you were interacting with this legacy stack.

The O(n) Complexity Problem

The fundamental flaw of iptables in a Kubernetes environment is its algorithmic complexity. iptables rules are evaluated sequentially. If you have 5,000 services in your cluster, each packet might have to be checked against thousands of rules before it finds a match. This is an $O(n)$ operation. As the number of services grows, the time it takes to process a single packet increases, and the CPU overhead required to manage those rules explodes. Furthermore, updating a single rule often requires the kernel to replace the entire rule set, leading to “stop-the-world” latencies in the networking stack during large deployments.

The Shift: eBPF-based Kube-Proxy Replacement

eBPF changes the game by allowing us to run sandboxed programs directly inside the Linux kernel in response to specific events, such as a packet arriving at a network interface. Instead of a static list of iptables rules, we can write programmable logic that executes at near-hardware speeds.

How Cilium Replaces Kube-Proxy

CNIs like Cilium can operate in a mode called kube-proxy-replacement. In this mode, kube-proxy is disabled entirely. Instead, Cilium attaches eBPF programs to the networking hooks in the kernel, such as XDP (eXpress Data Path) and TC (Traffic Control).

When a packet hits the network interface, the eBPF program triggers. Instead of traversing a long list of iptables rules, the program performs a direct lookup in a BPF Map. BPF Maps are efficient hash tables shared between the kernel and userspace.

Technical Note: Because BPF Maps use hash table lookups, the complexity of finding a service’s backend is $O(1)$. Whether you have 10 services or 10,000, the lookup time remains virtually constant.

Bypassing the Netfilter Chain

By handling the packet at the XDP or TC layer, eBPF can route the packet and perform NAT before it ever reaches the netfilter subsystem. This is the “fast path.” The packet is processed, redirected to its destination pod, and the kernel’s traditional networking stack is bypassed entirely. This results in significantly lower latency and frees up CPU cycles that would otherwise be spent on rule evaluation.

The Blind Spot: Why Your Monitoring Just Broke

The performance benefits of eBPF are undeniable, but they create an immediate crisis for SREs and DevOps engineers: The traditional observability tools stop working.

The Disappearance of Conntrack

Because eBPF-based networking (like Cilium’s implementation) bypasses netfilter, packets often do not generate entries in the nf_conntrack table. If your monitoring stack includes a Prometheus exporter that scrapes /proc/net/nf_conntrack to track connection counts or identify “top talkers,” those metrics will suddenly drop to zero or show only a fraction of the actual traffic.

The Failure of Iptables-save and Audit Logs

In a legacy environment, if you wanted to see how traffic was being routed, you could run iptables-save. You could see the NAT rules and the specific chains. In an eBPF-native cluster, iptables-save will return a nearly empty list. The logic is no longer in the rules; it’s in the compiled eBPF bytecode.

Similarly, many security auditing tools rely on LOG targets in iptables to record dropped packets. When the eBPF program drops a packet (due to a NetworkPolicy violation, for example), it happens silently as far as netfilter is concerned. There is no log entry in dmesg or /var/log/messages unless the eBPF program is specifically written to emit an event.

The “Black Box” Problem

This creates a “black box” effect. You have a high-performance network, but you have no visibility into:

1. Which pods are talking to which services.
2. Why a specific packet was dropped.
3. The latency of individual service-to-service hops.
4. Whether your NetworkPolicies are actually being enforced as intended.

Deep Dive: Comparing Architecture and Efficiency

The move from iptables to eBPF is not just a change in tooling; it’s an evolution in algorithmic efficiency. This shift mirrors broader trends in the technology industry where software optimization is being used to overcome physical or architectural bottlenecks.

In the realm of high-scale computing, the difference between $O(n)$ and $O(1)$ is the difference between a system that scales linearly with cost and one that remains sustainable. We see similar discussions in other high-performance domains, such as how algorithmic efficiency is becoming a geopolitical differentiator in the race for compute supremacy. In Kubernetes, eBPF is the “efficiency engine” that allows clusters to handle massive throughput without requiring a proportional increase in CPU cores.

Latency Profiles

In a cluster with 5,000 services, the latency added by iptables rule traversal can reach several milliseconds per packet. eBPF reduces this to microseconds. For microservices architectures where a single end-user request might trigger dozens of internal RPC calls, these millisecond savings compound, leading to a drastically improved user experience.

Bridging the Gap: eBPF-Native Observability

To solve the observability gap, we must adopt tools that are designed to speak “eBPF.” We can no longer rely on the kernel to provide metrics via legacy files; we need tools that can hook into the same BPF maps and events used by the networking layer.

Hubble: The Window into Cilium

Hubble is the observability layer built on top of Cilium. Because it sits directly on the eBPF path, it can see every packet, every connection, and every drop event without the overhead of conntrack.

Hubble provides:

• Flow Logs: Detailed records of every connection, including source/destination pods, labels, and namespaces.
• Service Maps: Visual representations of how services are interacting.
• Metrics: Golden signals (Latency, Error rates, Throughput) derived directly from the kernel.

Tetragon: Security and Runtime Enforcement

While Hubble focuses on the network, Tetragon uses eBPF to provide deep visibility into the runtime. It can track process execution, file access, and network activity at the system call level. If an eBPF program drops a packet due to a security policy, Tetragon can provide the context: which process tried to send the packet and which specific policy blocked it.

Pixie: Auto-Telemetry

Pixie is another eBPF-native tool that focuses on “no-instrumentation” observability. It uses eBPF to automatically collect traces and metrics from your applications without requiring you to change your code or add sidecars. This is particularly useful in an eBPF-networked cluster because Pixie can correlate application-level performance with the underlying network events captured by eBPF.

Implementation Guide: Migrating to Cilium with Full Visibility

If you are planning to migrate your cluster to an eBPF-native stack, follow this high-level guide to ensure you don’t lose visibility during the transition.

Step 1: Prepare the Environment

Ensure your nodes are running a compatible Linux kernel. While basic eBPF support started in 4.4, you really need Linux Kernel 4.19 or higher (ideally 5.10+) for full kube-proxy replacement features.

Step 2: Install Cilium in Strict Mode

When installing Cilium via Helm, you must explicitly enable the kube-proxy replacement. This tells Cilium to take over the Service handling and NAT logic.

helm install cilium cilium/cilium --version 1.14.0 \
  --namespace kube-system \
  --set kubeProxyReplacement=strict \
  --set k8sServiceHost=REPLACE_WITH_API_SERVER_IP \
  --set k8sServicePort=6443 \
  --set hubble.enabled=true \
  --set hubble.relay.enabled=true \
  --set hubble.ui.enabled=true

Step 3: Verifying the BPF State

Once installed, you can no longer use iptables to check your services. Instead, use the Cilium debug tool (cilium-dbg) to inspect the BPF maps.

# Get the Cilium pod name on a specific node
export CILIUM_POD=$(kubectl get pods -n kube-system -l k8s-app=cilium -o jsonpath='{.items[0].metadata.name}')

# List all Service LoadBalancer entries in the BPF map
kubectl exec -n kube-system $CILIUM_POD -- cilium-dbg bpf lb list

This command will show you the mapping of Virtual IPs (Services) to Backend IPs (Pods) directly from the kernel’s memory.

Step 4: Enabling Hubble CLI

To get real-time flow visibility, install the Hubble CLI and observe the traffic:

# Observe flows in a specific namespace
hubble observe --namespace production --follow

Best Practices for the Post-Iptables World

Transitioning to eBPF requires a shift in how SREs maintain cluster health. Here are three critical best practices:

1. Monitor BPF Map Capacity

BPF maps have fixed sizes. If a map (such as the connection tracking map or the load balancer map) fills up, the kernel will start dropping packets.

• Action: Monitor the cilium_bpf_map_pressure metric in Prometheus.
• Tuning: If you have a very large cluster, you may need to increase map sizes using the bpf-map-dynamic-size-ratio flag in your Cilium configuration.

2. Update Your Alerting Logic

If you have alerts based on conntrack table utilization, they are likely useless in an eBPF-native cluster.

• Action: Rewrite your alerts to use Hubble-native metrics. For example, instead of alerting on node_nf_conntrack_entries, alert on hubble_drop_count or Cilium’s internal drop metrics.

3. Kernel Homogeneity

eBPF behavior can vary slightly between kernel versions.

• Action: Ensure all nodes in your cluster are running the same kernel version. This prevents “heisenbugs” where networking behaves differently on a subset of nodes because of how the eBPF bytecode is JIT-compiled (Just-In-Time) by the kernel.

Future Outlook: eBPF as the New Standard

The “observability gap” is a temporary hurdle in what is otherwise a massive leap forward for infrastructure engineering. We are already seeing major cloud providers embrace this shift. Google Kubernetes Engine (GKE) now uses Dataplane V2 (based on Cilium/eBPF) by default. Amazon EKS and Azure AKS have followed suit with integrated eBPF networking options.

As the ecosystem matures, the distinction between networking, security, and observability will continue to blur. In an eBPF-first world, these are no longer three separate layers of the stack; they are a single, programmable engine running in the heart of the kernel.

The move away from iptables is more than just a performance optimization. It represents a transition toward “Intelligent Infrastructure”—where the network itself is aware of the applications it carries, capable of making sub-millisecond routing decisions, and providing granular visibility without the heavy tax of legacy connection tracking. For the modern SRE, the challenge is no longer about managing rules, but about mastering the maps and programs that define the future of the cloud-native stack.