Mastering Kubernetes Networking with Cilium Egress Gateway

In the complex world of Kubernetes networking, controlling outbound traffic is as crucial as managing inbound requests. Ensuring secure, predictable, and compliant egress traffic from your pods is essential for modern applications. This is where Kubernetes Egress Gateways come into play, and Cilium offers a powerful, eBPF-native implementation. In this post, we'll dive deep into Cilium's Egress Gateway, exploring how it enhances network policies, integrates with service meshes, and provides granular control over your cluster's outbound connections.

Understanding Kubernetes Egress Gateways

Traditionally, pods in a Kubernetes cluster egress traffic using the IP address of the node they are running on. This can be problematic for several reasons:

• IP Address Volatility: Node IPs can change due to scaling events, reboots, or network changes, making it difficult to maintain consistent firewall rules or access controls on external services.
• Lack of Granular Control: It's challenging to enforce specific egress policies for different sets of pods when they all share the same node IP.
• Security Concerns: Exposing individual node IPs can create a larger attack surface.

An Egress Gateway provides a solution by allowing you to route outbound traffic from specific pods through a designated node (or set of nodes) with a fixed, predictable IP address. This acts as a central point for controlling and auditing all egress traffic, simplifying security and compliance.

Cilium's Egress Gateway: A Deep Dive

Cilium leverages eBPF and the power of the Linux kernel to provide a high-performance, feature-rich Egress Gateway implementation. It allows you to define granular policies that dictate which traffic should egress through a specific gateway node.

Enabling the Egress Gateway

To enable Cilium's Egress Gateway feature, you typically need to configure it during Cilium installation or upgrade. This can be done via Helm or the Cilium CLI:

Using Helm:

helm upgrade cilium <CHART_RELEASE> \
   --namespace kube-system \
   --reuse-values \
   --set egressGateway.enabled=true \
   --set bpf.masquerade=true \
   --set kubeProxyReplacement=true

Using Cilium CLI:

cilium install \
    --set egressGateway.enabled=true \
    --set bpf.masquerade=true \
    --set kubeProxyReplacement=true

After enabling the feature, you'll need to restart the Cilium agents and operator for the changes to take effect:

kubectl rollout restart ds cilium -n kube-system
kubectl rollout restart deploy cilium-operator -n kube-system

Designating an Egress Gateway Node

Once the feature is enabled, you need to label a specific Kubernetes node to designate it as an Egress Gateway node:

kubectl label nodes <egress-gateway-node> egress-node=true

Configuring Egress Policies with CiliumEgressGatewayPolicy

The core of Cilium's Egress Gateway functionality lies in the CiliumEgressGatewayPolicy custom resource definition (CRD). This resource allows you to specify:

• selectors: Which pods this policy applies to (using podSelector and optional nodeSelector).
• destinationCIDRs: The destination IP address ranges the policy applies to.
• egressGateway: Which node should act as the gateway and how its IP should be determined (either via egressIP or interface).

Example CiliumEgressGatewayPolicy:

apiVersion: cilium.io/v2
kind: CiliumEgressGatewayPolicy
metadata:
  name: egress-sample
spec:
  # Specify which pods should be subject to the current policy.
  selectors:
  - podSelector:
      matchLabels:
        org: empire
        class: mediabot
    nodeSelector: # optional, if not specified the policy applies to all nodes
      matchLabels:
        node.kubernetes.io/name: node1 # only traffic from this node will be SNATed
  # Specify which destination CIDR(s) this policy applies to.
  destinationCIDRs:
  - "0.0.0.0/0"
  - "::/0"

  # Configure the gateway node.
  egressGateway:
    # Specify which node should act as gateway for this policy.
    nodeSelector:
      matchLabels:
        node.kubernetes.io/name: node2

    # Specify the IP address used to SNAT traffic matched by the policy.
    # It must exist as an IP associated with a network interface on the instance.
    egressIP: 10.168.60.100

    # Alternatively it's possible to specify the interface to be used for egress traffic.
    # interface: eth0

This policy routes all outbound traffic (0.0.0.0/0) from pods labeled org: empire and class: mediabot to egress through the node labeled node.kubernetes.io/name: node2, using 10.168.60.100 as the source IP.

You can also specify multiple CIDRs, use excludedCIDRs, or configure the egress IP by explicitly setting an interface:

# Using excludedCIDRs
destinationCIDRs:
- "0.0.0.0/0"
excludedCIDRs:
- "10.0.0.0/8"

# Using interface
egressGateway:
  nodeSelector:
    matchLabels:
      testLabel: testVal
  interface: ethX

Integrating with Network Policies

Cilium's Egress Gateway works seamlessly with its robust Network Policy capabilities. You can create CiliumNetworkPolicy resources to control which destinations pods are allowed to communicate with, and then use CiliumEgressGatewayPolicy to ensure that allowed traffic uses a specific egress IP.

For instance, you might have a policy that only allows pods to connect to api.github.com:

apiVersion: "cilium.io/v2"
kind: CiliumNetworkPolicy
metadata:
  name: "mediabot-egress-github-api"
spec:
  endpointSelector:
    matchLabels:
      class: mediabot
      org: empire
  egress:
  - toFQDNs:
    - matchName: "api.github.com"
  # ... potentially other egress rules

By combining this with a CiliumEgressGatewayPolicy targeting the same pods, you ensure that this allowed egress traffic originates from a consistent IP address.

Service Mesh Integration

When integrating with service meshes like Istio or using Kubernetes' own Gateway API, Cilium's Egress Gateway plays a vital role in managing outbound traffic from your mesh-enabled services.

• Control Egress from Service Mesh Sidecars: You can direct all outbound traffic from pods running service mesh sidecars through the Egress Gateway, ensuring that external communication adheres to defined policies and uses a consistent IP.
• Gateway API: Cilium's implementation of the Gateway API can leverage the Egress Gateway for outbound traffic originating from gateway controllers or services exposed via the Gateway API. This provides a unified approach to managing both inbound and outbound traffic control.

Cilium's Gateway API implementation allows you to define listeners and routes. If services exposed through the Gateway API need to communicate with external services, the Egress Gateway can manage that traffic flow. For example, you can configure a Gateway resource and then ensure that any egress traffic from the pods backing that gateway uses a specific egress IP.

Verifying and Troubleshooting

Cilium provides tools for verifying and troubleshooting your Egress Gateway configuration:

• cilium-dbg bpf egress list: This command inspects the eBPF configuration for egress policies, showing source IPs, destination CIDRs, egress IPs, and gateway IPs.

  kubectl -n kube-system exec ds/cilium -- cilium-dbg bpf egress list
  

• Nginx Access Logs: By running a simple web server like Nginx on your gateway node or a test pod, you can inspect its access logs to verify the source IP of incoming requests from your application pods.

  # Install and start Nginx
  sudo apt install nginx
  sudo systemctl start nginx
  
  # Monitor logs
  tail /var/log/nginx/access.log
  

• Hubble: Cilium's observability tool, Hubble, can be used to visualize network flows and confirm that traffic is being routed as expected or identify any dropped packets due to policy misconfigurations.

Conclusion

Cilium's Egress Gateway provides a robust and efficient solution for managing outbound traffic in Kubernetes. By offering granular control through CiliumEgressGatewayPolicy, seamless integration with Network Policies, and compatibility with service mesh patterns, it empowers developers and operators to enhance security, compliance, and network predictability. Mastering Cilium's Egress Gateway is a key step towards building more secure and reliable cloud-native applications.

Resources

• Cilium Egress Gateway Documentation: https://docs.cilium.io/en/stable/network/egress-gateway/
• Cilium Network Policies: https://docs.cilium.io/en/stable/policy/overview/
• Cilium Gateway API: https://docs.cilium.io/en/stable/network/servicemesh/gateway-api/