Loading post…
Red HatJames Force, Raffaele SpazzoliTutorial
Egress network quality of service on Red Hat OpenShift
Summary
Network bandwidth is a compressible resource, so setting a flat cap wastes capacity when the link is idle. The real goal is to prioritise production traffic while guaranteeing background workloads a minimum floor, allowing either class to burst to full capacity when the other is idle. The upcoming NetworkQoS CRD in OVN-Kubernetes will bring declarative traffic management, but until then administrators can use Linux traffic control (tc). A hierarchical token bucket (HTB) qdisc tree on the node’s physical interface, with filters matching background traffic by destination IP, provides the desired dynamic prioritisation. In a lab test with 25 Gbps bonded NICs, production traffic maintained 19.5 Gbps under contention while background traffic was throttled to 2.7 Gbps, just above its guaranteed floor.
Context
Kubernetes lacks built-in mechanisms to define traffic priority, and setting a flat bandwidth cap wastes capacity because network bandwidth is compressible. The real goal is to prioritise critical production traffic while giving background workloads a guaranteed minimum so they never starve, allowing them to consume spare bandwidth when production is idle. The upcoming NetworkQoS CRD in OVN-Kubernetes will bring declarative traffic management, but until then administrators can use Linux traffic control.
Approach / What changed
Deploy a hierarchical token bucket (HTB) qdisc tree on the node’s physical interface using the Linux tc command, with filters that match background traffic by destination IP and route it into a deprioritised class that guarantees a 10% minimum floor while allowing bursting to full link capacity when idle.
Takeaways
Network bandwidth is compressible, so a flat cap wastes capacity; a dynamic hierarchy that guarantees a minimum floor and allows bursting is the correct approach.
The upcoming NetworkQoS CRD (OKEP-4380) in OVN-Kubernetes will provide declarative, fine-grained QoS controls for pod egress traffic, including DSCP marking and metadata-based classification.
A hierarchical token bucket (HTB) qdisc tree on the node’s physical interface, with filters matching background traffic by destination IP, can deliver meaningful traffic prioritisation today using the Linux tc command.
FAQ
Why is a flat bandwidth cap the wrong approach for network QoS?
A flat bandwidth cap penalises background tasks even when the network has available bandwidth, wasting capacity. Because network bandwidth is compressible, a hard limit is almost always the wrong approach; a dynamic hierarchy that guarantees a minimum floor and allows bursting is preferred.
What does the upcoming NetworkQoS CRD in OVN-Kubernetes provide?
The upcoming NetworkQoS CRD (OKEP-4380) will provide fine-grained QoS controls for pod egress traffic, including DSCP marking, bandwidth policing, and metadata-based traffic classification using pod labels and destination CIDRs, expressed declaratively under k8s.ovn.org/v1alpha1.
How can the tc-based workaround be made to survive reboots?
The tc commands applied manually do not survive reboots or interface resets. To operationalise the workaround, you would wrap the tc rules in a systemd service and deploy them to worker nodes declaratively using a MachineConfig object.
Source and freshness
- Source
- developers.redhat.com
- Published
- Added to Yomu
Summaries are generated from the original by a language model. Check the original before relying on it.
Keep reading
Related reading
The Good Stuff
Paid picks we use ourselves. Links carry a referral code — a subscription through one earns us a commission at no extra cost to you.
The best engineering reads, once a week
One useful email. Unsubscribe anytime.