Software Security
Novel Fuzzer Uncovers Hidden Consistency Bugs in Clustered Network Controllers
Researchers develop a concurrency fuzzer that detects state divergence between distributed SDN clusters and their data planes, finding bugs that traditional testing methods miss.
A team of researchers has introduced GapFuzz, a specialized testing tool designed to uncover consistency failures in distributed Software-Defined Networking (SDN) controller clusters. The tool targets a vulnerability that existing test suites overlook: the window of vulnerability when flow state replication happens asynchronously across cluster nodes.
The Problem: Consistency Windows in Distributed SDN#
In distributed SDN architectures, multiple controller nodes must stay in sync as they manage network traffic rules. However, the asynchronous replication process between primary and backup nodes creates brief periods where two backup controllers can independently apply contradictory rules. The master node may then serialize both rules into the actual data plane, causing the network to execute an action that none of the controllers consider authoritative.
Existing fuzzing tools miss this class of faults because they either:
- Only examine the control plane
- Test single controllers in isolation
- Lack mechanisms to deliberately trigger replication races through timed concurrency
How GapFuzz Works#
GapFuzz operates by deliberately injecting pairs of conflicting requests into non-master nodes with carefully controlled timing delays. The tool then reconstructs the complete system state by:
- Querying every replica node
- Checking the actual kernel datapath action using
ovs-appctl ofproto/trace - A two-phase timing search that first detects whether divergence occurs, then narrows the vulnerability window through doubling and binary search on injection timing
The tool also classifies each bug as either transient or persistent and categorizes it based on the four possible types of cross-plane state inconsistency derived from the ONOS 2.7 controller source code.
Results from ONOS Testing#
When tested on a three-node ONOS 2.7 cluster, GapFuzz successfully triggered divergent states in roughly 82% of test runs. Every detected divergence fell between the cluster's stated authoritative state and what the kernel datapath actually executed.
The testing revealed striking variations in timing:
- One vulnerability window was as narrow as 5 milliseconds