More Than Throughput: Next Generation Wi-Fi Testing with Ixia's WaveDevice

It’s been quite some time since I first started writing for AnandTech, and without question there’s been a lot of changes that have happened to our testing methodologies in the past few years. One of the main issues that I’ve always been thinking about while working through reviews is how we could improve our testing methodology in a meaningful way outside of simply updating benchmarks to stay current. Internally we’ve been investigating these issues for quite some time now, and these changes have included the addition of SoC power efficiency comparisons and display power efficiency measurements. There are a lot of other changes here and there that I still want to make, but one of the major unexplored areas has been wireless radio performance.

Wireless performance testing is probably one of the hardest things that we could test, and for a time I had almost given up hope on deploying such testing within AnandTech. However life has a way of playing out differently than I expect, and in the past few months we’ve been working with Ixia to make Wi-Fi testing a reality. Ixia, for those of our readers who aren't familiar with the company, is a traditional player in the networking test space. They are perhaps best known for their Ethernet test products, and more recently have been expanding into wireless and security testing with the acquisition of companies like VeriWave and BreakingPoint Systems.

We have done Wi-Fi testing before, but in the past we were mainly focused upon a relatively simple and arguably not particularly interesting test case: maximum throughput in ideal conditions. It was obvious that Wi-Fi in many devices is still not perfect, as subjective differences in reception and reliability can feel obvious. However, without any data or methods of replication it was hard to really prove that what we felt about wireless performance was really the case.

A few months ago, Ixia brought me into their offices to evaluate their WaveDevice system, which a Wi-Fi testing device uniquely suited to solving our testing needs. This system is effectively integrates a number of tools into a single chassis, including: Wi-Fi access points, traffic generators, programmable attenuators to set path loss, channel emulators to simulate a certain kind of RF environment in terms of interference and bandwidth, packet sniffers and analyzers, and signal/spectrum analyzers. These tools are implemented such that each layer of the Wi-Fi protocol can be analyzed, from the physical link layer of raw bits encoded at the carrier frequency of 2.4 of 5 GHz, to the application level where everything is neat bitstreams transmitted or received from specified addresses.

Of course, hardware is just one half of the equation. In order to provide a full solution, Ixia also has a full suite of software to make it possible to run common tests without spending excessive amounts of time developing custom tests. While WaveDevice supports custom tests through its API, WaveDevice out of the box supports a simple throughput test, which is effectively like iPerf but with more advanced statistics. There's also a general data plane test to evaluate throughput when varying the traffic direction, traffic type, frame size, and frame rate. Other than these basic tests, WaveDevice also has tests for rate vs range, roaming latency, and traffic mix testing. In the case of rate vs range, it's possible to run an automated sequence of throughput tests while varying frame rate, transmit power, and physical link rates. Meanwhile in the interesting case of roaming latency, we can test how long it takes for a Wi-Fi device to hop from one access point to another when fading out the signal of one access point and fading in the signal of another. Finally, the traffic mix test allows for a test of throughput when faced with competing traffic that is also transmitting.

Ixia has also made sure to support a range of OSes in order to make sure that almost any device can be tested against WaveDevice. In addition to maintaining iOS and Android applications, it seems that WaveAgent is a simple C application at its core that can be easily used for embedded systems like wearables and Wi-Fi cameras, so it seems that the "device" in WaveDevice really refers to any client with a Wi-Fi chipset.

While at first glance it might seem like these tests are pretty simple, it turns out that there's a huge amount of nuance to them. To try and understand the nuance I'm talking about, it's best to start with a basic primer on how Wi-Fi works.