Figure 3: Wireless test setup
All routers / APs are generally reset to factory defaults and configured to act as access point, so the test results are in no way effected by the router portion of the products.
At each test location, the IxChariot Throughput.scr script (which is an adaptation of the Filesndl.scr long file send script) is run for 1 minute in real-time mode using TCP/IP in both uplink (client to AP) and downlink (AP to client) directions. The only modification made to the IxChariot script is typically to change the file size from its default of 100,000 Bytes to 300,000 Bytes of data for each file send for 802.11g products.
Although manufacturers continue to push throughput as the way they'd like you to compare products, the increasing use of wireless for time-sensitive applications like gaming, VoIP and audio and video streaming now means low throughput variation is becoming just as important, especially for streaming video applications. Fortunately, IxChariot provides a metric called Relative Precision that can be used to compare throughput variation. Here is an excerpt from Ixia's Relative Precision description:
The Relative Precision is a gauge of how reliable the results are for a particular endpoint pair. Regardless of what type of script was run, you can compare relative precision values. The relative precision is obtained by calculating the 95% confidence interval of the Measured Time for each timing record, and dividing it by the average Measured Time. This number is then converted to a percentage by multiplying it by 100. The lower the Relative Precision value, the more reliable the result.
Here is a formal definition of a confidence interval, using statistical terms:
A confidence interval is an estimated range of values with a given high probability of covering the true population value. To state the definition another way, there is a 95% chance that the actual average lies between the lower and upper bound indicated by the 95% Confidence Interval.
While QoS techniques such as WMM (Wi-Fi Multimedia) and the recently-ratified 802.11e IEEE standard are intended to compensate for wide swings in wireless throughput, they can only do so much if adequate, steady bandwidth isn't available. And if the time it took for manufacturers to incorporate WPA and WPA2 improved wireless security is any indication, it could be years before WMM and 802.11e are supported by enough vendors to be useful.
The new Wireless Quality Score (WQS) is intended to do essentially the same thing as showing IxChariot wireless throughput vs. time plots, but in a more quantifiable and easier-to-compare way. For each test location, the WQS is calculated by adding together the Average Throughput divided by the Relative Precision for each test location or:
WQS = (Loc1 AT/Loc1 RP)+(Loc2 AT/Loc2 RP)+(Loc3 AT/Loc3 RP)+(Loc3 AT/Loc3 RP)+(Loc5 AT/Loc5 RP)
where AT = Average Throughput and RP = Relative Precision. This simple score incorporates both throughput and variation in a simple way that moves the WQS up for products with higher throughput and lower variation and down for those with lower throughput and higher variation.
But since there are those for whom (high) speed is the only thing that matters, I also calculate total throughput of all locations, so that you can compare products on that basis if you prefer.