Our test procedure for 802.11a wireless products.
Updated August 2003
We use Ixia's free Qcheck utility and Chariot network peformance evaluation program to put wireless network gear through their paces. We use the simple test setup pictured below to run our tests under four different conditions.
Test Conditions
Testing is done in recently constructed wood frame two floor residence. Home has heating/cooling metal ductwork throughout. The Access Point (AP) or wireless router is placed on a table away from metal cabinets and RF sources in a corner lower level room for all tests. A set of measurements is then taken at each of the four locations.
- Condition #1: AP and wireless client in same room, approximately 10 feet apart.
- Condition #2: Client in upper floor room directly above AP, approximately 15 feet apart. No metal ductwork between AP and client.
- Condition #3: Client on same floor as AP but in another room approximately 50 feet away. Walls, but no metal ductwork between the client and AP antennas.
- Condition #4: Client on upper floor at opposite end of residence, approximately 55 feet away. Walls, floor, and probable metal ductwork between AP and client.
Tests Run
NOTE: Unless otherwise noted, tests are run from the Qcheck console in the laptop, with data flow from the laptop to the Ethernet LAN client, which is running the Qcheck endpoint.
1) Transfer Rate - More commonly known as Throughput, this test is a measure of how fast data moves between a wired client, Access Point or router, and wireless client. The test sends a file from client to client, measures how much time it takes, and calculates the result in Mbps (Megabits per second).
802.11a and 11g products typically have 4X (and higher) actual throughput than 802.11b products. Our initial experience with these products also indicates that there can be significant "pauses" in network activity (perhaps due to auto-ranging algorithms or other "housekeeping" chores). Both of these factors make Qcheck's throughput test of little value in accurately measuring 802.11a and 11g product throughput, since Qcheck caps the amount of data it uses for Transfer Rate testing at 1MegaByte. This small amount of data causes results to vary significantly (500% or more).
We instead use Chariot to run a 1 minute test. The test sends 1,000,000 Bytes of data, two times in a row for each timing record, and loops until one minute is up. We run this test once with data sent from wireless client to LAN-based test partner, then again with the data direction reversed, i.e. LAN partner to wireless client. We then take calculate the average of the two Average Throughput numbers and use that as our test result. This method takes into account differences in send / receive performance that can appear in higher speed networks.
We instead use Chariot to run longer transfer rate/throughput tests. We use the Throughput.scr script (which is an adaptation of the Filesndl.scr long file send script), but modify it to run two file transfers per timing record, instead of the default value of one per record. We set the transferred file size to 1Million Bytes, run the script for one minute in batch mode, and don't poll the endpoint. We run this test once with data sent from wireless client to LAN-based test partner, then again with the data direction reversed, i.e. LAN partner to wireless client.
See this NeedToKnow for information on what can limit performance of High Speed Network adapters.
We then use the Chariot Comparison program to generate comparison plots and statistics for the two tests. We calculate the average of the two Average Throughput numbers produced by the Chariot Comparison program as our test result. This method takes into account differences in send / receive performance that can appear in higher speed networks.
We usually publish selected Chariot Comparison plots that we think are indicative of the product's performance characteristics.
Of course, higher throughput numbers are better. We also run this test with 128 bit WEP enabled for Condition #1 to see if there is any throughput degradation.
2) Response Time - This test measures the delay (also known as lag, or latency) in the test setup, and is essentially what you'd measure by using the ping command. This test sends a small packet of data from the wireless client to the wired client and measures the time it takes to receive a reply. Our test uses Qcheck and runs the test 10 times in a row and calculates the average and maximum times. Lower numbers are better, especially for gaming and any voice or video applications, but anything under 10ms (milliseconds) is fine.
3) UDP Stream - This test measures how well a continuous stream of data is handled. In addition to giving an indication of whether you'll have trouble listening to Internet audio or watching video program streams, it tends to show whether there is a high bit error rate, or poor speed autoranging algorithm.. It uses the connectionless UDP protocol, which has less overhead and error recovery mechanisms than the TCP protocol (picture a fire hose being turned on vs. a water bucket brigade). Our test uses Qcheck and pushes data at a 1000 kbps (1Mbps) rate for 10 seconds and results in two numbers. You want the Actual Throughput number to be as close to 1000 kbps as possible and the Lost Data to be ideally zero. We run this test in two directions, client-to-LAN partner and LAN partner-to-client and average the results to get the reported numbers.
4) SNR (optional) - We prefer to record the Signal to Noise reading in dB, if it's provided by the client adapter's utility program. If this is not available, we record whatever reading that the client adapter's utility program provides that we feel is the most representitive of the signal quality of network connection. The value used is described in the test results notes.
Equipment Used
We tend to use the same computers to run the tests, with the Ethernet client running Win98SE or WinXP, and having 733MHz or better processor speeds and memory configurations in excess of 256MB. The test laptop, however, has a 1GHz Celeron, with 128MB of memory, running WinXP Home. The test machines have no other applications running during testing.