October 10, 2003
Worried about scalability of your network or certain Wi-Fi products in particular, but don't really want to setup a ton of clients to test it? The virtual stations generated by this product will make any access point think it instantly has 64 individual users.
Model tested: a/b/g
Price: $6500 (tri-mode)
Pros: Simulate up to 64 802.11a, b, or g clients
Cons: Expensive; doesn't currently support mixed client scenarios
In many cases, a radio frequency (RF) survey can be conducted with little more than an access point and an analyzer contained within a notebook or even a handheld computer. On the other hand, any kind of more comprehensive throughput performance or stress testing requires clients, and possibly lots of them, the setting up and configuration of which are a decidedly non-trivial affair.
A solution to this problem comes in the form of the Communication Machinery Corporation's (CMC) EmulationEngine, a device which resembles a typical wireless access point, but within its plastic chassis lays a powerful and flexible WLAN testing apparatus. Within a few minutes of setting it up, I had several dozen virtual WLAN clients associated and transmitting data to a wireless access point.
The EmulationEngine (EE) can simulate as many as 64 WLAN stations, each with unique MAC and IP addresses. The device comes in three flavors--802.11a, 802.11b, and a combination a/b/g model. It's not cheap-- the later model, which is the one I tested, uses a tri-mode Atheros AR5002X chipset and will set you back $6,500. Either of the single-mode units can be had for $4,600.
Once you've selected a system under test (SUT) you can then create a test scenario and configure one or more wireless stations, or vSTAs (Virtual Stations). EE scenarios can be saved either to a PC or handily, to the unit itself for portability (1.2 MB of Flash memory is available for user storage). The unit even creates automatic backups of any scenario file saved to the device. A glitch, however, prevents updating a scenario by overwriting an existing file--you have to first delete the original file.
vSTAs can be organized into one or more groups. The groups can then be edited, executed, and monitored separately. Individual vSTAs can also be manipulated and monitored in real time and individual or groups of vSTAs can be brought online all at once or in a scheduled fashion to test scalabilty.
Used alone, the EE can generate ping traffic and measure a wide variety of WLAN network parameters like signal strength, data rates, associations and authentications, packet loss, and so forth.However, the EE can't internally generate large amounts of application traffic, so to do application-level throughput or stress testing, the EE can be used in conjunction with third party load generator utilities like NetIQ Chariot. Getting the EE to interoperate with Chariot takes a bit of extra configuration and coaxing (mostly on the Chariot side to spoof multiple IP addresses on a single endpoint), but it didn't take long before I had the EE forwarding traffic from a Chariot performance script.
I was able to get the EmulationEngine to successfully interact with several access points from different vendors, representing each of the three flavors of 802.11b. Only with a couple of TI ACX100-based 22Mbps 802.11b+ devices was I unable to get the clients to associate properly, even though the access points were configured to use an 11Mbps signaling rate.
At the moment, the EE supports three levels of WEP encryption (up to 152-bit). WPA encryption isn't currently available, but it's coming in the next revision of the firmware, which is due before the end of the year.
When using the EE in conjunction with Chariot to test a couple of 802.11g access points, I observed data throughput rates of only about 10 Mbps, about half of what it should have been given the short distance involved.
After working with CMC, it was determined that the EE was for some reason implementing RTS/CTS on the vSTAs, which increased the overhead and lowered the throughput accordingly. Troubleshooting this required use of the CLI since these parameters aren't exposed through the Web UI. After some experimentation with various settings and disabling RTS/CTS mode, the unit returned throughput more in line with expectations. For its part, CMC indicates that it is aware of and investigating some issues the EE has in 802.11g environments.
To collect and interpret the various data the EE can generate, the device offers several dozen counters that can be used to monitor the client activity in real time. Unfortunately, once you create a monitor, it can't be edited or modified, so adding or removing parameters means recreating it.
The EE offers event logs and a number of detailed reports that can viewed on the device or exported into HTML and comma-delimited text formats. CMC also offers a PERL SDK for the EE, to facilitate scripting of test operations.
The product does have some practical limitations you might run up against if you have certain complex testing requirements. For example, you can't split test traffic across multiple access points, since the device can only communicate with one AP at a time. Similarly, while the tri-mode unit can operate in a, b, or g modes, it can't support multiple modes concurrently. As a result, tests requiring mixed mode clients or multiple target access points will require more than one EE.
There's no question that the EmulationEngine can be immensely useful when performing a variety of WLAN performance tests. While the unit does require some effort to learn how to configure and tweak (especially with 802.11g mode), use of EE is clearly preferable to the time and effort involved in the setup and configuration of multiple individual client stations.