Measuring Metro Wi-Fi
November 14, 2006
Novarum tests citywide wireless broadband networks to measure something unique: usefulness.
The co-founders of Novarum have given themselves a rather unique mission. Instead of pontificating about what's happening with wireless IP data/voice networks in big cities, theyre actually going to measure just how useful those networks are for end users in accomplishing what they call meaningful work.
Managing partners Phil Belanger and Ken Biba worked together at Vivato (they left before it imploded). Belangers Wi-Fi pedigree goes way back: he was the original chairman of the Wi-Fi Alliance and a co-author of the original 802.11 specification. Their new company started out doing analysis and consulting, and while they admit theyre not shy with opinions, they say they wanted to base it on facts.
To have those facts in hand, they have started an ongoing survey a site survey, really of cities that have installed wireless networks. That means citywide Wi-Fi as well as 3G network technologies like HSDPA and EV-DO from the cellular carriers. Theyre even looking forward to the day real mobile WiMax is up and running. They call it the Novarum Wireless Broadband Review.
It wont be a quick stop in each city, nor just a one-time test per city. We hope to confirm that the performance [of a network in a city] will change over time, and providers need to invest to keep up with the use, says Belanger.
Biba says, Were using tools that have reproducible results, adding that what he finds really amazing is that no one has done this.
The testing consists of driving around and hitting the entire footprint of a network as indicated by a service provider.
We do two things, says Biba. First, verify coverage, using essentially a laptop with a specialized processor, a Wi-Fi card, and a little better antenna. We war-drive the city. We have overlay maps on Google Earth and create coverage maps for the Wi-Fi service, inferring it based on what we see. Detailed maps will be available in the paid report.
As we drive, we randomly pick 20 points, scattered, usually about four per square mile, to do an explicitly high performance test with the tool IxChariot, Biba says. We run the same scripts on all the networks, Wi-Fi or 3G, both uploading and downloading. They even record the exact GPS location of where theyre sitting in the car, so they can come back to it later.
So what did they find?
Biba says, I hate to say we have a bias... but we were skeptical of how Wi-Fi is deployed in cities. What surprised us most, if you go to a place with deployed Wi-Fi and theres also 3G service you have the same probability of getting service. Its a factor even more accurate if you mean real 3G, as most providers dummy down to a 2G level of service in some cases, say 100 Kbps downloads, to support more users.
On average, in the six cities Novarum tested, there was a 60 percent chance of getting a useful Wi-Fi connection at a random spot outdoors. Anaheim the first citywide Wi-Fi network turned on by EarthLink provided a 72 percent rate of access.
The term useful is key, as they point out that a sniffer might sense beacons from the network, but that doesnt mean a connection could be made that was usable.
For it to be meaningful to users, it means doing meaningful work, says Belanger.
Not all providers wanted to hear Novarums results. They dont like what they hear on first blush all the time, says Belanger. But when we explain our methodology, they want the information. We dont say, It didnt work well; we show them exact locations so they can verify it.
Not all providers react the same way to news of low availability, either. EarthLink was unhappy to hear of the 72 percent attach rate for Anaheim. Yet MetroFi, which unwired Santa Clara, California, wasnt at all surprised or upset to hear of Novarum getting a 55 percent service availability there.
They said they expected 50 percent once we defined our criteria, says Belanger. MetroFi, of course, offers free service, and only puts up about 20 nodes per square mile. EarthLink puts up closer to 30 nodes. [MetroFi told Wi-Fi Planet on January 31, 2007, that Novarum's test in Santa Clara was of an older network with original test bed software dating back two years, on a network the spokesperson says "was never made to be rolled out in primetime." It indeed has 20 nodes per square mile, but MetroFi says it has gone to 30 nodes per square mile on subsequent "primetime" networks. MetroFi says it has asked Novarum to stop publicizing data from this test. However, Novarums first top ten list of metro Wi-Fi networks lists MetroFi's Santa Clara net at #6, based on a 70% service availability there -- Phil Belanger says the original 55% number was in error and that "70% is pretty good for the nets we tested."]
In our experience, performance is constrained by the providers business decisions, not the limitations of the equipment, says Belanger. That includes items like the backhaul, the Internet pipe and, of course, the number of nodes per square mile. MetroFis node density is likely very consistent with its business model. If they got more paid subscriptions, theres nothing holding back putting in more nodes, Belanger says.
All in all, where available, they were pleasantly surprised by the Wi-Fi. So what about the 3G services of Cingular (using HSDPA), Sprint and Verizon (both with EV-DO technology)?
While 3G is more mature technology than the metro Wi-Fi, it still offered half the data throughput in most cases. And it just wasnt as available. Biba found only 34 percent service availability (jumping to 78 percent if you combine it with the 2G fallback). Their report states, Metro Wi-Fi is more available than true 3G. The fallback was okay for smartphones, but not for laptops.
Novarums first six cities were all California towns, but theyre branching out for the fourth quarter.
In Anaheim, Cingulars HSDPA has the best 3G performance, but even more limited availability; Sprint had the best service overall, with 100 percent service availability. Best performance by far was EarthLinks Wi-Fi, with 811 Kbps downloads and 380 Kbps uploads. As stated before, they had 72 percent availability.
Other cities tested included Galt, with Sprint and EarthLink (similar results to Anaheim); Palo Alto, with Verizon and its own unorganized open access point ubiquity; and Santa Clara, with Cingular at 100 percent availability compared to MetroFis 55 percent.
After enough testing, the individuality of a citys network comes out. Each of these networks has a personality, says Biba. Each feels different. He attributes that to everything from the node density to the congestion to just the connections available. The whole Bay Area has a much different feel from Anaheim in southern California.
Novarum plans to keep the Wireless Broadband Review going with tests of 10 cities per quarter, and repeat testing in some locations. A paid annual subscription can get providers and other interested parties full access to all the reports and data. Theyre even eager to come in as a third party and do the testing and technical due diligence for cities. After all, its not all altruistic. We want to make money, Biba says. That money should keep them testing for a long time, until all the kinks are worked out of 3G and metro-scale Wi-Fi. And that should take several years.