With TVs on every seatback, more leg room and leather seats, it's no wonder JetBlue Airlines Corp. has won awards for pleasing customers. But the website is the company's customer interface, which is why the airline's quality assurance team has zeroed in on website performance testing to simulate hits and measure response times.
"We measure the footprint of the applications we test,"said Sagi Varghese, QA manager at the Forest Hills, N.Y.-based carrier. "Our website gets up to a million hits per day, so we simulate that many hits to determine the physical system resources needed to support this volume of traffic."
Using the analogy of an ice cream shop, Varghese explained that as people wait in line with only two people serving, the line gets longer and the wait time increases unless more people jump in to serve. To prevent website hits from building up, JetBlue uses Hewlett-Packard Co.'s LoadRunner to determine what the variable drain on system resources is between peak and average periods. This ensures that website response times remain around three to five seconds. "The load changes, but our response time is constant," he said.
JetBlue has been using HP's LoadRunner performance testing software for more than four years. Last week, HP announced a Software as a Service (SaaS) version to give companies a pay-as-you-go option. Because of the difficult economy and the investment JetBlue has made in site licenses for LoadRunner, however, the airline has not made a decision to transition to the SaaS option yet.
"Today, we have purchased the maximum number of licenses, which is safe for us but not as cost-effective as having the optimal number of licenses, which is what we would have in the cloud," Varghese said.
Whatever the purchasing model, it's clear that companies need to test the scalability of their websites to be sure that when the load is high, the site doesn't crash, according to Tom Lounibos, president and CEO of Soasta Inc., a Mountain View, Calif.-based provider of another website performance testing offering called CloudTest On-Demand. Three years ago, for example, TurboTax expected 150,000 customers to hit its website on April 15. But Intuit Inc., the Mountain View, Calif.-based maker of TurboTax, received a surprise when twice that many people tried to load their tax returns. The system crashed, forcing the Internal Revenue Service to grant an extension to frustrated and anxious TurboTax users. This year, Intuit contracted with Soasta to test its site and find the vulnerabilities.
"We simulated thousands of users hitting the site, looking for latency and crash points," Lounibos said. Of course, Intuit can predict when users are going to hit the TurboTax site; not all companies can do that. "The folks [at Intuit] will tell you this is the first year they've been able to sleep on April 14."
The inability to measure application performance from the user perspective has been a top challenge for organizations, and may in turn be the cause of declining customer satisfaction, page views and conversion, according to a recent report by Aberdeen Group Inc., a market research company in Boston.
Advantages of SaaS website performance testing
Now in its 10th year, JetBlue employs about 15,000 people in 60 destinations. For an airline that prides itself on making good cost-cutting choices (onboard snacks not among them), the cost of LoadRunner site licenses ($1,328 each, according to HP) really adds up. But saving money isn't the only incentive to consider SaaS website performance testing; more importantly, the cloud enables "elasticity" -- the ability to scale on demand as users communicate with a cloud-based application.
Our website gets up to a million hits per day, so we simulate that many hits to determine the physical system resources needed to support this volume of traffic.
Sagi Varghese, QA manager, JetBlue Airlines Corp
"We often have fluctuations in demand," Varghese said. "Today, for example, we're struggling to run two different tests on one licensed server. We'll have to schedule them for different times. In the cloud, it would make it much easier for us."
Or take the Super Bowl ad by Denny's Corp. for a free "grand slam" breakfast. "Denny's couldn't have simulated 49 million people hitting their website in two hours without the cloud, nor could it handle that spike without cloud elasticity," Lounibos said.
JetBlue chose LoadRunner before SaaS options existed because the software supports multiple protocols, including the remote deskop protocol and the Citrix network protocol. In addition to website validation, the airline uses LoadRunner to test response time for thin clients on JetBlue's Citrix network. The thin clients are used by 500 people simultaneously around the clock at domestic and international destinations. The network response time varies, based on whether people are booking a flight or changing a reservation, but the test at least ensures "a reasonable" wait, Varghese said.
"Four or five years ago, we did an assessment of LoadRunner with other products; the other products were very specialized to certain protocols -- a lot of homegrown programs that would allow you to do some website testing. In our environment, LoadRunner supports a number of protocols," he said.
A spate of cloud-based website performance tests
While LoadRunner is a popular on-premise performance testing package, HP is playing catch-up in the cloud. Compuware Corp.'s Gomez, Soasta's CloudTest and Capacity Calibration LLC's CapCal are just three of the many cloud-based programs that enable CIOs to make sure enough website resources -- and no more -- are provisioned for optimal response times.
Gomez tests applications from 100,000 locations on Compuware's network, while CloudTest and CapCal are available as services from Amazon Elastic Compute Cloud, the same cloud platform HP chose for LoadRunner in the Cloud. The SaaS version of LoadRunner is currently a free beta. Pricing for the pay-as-you-use service, or the length of the free beta trial, haven't been determined, an HP spokesperson said.
Soasta's CloudTest is an hourly service that simulates up to 10,000 virtual users "at a very low cost" and scales up on a tiered basis, said a Soasta spokesperson. Plano, Texas-based Capacity Calibration is likewise vague about pricing. Another company, Ness Technologies Inc., offers a SaaS performance monitoring service that includes contracted experts in a work-on-demand model.
Let us know what you think about the story; email Laura Smith, Features Writer.