When Mike Grillo joined Foxwoods Resort Casino in the 1990s, the computer room was a 6-by-6-foot closet. A decade
later, the casino's IT system moved to multiple computer rooms and the staff was backing up 2.5 TB a night from servers to tapes.
But data protection at the sprawling complex in Mashantucket, Conn., was starting to seem more like a crapshoot than a sure bet.
Billed as the world's largest casino resort, Foxwoods houses six casinos offering hundreds of table games and 7,400 slot machines. The site includes three high-rise hotels and attracts 40,000 customers a day; 13,000 people work there.
While customers play, an array of computer systems is going full tilt. That includes hotel reservations systems, sophisticated financial management, slot machine monitoring -- and the "cream of the crop," a casino management application that tracks players and their ratings.
The gambling complex is owned by the Mashantucket Pequot Tribal Nation, so the IT staff also provides crucial services such as 24/7 E911 communications to that community.
"Our biggest concern was that we were having multiple media failures in our backup environment,'' said Grillo, principal engineer, MIS Enterprise Systems. "When we could recover, it was slow because the tape drives could only spin so fast." Restoration times could take as long as 26 hours.
The casino had been using IBM's TotalStorage 3494 Tape Library (with 10 IBM TotalStorage Enterprise Tape System 3590 tape drives) and IBM Tivoli Storage Manager software for data backup and restore functions.
Trouble was, the tapes never left the library. The data in the server was backed up in the same room. One computer-room disaster would bring down the house. "Not good," Grillo said, but it wasn't until late 2004 that he was able to get the funds approved -- about $500,000 -- to tackle the problem.
The task, the trouble, the solution
The task: to improve backup and restoration time; eliminate media or drive failures and reduce hardware maintenance costs on the two-tape libraries and the casino's 10 tape drives. IT also had to create a disaster recovery site.
Grillo said he initially thought disk was the way to go, but it proved too expensive. He then identified virtual tape libraries from three companies -- EMC Corp., StorageTek Corp. and Sepaton Inc., a Marlborough, Mass.-based company that came knocking on a cold call. The casino looked at the Sepaton S2100-ES2 virtual tape library. "Sepaton was very helpful. They actually drove their test equipment directly down and we did an on-site test very quickly," Grillo said. The speedy delivery wasn't the determining factor, but speed did win the day. Sepaton out-performed EMC on restoration times. "That was my main goal."
Email is like a lifeline here. For whatever reason, if we don't have it, the world has ended.
Mike Grillo, principal engineer, MIS Enterprise Systems, Foxwoods Resort Casino
Grillo shipped the casino's APC server rack to Sepaton for integration, so "all the racks looked the same." Then came the on-site installation of the Sepaton unit, which holds 57 TB of uncompressed data and includes the hardware and virtual tape library software. "The entire process took 35 minutes to integrate," Grillo said.
They set up the configuration of the virtual drives and tapes of the library, cabled it to the Tivoli Storage Manager server and configured TSM to recognize the new library and tapes.
Since making the change, Grillo has seen a 200% speed improvement for data restorations and no media or tape drive failures. "I don't have to worry about 200 data tapes being marked as damaged," he said. Hardware maintenance costs were reduced by 10%. The number of physical tapes the staff had to manage dropped from 1,500 to a few hundred that stay in the library off-site. "So where I had 1,500 tapes in a non-disaster recovery solution that were just between two tape libraries in one spot, I put the virtual tape library up as my primary storage and I'm using a new tape library with maybe 120 tapes in my facility here, nine miles away, in the disaster recovery site," Grillo said.
The payoff? "Email is like a lifeline here. For whatever reason, if we don't have it, the world has ended," Grillo said. In the past, a failure could take up to 24 hours to restore, because he had to wait for the drives to be free. "Now, I don't have to wait for the data to migrate. My restoration time for a recent outage we had was under two hours. That's huge."
Let us know what you think about the story; email: Linda Tucci, Senior News Writer.