Chris Powers, IT supervisor at Palo Alto, Calif.-based financial services firm Mohler, Nixon and Williams was fed up with sleeping at the office to complete backups and tired of hauling piles of tapes home. Something had to change.
Backing up the company's Windows Exchange e-mail had been getting out of hand for a while and was running into production hours. The firm has 110 employees spread across three offices in California and almost 4,000 clients.
Powers was responsible for backing up the company's 250 GB of data daily and managing a total of about 150 tapes. All the disk storage was directly attached from 13 file or application servers (each also running backup software) to an HP Surestore LTO tape library. That was just about manageable in normal production hours from 6 a.m. to 6 p.m. But during the tax season when the partners work 50 hours to 60 hours a week and employees sleep over at the office to get work finished, the backup window cracks under the pressure.
This year, for example, "the database where the tax software resides was accidentally kept open and couldn't be backed up. Then we ended up doing backups in the day time," Powers complained. "The slow network performance drove everyone crazy so we had to deal with it at 11:00 at night ... it was a huge problem at this point.".
The partners refused to limit their mailbox sizes, so the IT department were unable to give them quotas. "They use their e-mail for research purposes ... one partner has upwards of 6 GB," Powers said.
The firm looked at Network Appliance's line, particularly its SAN offerings, and had an account team from the company out several times. "It's a real nice product but the total cost of moving to a SAN was just too expensive for us, we couldn't get the partners to agree ... there's a big cost associated with switching from DAS to SAN," Powers said.
An outside consultant suggested the firm buy JBOD arrays and build a SAN itself. They looked at this idea for about five minutes, realized it would be too difficult and then bumped into a company called Data Domain, a three-year-old disk-based-backup startup from Palo Alto, Calif.
Data Domain redesigned Mohler's backup process. The Data Domain DD200 Restorer disk-based backup-and-recovery file server now directs backup, working with Veritas backup software that now runs on only six of the 13 servers. The DD200 is essentially a NAS device dedicated to and optimized for backup. Now the firm performs incremental backups twice a day to the Data Domain box. Once a week, Mohler does one full backup from this server to tape. The full backup is done offline.
The process has cut the backup window from 12 hours down to six and Powers no longer has to lug around quite so many tapes at the end of each day. The DD200 server works with existing backup applications. Perhaps more importantly, through compression and incremental backups that only save the changed blocks of data, the company has significantly reduced the amount of data that needs to be backed up. "I used to haul eight tapes with me for off-site security; now I haul three," Powers said.
Aside from the cost savings associated with reducing the amount of data it stores, the overall disk price from Data Domain was cheaper than NetApp's offering. "We're paying $1 to $2 a gigabyte, which is half what NetApp was asking," Powers said.
Interestingly, as part of his research for a new backup solution, Powers called associates at other companies to see what they were using. "I called some buddies but they keep their backup solutions close to their chest ... competitive business, this backup stuff," he quipped.
The biggest challenge in converting to the new system was configuring the Veritas software for the new environment. "It was a bit fiddly at first but we got there," noted Powers. Also, initially Mohler was performing one big backup and not getting the performance it had hoped for. Multi-streaming the jobs took care of this.
Eventually Mohler plans to replicate data to a second site for disaster recovery purposes and is checking out Data Domain's beta software for moving compressed data over T1 lines. That product will be generally available in the third quarter of this year.