Disk management was a major issue for Health Alliance Plan (HAP). Five years ago, the Detroit-based healthcare solution provider suffered from the usual problems and limitations associated with direct-attached storage. With storage devices "owned" by specific servers, there was no way to allocate capacity where it was needed. As a result, the IT department had to keep purchasing and deploying new storage to meet the demands of growing applications, while large amounts of storage "just sat there, unused," notes Dan Trim, the company's manager of operations and technical services.
Deploying new disks was a time-consuming process that required taking the server down. Plus, IT was stuck in reactive mode, playing catch-up with new capacity demands about 80 to 90% of the time, Trim recalls. "Everything was jump through a hoop, stop what you were doing to put some disks on, buy me another unit now."
The technology deployment
The company tackled its storage problem in a phased, multiyear technology deployment that's still being fine-tuned, and whose full benefits the company is just starting to realize, Trim reports.
The IT group began by installing a storage area network (SAN): two Brocade Fibre Channel switches. It also installed IBM Shark ESS800 and Hitachi 9580V storage subsystems, and a tape area network (TAN).
The network makes the IBM and Hitachi storage subsystems available to the company's full complement of servers, which currently comprises about 45 Sun Microsystems Inc., Windows, Linux and AIX systems. "When we bring in a new server, we just connect it up to the tape area network and SAN, and tell it what storage devices it can see," Trim says. Both the IBM and Hitachi systems support RAID, automatically shifting to a backup disk when one is about to fail. Previously, an administrator often had to take the server down in order to restore the data on the failed disk from tape. "Now the system can rebuild a failed disk on a spare disk without ever going down," Trim reports.
While the SAN infrastructure was a vast improvement over the old direct-attached storage, Trim was not satisfied. "We still had each administrator highly involved with managing storage for individual servers," he recalls. Administrators had to query each server, using a different set of tools, in order to get critical information like how much of its assigned disk space a server or a database application had used up, or how much was left.
Furthermore, provisioning space for a new server remained time-consuming and complicated. Administrators had to physically go to the server and lay out disk space using the server's own tools. That meant they had to be familiar with several different volume and disk management software tools.
Revamping the storage infrastructure
Two and a half years ago, HAP deployed Veritas' (now a Symantec Corp. subsidiary) CommandCentral Storage. "Instead of having to look at each server as a single entity, we can look at the entire SAN and TAN storage infrastructure across the enterprise," Trim says. Administrators can see all servers on one page, then drill down to view storage utilization levels for individual servers or applications.
Savings
The platform has enabled HAP to reduce storage administrator man-hours by 90%, Trim reports. Furthermore, the company now does troubleshooting and capacity planning on a proactive basis. Previously, administrators had to use special tools to access and then analyze utilization data from individual servers. Now they automatically get color-coded threshold alerts when a file system or application has used up 90% of its capacity. "A picture is worth a thousand words," Trim says. "We can allocate additional capacity before an application runs out or there's an outage."
With CommandCentral's help, Trim's group has also "been able to meet storage demands quickly without outages or business interruptions," and maintain service levels "well above requirements: at 99.9% availability," over the past three years, he says. This includes a two-year period when the company was experiencing 40% annual growth in storage capacity demand.
Meanwhile, total cost of ownership for storage is down by 12%. There have been zero unplanned outages due to disk problems during the past three years.
Storage forecasting
The IT group now conducts an annual forecast of storage capacity demands. It then uses CommandCentral to clean up the old volumes, take back unused space, and lay out new volumes to meet a server or application's expected capacity requirements for the coming year. "We're no longer chasing disks around," Trim enthuses. "We're proactively handling disk management, preparing for next year before it gets here."
Elisabeth Horwitt is a contributing writer based in Waban, Mass. She can be reached at ehorwitt@comcast.net.
