Every CIO has had to deal with a network going down or a dead server. It's par for the course. But what would you do if faced with a complete IT systems failure? Four years ago, Paul Stokes was staring down this very scenario -- and to make matters worse, his budget had just been slashed, and he was severely understaffed.
Almost every IT system began to drop two months into Stokes' tenure as CIO at the University of Northern British Columbia. With 3,500 students and 600 faculty and staff members relying on his department, Stokes did the only thing he could do -- rolled up his sleeves and pushed forward. Now CIO at the University of Victoria in Victoria, B.C., Stokes looks back at his former CIO role and talks about how virtualization helped him reverse this widespread IT systems failure:
What was the situation you walked into four years ago?
Stokes: On campus they had two [IT] departments -- academic and administrative computing -- and a year prior to me taking over they had merged those two departments together. It didn't go so well. That same year, they had a new president of the university, and the president decided to do a major restructuring across campus. In that restructuring, IT got a significant budget cut. We ended up with about 33 people for all functions of IT, from servers to networks, telecom, enterprise applications, software development, desktop support, Web development, project management, audio/visual, the whole thing, plus institutional research. Essentially, there wasn't anything we didn't support or do. It was full-service IT to everybody. Much of the hardware infrastructure, as well as software infrastructure, hadn't been upgraded for about five years. The merger had affected morale. Two months after my role started, all the servers and applications just started failing.
How did you react to the realization that you had such a widespread IT systems failure on your hands?
Stokes: We were really in break/fix mode from that point on for about a year. The objective was stability. John P. Kotter says to create a sense of urgency -- we already had one -- but we had to figure out how to get it moving forward. We did that by getting people vested. It created motivation for the staff to say, "We can't handle just fixing things daily; let's start solving the problem." It was really a catalyst for moving forward.
How did you begin solving the problem? What role did virtualization technology play?
Stokes: Virtualization was at the core of our strategy. Prior, it was single server/single server. You had a huge sprawl that had occurred in IT, which is why we had so many failures. We had three or four SANs also failing. The goal was a complete redesign of everything from the ground up, and this couldn't be done across the board in one fell swoop. We focused on core infrastructure. We had to get storage up and running, [and] we had to get virtualization in place. Those didn't fall hand in hand together. Virtualization came before storage because we had to get the servers migrated. We put in VMware for the virtualization technology infrastructure, which solved a lot of our problems and needs, both on the server side as well as moving toward a virtualized desktop piece. The other piece that helped was a NetApp solution, [which] solved a lot of time and energy issues we were having, allocation of storage, duplication.
How did you address the failing business applications and operating system?
We had to upgrade all the applications because many weren't being supported by vendors anymore. It was basically re-implementation of the product itself. As we migrated a server, we decided whether to do an upgrade of the application in place, or we just simply moved the server to the virtual machine and then waited for the application upgrade to address other critical issues. We replaced our backup system because we couldn't recover anything at that point. Now we use a storage-based backup with a little bit of tape. It was a huge flip. In terms of operating systems, we moved from Windows Server 2003 to Server 2008.
How were you able to staff back up to 50 people on your team?
Stokes: We determined we had funds we could re-allocate. So the institution put forth funds to hire new positions because they knew that we were understaffed and all that we could get done at that time was operations, at best. There was no growth, there were no new systems, there was nothing, and it was all falling apart. It came in two waves. The sell point on that was really the plan that was brought forth. It was rationalized and prioritized. But also, IT was re-investing the money back into those positions. We were willing to work with them to figure out how to accomplish this. It was really a partnership.
What needs and skill sets did you address?
Stokes: There was a lack of training and development that had gone on for many years. We did a large investment in training around the projects. We really tried to leverage our vendors to get the training we needed. We also invested in sending people strategically to conferences. It was an evolution of existing skill sets to move us to the new world.
IT had picked up a negative reputation on campus due to the IT systems failure -- how were you able to change that?
Stokes: We made sure that we prioritized what needed to be fixed first, we communicated it well, and when we made a promise, we kept it. So it was trying to rebuild that relationship with trust. People had the option of going rogue, but they decided not to. Without the relationships across campus with all the VPs and the president, it would not have happened. The change in mind-set we had to have was it's not about us, rather it's what we provide for others, and making sure people understood that. This is not an IT thing -- yes, the technology helps you do what you do, but really, it's about the services that you're receiving. We have to make sure those services always work for you. The main thing was, "Don't commit if you can't deliver." P eople don't want empty promises.
Let us know what you think about the story; email Karen Goulart, Features Writer.