As wildfires forced the largest evacuation in California's history, CIOs jumped into action this week, taking advantage...
of everything from wireless and wide area networks to Web-based collaboration tools to help keep their businesses running, employees reassured and shuttered physical properties protected. They also had time to lock down their companies from cyberspace attacks.
Their technology strategies are proving up to the task in this trial by fire -- so far. Several of the CIOs we were able to contact said things would have been much worse had their companies lost power.
Tech-savvy mobile workers, armed with laptops and accustomed to accessing their workplaces remotely, are playing a big role in business continuity, many pointed out. The move from centralized to distributed workplaces, like the Internet itself, helps ensure continuity because there is no single point of failure. The benefits of connectivity underscore the importance of the grid.
When it became clear that the Escondido, Calif., headquarters of training and management firm The Ken Blanchard Cos. was in the path of the rapidly spreading fires, the company's CFO set up a Sharepoint site. The site became the family-owned firm's lifeline.
"When I went to bed on Sunday, I wasn't even aware of the scope of the problem," said Terry Orletsky, vice president of IT. "There was the Harris fire, which was getting a little serious, but it was nowhere near here. And then the Witch Fire started."
The Witch Fire has since consumed about 197,990 acres in northern San Diego County, from Witch Creek to Rancho Santa Fe.
"I was actually getting ready to go to work when I got a call from the CFO saying 'you better stay home.'"
The Escondido campus was closed, and one-quarter of its 210 Escondido employees were evacuated from their homes. The house that owners Ken and Marjorie Blanchard have lived in for 30 years burned to the ground.
But the Sharepoint site was up and running, so Orletsky said he "jumped up there and categorized the announcements and we gave it some order."
Sharepoint is a portal-based collaboration and document management platform from Microsoft that lets people work together while located remotely from each other. Sharepoint became The Ken Blanchard Cos.' town center for everything from information about health care prescriptions and housing to listings of contact numbers and details on how to donate to fire storm relief. Groupware programs such as Sharepoint can also include the sharing of calendars, collective writing, email handling, shared database access and electronic meetings.
"I thought it was kind of funny that our financial guy set this up," Orletsky said, noting how easy it was to do. "And the human resources director took over the leadership of the housing and emergency plans. It was a good collaborative effort."
But the flames and smoke also brought into sharp relief the need -- even for smaller companies on tight budgets -- for a disaster recovery site.
"We don't have a disaster recovery site," Orletsky said. "I was very worried about us losing power. We have battery power here, but only enough to allow us to shut down gracefully."
IT maintains a colocation site in San Diego County for the company's Web assets, but "that's not the best plan in Southern California," Orletsky said. Ken Blanchard Cos. is about 60 miles from the San Andreas Fault.
"We don't have the resources to compete with big companies. In a widespread disaster we could not get equipment as quickly as someone who could pay premium prices. I think it is imperative that we build that thing, but it's on the leadership of the company. We're privately held. It can run into a couple of hundred thousand dollars," Orletsky said, not counting other issues like bandwidth problems that would have to be addressed.
IT has laid the groundwork for a remote disaster recovery site over the past year and half, he said. "We're beautifully set up to do it." Almost all the servers are virtualized, and a storage area network is in place. "If they can release the funds to buy the hardware, then we'll go ahead. And I suspect that at our team meeting tomorrow we may just get that in the capital budget."
Construction company Rudolph and Sletten, a wholly owned subsidiary of Perini Corp., is headquartered near San Francisco, but has billions of dollars worth of construction projects in Southern California, and offices in San Diego and nearby Irvine, Calif. The company maintains a disaster recovery site in Irvine and a coordination room 65 miles from its headquarters on a different fault line.
I was actually getting ready to work when I got a call from the CFO saying 'you better stay home.'
Terry Orletsky, vice president of IT, The Ken Blanchard Cos.
Although those sites were not closed down, many Rudolph and Sletten employees in the San Diego and Irvine sites were either evacuated or were unable to get to work.
"It was an interesting challenge to maintain security, establish communications with employees who were evacuated or lost their homes in the fire, and to give clear communication on which jobs were open and which were in peril," CIO Sam Lamonica said.
The construction company deploys wireless networks in offices and job sites companywide. BlackBerry and Palm personal digital assistants are standard issue for most of its employees, so email is at the ready.
Cell phones were lifesavers, too, Lamonica said. Starting early Tuesday, two teams began calling every Rudolph and Sletten worker in the fire area, including those subcontracted through the unions and craft guilds. The company keeps lists of emergency numbers for all employees and families, which were updated three times a day during the worst of the fires.
"We've been dealing with everything from people who own ranches in San Diego whose horses were in jeopardy, to people who lost their homes." As CIO, Lamonica said he acted as the voice of the company, literally, giving updated messages on a 1-800 number.
The Irvine office functioned as an emergency response location, supplying equipment such as fans to suck out smoke from homes and job site trailers, and spelling out emergency procedures. "The smoke even in Irvine was really bad. So we briefed them on things like making certain air conditioning filters didn't get clogged, wearing masks, you name it."
There are always lessons learned in situations like these, Lamonica said. In retrospect, he said, he would have started emergency efforts for the wildfires four to six hours earlier. "But our being prepared from an IT side was extraordinarily important." The company honed its disaster recovery process years ago, after the Loma Prieta earthquake. "I am sure you have heard this 100 times, 'Nobody puts a lot of emphasis on disaster recovery until after the disaster happens.' But we really focused on it here, because we knew there would be employee and job site safety as well as calls from customers asking us for help."
Let us know what you think about the story; email Linda Tucci, Senior News Writer.