Planning for the unknown
Maintaining your network infrastructure during a disaster.
By Jason Free, Features Editor, February 2014
Due to the critical nature of hospital care and patient information management, every facility must create a plan of action for maintaining its network infrastructure during times of crisis such as earthquakes, tornados or manmade disasters. I sat down with Rick Drass, Manager of Information Systems at Sarasota Memorial Health Care System in Sarasota, FL, to learn about the considerations he and his team make when preparing to face the unknowns that accompany times of crisis.
What are some of the fundamental considerations you make when formulating plans to sustain your hospital’s network infrastructure during a disaster?
The first thing we have to assume is that during any disaster we will lose everything. When that happens, I have to ask, where is my backup and where is my backup-up location, my co-location (co-lo) facility? This location is usually a bunker-based facility that can withstand wind forces of a Category 5 hurricane. This location should be somewhere that is not in the same general location of my facility. I need to make sure that at that location, all of our major systems are redundant. In other words, any critical systems that are operational within the hospital need to be duplicated at that co-lo facility. For example, our patient care information systems, you probably know them as health information systems, have to be fully backed up and running at our secondary location. So if the primary system is lost, the secondary system can immediately step in and pick up the work load.
We also have to consider how we plan to maintain new information. If the main hospital information system is lost, and we find ourselves in a disaster situation like a hurricane, we still need the ability to triage new patients and to gather their pertinent information. So there has to be some kind of tagging and triage information system that is mobile and independent of the main hospital information system. This back-up system must be operational here at the hospital and deployable into the field at any given moment to handle any causalities in the area. In addition, the information that is gathered during the event must be able to be integrated back into the main information system once that system is restored.
What types of vendors are most helpful to you when you are preparing your disaster plans?
Initially, I would need to find a vendor who had a leasable co-lo facility that can withstand the effects of a disaster. Otherwise, a secure back-up location would have to be built from scratch.
Next, we need to partner with our current major software vendors. In our case, that would be Allscripts. They are our vendor of choice for our patient information system. We have to partner with them and make sure they have redundancy built into their system. We also partner with companies like Microsoft to make sure that we can install a clustered environment that is fault tolerant. So, if one server goes down another server picks up the work load without a heartbeat being skipped, technically speaking. We also partner with local communication companies. These communication companies help us make sure that we have fiber running from Point A to Point B and enough bandwidth to meet any contingency. (Point A being the hospital system and Point B being the co-lo facility.)
We will sometimes partner with vendors who specialize in disaster recovery drills or disaster recovery testing. The idea is that once your system is in place, how do you know it is going to work unless it is tested? The only way you know for sure is if you turn the switch off on “A” and let “B” take over. We run these kinds of drills on a regular basis.
Do you ever partner with companies who specialize in disaster recovery solutions?
Not generally. While we partner with a few companies who help us with subsets of our disaster recovery solution, we do not rely on an outside company for our enterprise solution. Most of the companies who specialize in disaster recovery will come on your site during a disaster and bring you servers and other equipment to get you back on line after the disaster has passed. However, we find that those kinds of solutions are not useful for our needs. We need to keep the information technology systems running during an event.
Many of the companies we have talked with seem to be fairly one dimensional. They are usually pushing a product that is not all inclusive. We have to think about many situations. We cannot assume that only a hurricane will hit our facility just because we are in Sarasota. Many disasters occur without any kind of notice. We could face other problems like a grid power outage, a tornado strike, a plane crash, etc. These are all scenarios where we are required to keep our systems online and operating.
We have found that by talking to these specialty vendors that very few of them are adaptable to the many contingences that a hospital technology group might face. Most of them have a service where they deliver to your site a truck or several trailers where they have servers and PCs for you to get back up and running. In our case, our needs are far greater than that. Medicine these days has become more and more dependent on information systems. The products and services of specialty companies usually take time to initiate, but we do not have the luxury of waiting. We can’t wait for that trailer to show up. We can’t wait for those servers to be spun up. We can’t wait for those PCs to be put in place. There has to be something already in place and ready to go. We have found very few partners who have wanted to guarantee that kind of delivery and service level.
Hurricane Charley threatens Florida in 2004 (Source: NOAA)
When disaster strikes, how do you move into action?
Let’s use the obvious choice for us, a hurricane, as an example. For that situation, we have two teams. One team, Team A, is responsible for working during the event. The second team, Team B, is responsible for all the work after the event. Team A is based inside the facility during the event and facilitates the technology continuity of our systems. They meet the information needs of any staff member whether they are outside at an ancillary location or inside our local facility.
Once the hurricane passes, Team B comes in to relieve Team A. Team B will restart any systems that had to be taken down during the event. They also recover any lost data, and they get things back in normal working order. Whatever is initiated by Team A, it is finished by Team B until the effects of the disaster are mitigated.
Of course, there are other precautions that are taken if we are preparing for a hurricane where there is a likelihood of a storm surge. Where possible, we will move computers ahead of time to a higher location (off the floor). However, unlike many businesses, we do not have the luxury of turning off our systems during a disaster, so we do our best to place our technology devices in a safe location where they can be utilized.
The drills we run through on a yearly basis help us to be as prepared as we can be when a disaster strikes.
What are the types of drills or simulations that you undergo to prepare for a disaster?
At our hospital, in addition to simulating medical disasters where we prepare for the treatment of a surge of causalities, we simulate situations where we experience a loss of data. During these simulations, we fire up our secondary site while the medical personnel are working on their particular drills. We also simulate power outages where we would experience a complete loss of our systems inside the hospital.
We have an Emergency Management Committee that meets once a month to discuss the findings of our simulations and any possible revisions that need to be made to our disaster recovery plans. We discuss what systems and practices need to be re-tested based upon the findings and observations of representatives on the committee. These representatives discuss the tests that have been run and what holes, if any, they have found. Discussions then revolve around possible solutions and improvement plans. We always strive for improvement.
Do you ever evaluate the practices used at other hospitals to modify your own plans?
Yes, our Emergency Management Committee meets quite often to review reports from hospitals that have experiences with events such as Hurricane Sandy or tornados in Oklahoma. We create a “lessons learned” report that helps us to see how we might need to make adjustments to our own emergency planning. What did they learn, and what did they experience that may be similar to our facility’s plan? We incorporate those experiences and findings into our own practices. Even if we have not experienced those sorts of events first hand, they do affect our plans and decision making.
What is an example of a practice currently in place at your hospital that was created as a result of these meetings?
We created a mobile emergency triage data information system that can be deployed to the field that facilitates the tagging and triaging of patients away from our facility such as when the area may experience an airline disaster or building collapse. During a disaster, there are a number of different types of victims: the walking wounded with minor injuries, those who are not ambulatory, fatalities, etc. All of these situations require the gathering of important information that will be placed within our main information system either when the hospital system goes back online or when those field doctors return to the hospital. For this need, our committee found a gap between our needs and the products or services offered in the industry, so we filled it ourselves by creating the necessary interface to bridge several systems.
Another good example is this one. Let’s say we are in a situation where we lose all of our patient information like those relative to patient charts and medication. Let’s say all of the servers go down and there is no way to get patient information on a particular floor. Our care providers, our doctors and nurses, cannot get the information they need. That is a huge hole that no a single company has been able to fill.
We filled it ourselves by developing our own solution. We are now able to send encrypted patient information to a device on each of the floors of our facility. Even if all the power is out, all we need is temporary power to this computer, and a nurse can punch in the key code and immediately have access to all the patient chart information. There is a printer nearby where the information can be printed and then used for patient care on the floor. As far as I know, there is not a single company out there that has anything like this available. That is something we had to invent ourselves.
Unlike most businesses during a disaster that only have to mitigate damage and restart their operations, hospitals have to also prepare for a surge in patient need, both inside the facility and in the field. How do you make plans to handle this surge of work?
In this case, the technology in the marketplace does help. There is a company called Disaster Management Systems who produces a product called Triage Tags, which is a portable IT system that can be run on a single laptop. It is a tool that is used to tag patients in the field. However, there is no way to import that information into your main patient information system. We have to create that interface on our own, so there was a gap. We had to write an interface between their system which sits on a laptop and our main enterprise patient information system. As in most cases, we rely on outside vendors for our products but we cannot rely upon them for creating processes to use their tools during a disaster or surge situation.
We have internal processes and programs to manage a medial surge situation. I have already mentioned the information technology continuity plans involving Teams A and B. However, every department in the hospital has similar processes in place to manage surge situations. We have teams for communication systems, labor pool, medical staff, facilities management, counseling, security, etc. Each department has emergency readiness action plans established to handle any surge that may take place.
So, in your opinion, there is an area of need in the industry in terms of providing interoperability?
Absolutely. I wish that I did not have to write interfaces between disparate systems. I wish someone else would do that. Although I enjoy it, my time could be better spent doing something else.
While interoperability is an issue, is it difficult to find the types of servers, switches and routers that you need in the event of a disaster?
No. We generally go mainline with most of our equipment. We use Cisco, Microsoft, Dell and HP for much of our technology due to the wide level of support offered by these vendors. These devices and technologies are configurable and portable if necessary. In the case of a disaster, these are the technologies we would depend on for business continuity.
Can you provide an instance where a new product in the field caused you to alter your plans?
Sure, infusion pumps are a good example. These pumps are used to administer patient medication on a regular interval. We have recently added Real Time Location (RTLS) tags to our pumps. Let’s say during a disaster there is an emergency in the ER where they need 20 additional infusion pumps, and we need to locate these pumps quickly. Our real-time location system allows us to quickly determine the number and location of infusion pumps that are not in use. We can make that determination instantaneously and get the needed devices to the ER right away.
In the past, we had to look for pumps that were not in use by searching on foot. The RTLS system changes things. No longer do we have to waste valuable time searching for important equipment during a time of crisis. Due to the RTLS system, we have been able to alter our older plan for managing equipment during a disaster in favor of a new, more efficient plan.
If you could speak directly with the vendors about your needs specific to making plans to maintain your network infrastructure during a disaster, what would you say?
Well, several of us have spoken with a number of vendors about this point. I hate to say it, but most of them are not willing or able to help us with many of the challenges we face. As I mentioned before, we spoke with two or three different vendors over the course of several years and asked them to provide a method of printing out patient records when power is lost to our information systems. They said it was not possible. Of course, we now know it is possible because we found the solution to this problem and implemented the solution on our own.
The best way for vendors to determine what hospital technologists and administrators need in the way of disaster recovery is for them to walk a mile in our shoes. They need to observe our drills and simulations to see for themselves the real life-and-death issues we face. That is the only way I know of that they will ever truly understand our challenges in preparing for disasters.
Rick Drass is Manager of Information Systems at Sarasota Memorial Health Care System in Sarasota, FL.
Options to consider
The need for reliable and sustained communication is never more important for a healthcare facility than during a time of crisis. With over 1,000 certified communications professionals, Sprint Emergency Response Team (ERT) supports the planning, implementation, execution and evaluation of your facility’s disaster drills and field activities during a crisis.
Sprint ERT also offers an inventory of equipment to ensure scalable and robust communications for facilities during any catastrophic event. With a Sprint ERT Go-Kit, a robust tool containing a cache of Sprint communication devices and mobile broadband cards, healthcare facilities can be assured that wireless devices and IP services will be ready at a moment’s notice regardless of the emergency.
Comcast Business owns and operates its fiber-based network, which is completely separate from traditional phone carrier networks. The Comcast network covers 39 states and is serviced directly by Comcast 24/7/365. It gives healthcare providers another option for network connectivity. Having multiple connections allows healthcare organizations to establish redundant paths to the Internet. In the event of a disaster, this redundant connectivity can keep a healthcare provider connected to its business continuity/disaster recovery facilities for access to medical and patient information, and to help with regulatory compliance. Within the Comcast network, there is extra redundancy built into the network design to maximize availability.
Tags: Tactical Operations