Disaster recovery solutions underscore the importance of security
Regardless of the selected solution, experts agree the most important criteria for a disaster recovery (DR) backup system is that it is secure.
By Phil Colpas, Editor, May 2013
As a graduate of the University of Florida’s College of Journalism & Communications, I was struck especially hard by recent news that nearly 15,000 patients at the UF&Shands Family Medicine at Main practice, where most UF students receive medical care, might be the victims of identity theft. (For more on this unfortunate event, see Health Management Technology’s April 9 e-newsletter.)
Unfortunately, incidents like this one have become much more prevalent of late, as hospitals and practices struggle with how to keep records secure while also ensuring they are properly backed up in case of a disaster. And while backing up all patient records is a necessity, it also increases the chances of a breach occurring.
Accellion, a provider of secure file-sharing solutions, estimates data breaches cost the healthcare industry approximately $6 billion a year.
From augmenting security through a plethora of means, to exploring various ways of fending off cyber attacks; from maximizing audit readiness, to the challenges of securing mobile media, keeping protected health information (PHI) safe has become of paramount importance to the healthcare industry.
According to the U.S. Department of Health and Human Services (HHS), The Health Insurance Portability and Accountability Act (HIPAA) Security Rule establishes national standards to protect individuals’ electronic personal health information that is created, received, used or maintained by a covered entity. The Security Rule requires appropriate administrative, physical and technical safeguards to ensure the confidentiality, integrity and security of electronic protected health information.
So how do we ensure that patient information is both secure and able to be recovered in case of a disaster?
HMT asked select industry experts that very question.
VP products, Axcient
Minimize data loss and maximize recovery
As hospitals and healthcare providers move more records and sensitive information online, data protection becomes an increasingly important issue. Ponemon Institute’s Benchmark Survey of Healthcare Providers on Patient Privacy and Data Security (December, 2012) indicates a rise in both occurrence and costs of data loss. In fact, 96 percent of organizations admitted to at least one breach within the past two years. The estimated financial impact of these breaches was in the billions, with 81 percent reporting time and productivity loss, as well.
Just as risks are increasing, so are the penalties of noncompliance. Beginning Sept. 23, 2013, HHS will start enforcing HIPAA rules for maintaining the secure data backup and recovery of PHI. Healthcare businesses and their associates (including IT providers) will face investigations and penalties up to $1.5 million per violation if they do not comply.
Data loss and downtime can be caused by a variety of daily challenges and risks, ranging from natural disasters to equipment failure and cyber security threats. As a best practice, all healthcare providers should conduct a risk analysis and then develop an action plan to address vulnerabilities. They should also test their backup and recovery technology solution to identify potential gaps in data protection. Testing should evaluate whether the solution in place can ensure the complete protection and immediate recovery of patient records, scheduling systems and billing programs so that healthcare providers can continue serving patients and maintaining HIPAA compliance without loss of productivity.
Jon Ryalls, records and information solutions architect,
Canon Business Process Services
Three critical elements that make a difference in disaster recovery
Because patient data includes sensitive information, it is critical to manage and protect it as much as possible against potential disasters. Based on my experience providing managed services to healthcare organizations, I’d like to spotlight three critical elements I believe can help meet these goals:
- Identify how medical information is being managed and how it can be retrieved in the event of a disaster. This includes clarifying gaps in compliance with such industry regulations as HIPAA and HITECH.
- Control information more effectively by implementing policies and technology that ensure secure access and provide proper retention so necessary information is available when needed. Mobile devices utilized for remote access and sharing of patient data also need to have controls to ensure patient data is not compromised.
- Educate employees via regularly scheduled training sessions on how patient information is accessed and collected during a disaster. Continually update and improve the training by learning from recent disasters, such as Hurricane Sandy.
Let me bring these principles to life with an example. Two years ago, a major hospital in Joplin, Mo., was devastated by a violent tornado. The hospital had implemented electronic health records (EHRs) six months prior. It also had in place data management policies and an employee education program that included disaster recovery procedures. As a result, the hospital was prepared to engage sister facilities in the area to help treat local patients, including emergency surgeries, within 48 hours of the disaster. For these patients, disaster preparedness made a difference that can’t be measured.
Pete Lamson, SVP,
The DR solution lies in the cloud
One of the most important components of any disaster preparedness plan is determining how you will protect and recover your business data in the event of an incident. A recent survey conducted by Carbonite found that 68 percent of small medical offices (less than 20 employees) back up their own data, but many smaller practices are actually using back-up methods that leave business data vulnerable to data loss, including external hard drives (45 percent), USB and flash drives (36 percent) and CDs or DVDs (29 percent). Manually backing up with these devices is time consuming and cumbersome, and you run the risk of device failure, theft or having your backup hit by the same disaster as the original data.
Medical practices can be relieved of these burdens by protecting their data in the cloud. It sounds complex, but cloud back-up solutions do the work automatically in the background, plus they are secure and affordable. Many providers already use cloud computing in the form of electronic medical record (EMR) solutions that store patient data, but there is a need for HIPAA-compatible and secure backup for other types of files, including documents, financial and accounting records, and email. Look for a service that transmits your protected files offsite to secure servers to ensure they are safe from just about anything that might happen in your office.
Once your data protection strategy is set, add it to your written disaster preparedness plan so you can share it with all employees and key external contacts. Then, it’s time to practice. While it may generate some eye rolling, companies that walk through a disaster simulation have a far greater chance of successful recovery.
Move DR to the cloud: Cut costs, improve workflow
As healthcare providers trim costs and redundancy in healthcare IT systems, they are considering an attractive new option: moving DR to the cloud. A pay-per-use model is much less costly than redundant data centers and allows for storage of patient records, lab results, medical images and other types of fixed content. The need for DR also creates incentives to use cloud-based, vendor-neutral archiving (VNA) for DICOM (digital imaging and communications in medicine) and non-DICOM data.
Flexible cloud-archiving platforms allow healthcare facilities to consolidate storage silos and improve workflow management of radiology, cardiology and other departmental data storage systems. Vendor-neutral archiving can also expedite PACS (picture archiving and communication system) upgrades, since information can be accessed at the metadata level – negating the need for a costly, time-consuming migration of legacy archives. Advanced VNA solutions feature tag morphing, which allows images to be displayed in virtually any PACS solution – allowing easy access across the enterprise with no additional investment.
DR has traditionally been a cumbersome and costly necessity, but new technologies allow it to serve as a stepping stone to more efficient archiving and data sharing for multiple departmental systems.
Data loss in the healthcare industry
CDW’s Data Loss Straw Poll surveyed 151 healthcare IT professionals to determine their most prevalent security concerns. All respondents surveyed were familiar with their organization’s IT data security strategies and systems, and they uniformly identified data loss as their top cyber-security concern.
Of those surveyed, 26 percent noted that their organization had experienced data loss in the past two years. Other security concerns cited as most worrisome included viruses, worms and breaches, as well as malicious attacks and mobile threats. When asked about the types of information targeted most often, 63 percent noted that employee or patient records and other personally identifiable information were the most likely targets of a cyber attack.
Security concerns are growing as the number of people accessing healthcare organizational networks increases. CDW’s report found that the number of individuals accessing these networks increased by an average of 52 percent last year, due to a growing number of office locations and mobile device deployments.
Mobility and the proliferation of mobile devices add another layer to security concerns. More than half of healthcare IT professionals surveyed by CDW stated that both employer-owned and employee-owned devices access their network. Despite best intentions and bring-your-own-device (BYOD) policies, 45 percent of respondents said their organization’s policies for employee-owned devices were less strict than those for employer-owned devices, and 15 percent noted they did not have policies for employee-owned devices.
For more information on CDW’s Data Loss Straw Poll, visit www.cdw.com/datalosspoll.
Brian J. Escott, P.E., director, project management,
Emerson Network Power’s ASCO Business
Examine power back-up plans before disaster strikes
Technology adoption often requires a re-evaluation of business continuity systems to ensure uninterrupted service. We continue to see healthcare facilities that are up to code but could not maintain service during extended power outages.
In such an event, just about every hospital system becomes business critical. Exploring what-if scenarios, such as a breaker failing to open or a generator failing to start, can improve contingency planning, while testing and monitoring reduces the chances systems won’t perform as expected. Remote monitoring provides the visibility to enable a more proactive approach to system maintenance and can be used by facility teams to test systems during off hours. Going above the NFPA/Joint Commission minimums in contingency planning will help ensure you’re ready for anything.
Many hospital data centers consume more power than may have been projected just two years ago, with the influx of patient and imaging data. Back-up power must be scalable to keep pace with growth, while maintaining protection to all essential systems. Knowing what facility applications are aligned to which back-up power system helps you prioritize if you must shed loads during a prolonged outage. And back-up power systems protecting business-critical systems should employ some degree of redundancy so that the failure of one back-up unit does not bring down the protected system.
Finally, usage of cloud-based services for data storage and application delivery means that their business continuity plans are your business continuity plans. Ensure that you have partnered with a provider that has the high-availability infrastructure that you require.
The solution is automated DR
A healthcare organization’s data and IT services are the backbone of the organization. For example, most doctors use laptops for electronic files; imagine the chaos that would ensue if that data was not accessible. Therefore, healthcare IT professionals need to ensure that their organizations have continuous, uninterrupted access to data, IT services and applications.
DR is important to healthcare IT professionals because an incident can strike at any time, whether it is caused by a natural disaster, human error or malicious attack. And when that disaster occurs, healthcare organizations need to be prepared. If not, they can be stuck wasting hours of time conducting complex manual recovery processes. Just a few hours of downtime can put patients’ health and safety at risk, decrease the healthcare organization’s productivity, cost the organization approximately $640,000 per hour and damage its reputation.
This is why healthcare organizations are turning to technologies such as continuous data protection, replication, de-duplication and automated disaster recovery solutions to build IT resiliency and diminish downtime. Continuous data-protection technologies allow companies to take snapshots of data within the data center, back up this data more often and replicate it to offsite data centers, cutting data loss to almost none. Automated DR combined with continuous data protection removes and mechanizes the detailed and time-consuming manual recovery process. Automated DR solutions also allow healthcare organizations to test their recovery processes before issues occur. This testing capability allows healthcare IT professionals to sleep soundly at night, knowing their data is protected and that it can be recovered with the click of a button.
David Kidd, director of quality assurance and compliance,
Utilizing the cloud for DR, compliance
The HIPAA Security Rule requires that healthcare companies establish and implement DR plans to restore lost data in the event of a disaster. Many companies are now turning to the cloud as a flexible, cost-effective way to implement DR plans while remaining compliant with federal regulations.
There are numerous benefits to using the cloud for DR. Depending on the technology employed in the cloud solution, it is possible for companies to receive near real-time data protection, versus a scheduled replication that happens a few times a day. This can result in less data loss in the event of a failure. It is also possible to replicate customers’ cloud servers, no matter where they are housed, and ensure rapid recovery times in the event of a disaster.
Choosing the right cloud partner is essential to successfully implementing a DR plan. The provider should be able to customize recovery point objectives and recovery time objectives to ensure that failovers can be initiated within minutes, and that companies can select the point in time at which a test or failover occurs. This essentially means that data can be restored to a past date or time, just prior to the disaster.
It is also helpful for the cloud provider to have disaster recovery specialists available 24 hours a day, 365 days a year, to help execute testing and actual recovery processes. Most important is to work with a provider that is HIPAA compliant. Compliance helps to mitigate many of the security concerns that healthcare companies have where cloud computing is concerned.
CEO, Spectra Logic
You can count on disaster, so commit to a DR plan
DR has not been high on the list of healthcare IT priorities until pretty recently. Press from hospitals hit hard by natural disasters – Hurricane Sandy and some powerful tornados, to mention a few – remind healthcare executives how important DR planning is. Specifically, executives need to know how to recover if a catastrophe strikes a facility’s IT infrastructure.
Also driving the need for comprehensive DR planning is the increasingly integrated use of technology in medicine, perhaps best exemplified by EHRs. Technology is now tightly integrated with, and essential to, patient care. That means that any disruption in the technology has immediate and dramatic consequences.
The eventual goal of a healthcare DR plan is to have a back-up system ready to take over – in seconds, if not milliseconds. Typically, a redundant system should be located far from the primary data center – hundreds of miles away, if possible. For many facilities, a private cloud may best address requirements, especially given the importance of HIPAA-mandated patient information privacy. In this context, private cloud refers to a privately maintained, well-equipped data center.
Incremental steps toward such a goal include using contemporary digital tape (such as LTO, or linear tape-open) to store the enormous data archives used in healthcare: PACS, EHRs and large data sets, such as those for ophthalmology and pathology, with tape copies at both the primary and the cloud sites. Steps such as these support the ultimate goal of a fully redundant site.
The remedy to DR: It’s all about planning
A common theme in the world of data protection is DR. When it comes to protecting your data, it is important to understand that DR is all about planning. This issue is the same if you are a mom-and-pop shop, school district, multi-national bank or major healthcare provider. According to IBM, in 2011, of the companies that had a major loss of business data, 43 percent never reopened, 51 percent closed within two years and only 6 percent will survive long-term.
Data requirements for healthcare providers are well spelled out via government regulations; however, these regulations only provide a framework of how to treat the data. DR is 99 percent planning and 1 percent execution. If a healthcare provider, or any other business, is considering DR, forethought must be 20/20.
Let’s be honest. In a disaster, few people care about what happened to cause the data loss; they only care that the data is recovered quickly.
The root of DR is that data is kept in a secondary site, and plans are made on how that data will be recovered so that the business can access it again. One item to note is that the data is not accessible during the disaster. It must first be recovered, and the speed at which the data is recovered is solely dependent on the planning, infrastructure and processes that are set forth and tested.
Recovery of data may well be the only issue that the bulk of IT managers and C-level officers have time to address. It’s a good start, but it’s not the whole story. You must understand the “what” and “how” in order to get your data back in operation. During that recovery planning, these same managers and officers will run into continuity questions. A quality recovery system (in terms of reliability, scope and scalability) greatly improves the chances of solving the continuity issues. The more feeble and antiquated your recovery planning, the more certain a corporate failure may ensue while trying to continue business as usual during a disaster.
Paul Luehr, managing director and chief privacy officer,
Best practices for a healthcare data breach: What you don’t know will cost you
Hospitals and healthcare organizations should consider DR and data-protection measures that are both practical and cost effective. Security needs to be of the highest priority within the entire company, not just within IT. Good security often depends more on people than on machines, so sound governance, training and daily habits are often the best bulwark against disaster.
When protecting data from a natural disaster, precautions used for everyday prevention (e.g., saving backups, rolling to alternative locations) can serve as good roadmaps in most scenarios. If a hard drive has been compromised by nature or intentionally damaged, a computer forensics shop often can recover the data if the drive still spins. If the drive is no longer functioning at all, it may need to be submitted to a “clean room” where techs can harvest uncorrupted data manually.
The first 72 hours following a data breach are critical to the outcome. An immediate and flawless investigation is most important; one early misstep can destroy crucial evidence, delay an effective response and trigger government penalties or class-action lawsuits. There are three key steps to follow in the days after a breach:
- Preserve data and digital evidence. Secure the premises and take an inventory of missing items. Do not investigate any machines without the help of digital forensic experts; any intervening by IT or business managers can alter or overwrite important dates or data points that are key to determining how a company was hit and when. Beyond servers and hard drives, save full log files and recent backups in order to preserve the best evidence possible for the breach investigation.
- Identify the compromised data. Coordinate with IT, HR, legal and forensic experts to interview key custodians and analyze pertinent data. Determine what data was taken and how it was taken. Identify the consequential risks. If data is missing altogether, turn to the backups that you just saved in Step 1.
- Communicate and track progress. Document your work at all times and take note of conversations with law enforcement and pertinent individuals. Provide clear and frequent updates to your data-breach response team, C-level executives, regulators, employees, shareholders, patients and/or customers. Set realistic time expectations about an investigation, as it can often take several weeks.
Laurie Elliott, North America director of premier accounts,
SunGard Availability Services
DR: Critical for healthcare
Large-scale disasters, such as Hurricane Sandy, can strike anywhere. This explains why healthcare IT managers are increasing their focus on DR planning. The risk is too great to ignore.
Patients’ lives can depend on systems being up and running. Reflecting this, federal HIPAA regulations require healthcare organizations to maintain up-to-date DR plans for most situations. The plans must detail how the provider will protect and restore access to electronic data during and after unforeseen circumstances. It’s also vital that such plans include how to recover from disruption of a provider’s health information system.
Much DR planning comes before an adversity. A healthcare organization’s information should be at a remote and secure location. The organization should determine the best method to transport data to the remote site. It should consider security factors such as data encryption as well as recovery time factors including data transport technology (e.g., tape, disk-based backup and replication).
People, processes and programs also must be reflected. IT staff with the expertise to develop and execute recovery procedures and plans must have the resources to scale with the healthcare provider. It also must have remote locations available to recover systems outside the danger zone.
DR is an ongoing process in IT. Because of the staff and expertise required for such a program, most healthcare providers rely on third-party-managed hosting providers to facilitate and execute the need. Their goal isn’t just to develop a recovery plan and procedure. It’s to employ a change program that, through regular testing, delivers process improvement to make DR more cost effective.
Make sure the right DR solution is in place. Then, when the next unexpected outage occurs – and it will – patients won’t be put at risk.
DRaaS: A viable option for organizations of all sizes
As the healthcare industry continues to adopt EHRs, the importance of having a proper DR plan in place has increased dramatically. A DR plan is essential in protecting critical data and ensuring business continuity for any business, but especially for healthcare organizations.
In the past, hosted redundancy and recovery services were often reserved strictly for enterprise organizations, but fully managed DR as a service (DRaaS) allows healthcare organizations of all sizes – from a large hospital network to a small private practice – to find viable and affordable recovery options. DRaaS is a scalable solution that grows with the needs of an organization, and it is easily customized to meet the highly regulated compliance requirements that come with sensitive EHR data and HIPAA privacy laws.
Storing data in the cloud relieves IT or other personnel of the burden of physically transporting backups, which begins the DR process by manually rebuilding servers and loading back-up files. Instead, the data and applications are stored and mirrored offsite, and server recovery is managed by the DRaaS service provider. Because of the cloud’s efficiency, a hospital or other healthcare facility is able to recover data in a matter of a few hours, not days. DRaaS allows healthcare providers to focus on more immediate issues affecting day-to-day operations, leaving the safety of their data in the hands of qualified and certified professionals.
Kevin Crowe, IT infrastructure manager,
University of Louisville Physicians
Virtualized replication cures DR ills
The University of Louisville (UoL) Physicians group is the largest multi-specialty physician practice in Louisville, Ky., with 78 sub-specialties, 1,500 staff professionals and more than 600 primary care and specialty physicians. UoL Physicians treats patients throughout Kentucky, southern Indiana and across the region. We rely on a mix of crucial applications, including AllScripts EHRs, GE Healthcare Centricity Business and Impact document imaging. Our data center houses more than 36 TB of data.
Keeping these applications running in our heavily virtualized environment is critical. An application or system outage means that our doctors, nurses and other healthcare professionals may not be able to provide care for patients. With patient care and core business functions on the line, our DR technology must be robust and reliable. We tried two well-known, top-tier replication solutions, but neither provided the performance or ease of manageability we require for our environment.
Instead of looking for DR software that replicated storage, we implemented hypervisor-based Zerto Virtual Replication. By replicating our 100+ virtual machines rather than storage volumes, we are protecting our mission-critical applications, instead of just pure data. We now experience an RPO (recovery point objective) of four to six seconds and an RTO (recovery time objective) of less than five minutes, which minimizes the impact to our operations in case of an outage or data loss.
Implementing replication at the hypervisor level was definitely a new approach for us. Traditional replication occurs in the storage infrastructure, but now our DR is part of our virtualization strategy.