From the October 2005 Issue

Better Business, More Money

Deals on Wheels: Case History

Five Trends Impacting Hospitals’ Business Processes and Financial Health

An Ideal Disc Storage Solution: Case History

Managing the Business of Surgery: What Works





An Ideal Disc Storage Solution

When there’s a will, there’s always a way for a dedicated IT staff to keep data flowing, protected and accessible.

 By Mike McBride, Associate Editor

Today’s electronic, image-driven, Web-based medical industry handles enormous amounts of information. EMRs, EHRs and EFRs (electronic medical, health and financial records), PACS (picture archiving and communications systems) and DICOMs (digital image and communications) are now realistic methods of medical information exchange and may soon be required for HIPAA compliance. With smooth IT operations depending on massive IT infrastructures, IT managers must constantly evaluate and anticipate their companies’ present and future IT needs to ensure adequate network resources, such as disc (storage) space, to handle the load.

Database and network server management have evolved over the past decade to become more sophisticated, but storage management has lagged behind. Ten years ago, disc space was measured in gigabytes; five years ago it was terabytes; today, some IT departments need exabytes and even zettabytes (over 1,000 exabytes or 5 million million bytes) to store and secure medical information.

The physical footprint of such an array and the energy requirements to run it would present a challenge, however managing and guaranteeing accessibility to that information is beyond the means of many companies. It’s one thing to store it and quite another to provide 24/7 access to providers, physicians and patients.

Facing Challenges
Health Alliance Plan (HAP) of southeastern Michigan provides health coverage to more than 500,000 subscribers, making it one of the largest health insurers in that state. Its providers, members and employer groups must have 100 percent unfettered access to appropriate membership information. Maintaining HAP’s Unix infrastructure is the responsibility of Dan Trim, HAP’s manager of operations and technical services.

IT managers must constantly evaluate
and anticipate their company’s present and future IT needs to ensure adequate network resources, such as disc (storage) space,
exist to handle the load.

When Trim joined HAP in 1999, they had several small servers and approximately 3.5 terabytes of disc space in operation, which included the IT department’s desktop computers—not much disc space, according to Trim. Due to the provisioning process, it took up to 30 days for the IT department to add disc space when it was requested. To increase disc space to a single server, HAP technicians would purchase a disc array that met the payer’s requirements and physically install it on the server; however, the array was only available to the server on which it was installed. “If it was too much space, the rest couldn’t be used by the other servers. It was just wasted,” notes Trim.

At that time, HAP was switching over from proprietary software to packaged insurance software on an open-source Sun Solaris system. After reviewing the statistics, Trim discovered HAP’s servers were experiencing I/O (in/out) bandwidth utilization problems, resulting in system failures. More processors were needed to keep up with the demands that customers placed on the system. Consultants recommended Trim add eight more processors to handle the current load.

Then, HAP experienced database failures when the number of electronic claims exceeded the system’s available disc space. “When that happens, the system dies. It just crashes,” says Trim. His staff had to manually bring the system back online, which affected the payer’s business. For Trim, the bottom line was that the nonproductivity caused by system outages had to end.

Finding a Solution
According to Trim, system administrators spent too much time manually allocating, reallocating, monitoring and documenting each server’s disc usage, in addition to the time lost recovering from crashes—time that that could be better spent on strategic planning. “It was a reactive and inefficient system,” that needed streamlining, he says.

He looked at various SAN (storage area network) and NAS (network attached storage) products from several leading vendors and determined a SAN solution was best because “it kept traffic off of [HAP’s] network backbone.” Eventually, he installed a Veritas SAN hardware appliance, but there were problems. “It had all the redundancy and resilience that we needed, but there were some flaws in it that could not be overcome by Veritas,” says Trim. Two weeks after installation, Veritas ended the life of the SAN product, unable to resolve the technical flaws.

Faced with the task of starting from scratch, Trim’s staff architectured their own solution using the Veritas appliance. Says Trim, “Even though we knew we had major problems, you just can’t replace half a million dollars worth of hardware without budgeting for it.” The product worked well enough that HAP gained most of the efficiencies Trim wanted after increasing the I/O bandwidth to their servers. “We were able to pump about 600 percent more data to our servers with the same number of processors,” says Trim. “We didn’t have to get new computers, because our bottleneck wasn’t at the computer level or the processor level. It was at the storage level.” The short-term solution significantly extended the life of HAP’s network servers and reduced disc allocation crashes to nearly zero.

Eventually, though, Trim replaced the Veritas SAN appliance with an IBM Shark, onto which he installed 26 TB of disc space. The Shark had its own console, which allowed Trim’s staff to manage the disc space from a central location. “That’s a good solution if all you’re going to do is buy a Shark every time you need more disc space,” he notes.

The long-term goal, though, was a tiered storage infrastructure where the IT department could provision disc space according to actual need. Such a system uses high-performance discs to support OLTP (online transaction processing), where quick response time and batch processing of large amounts of information is an ongoing occurrence. The system would be required to support HAP’s claims processing, customer service, and membership and billing departments, all of which needed high-speed access to membership information.

Dan Trim
Health Alliance Plan

Trim chose Veritas Comm-andCentral Storage because it was reliable and would integrate with HAP’s existing systems, and also because central management could allocate disc space to the high-volume servers that needed it most—those that kept HAP’s business processes running. He felt the system was intuitive out of the box, so the learning curve would be short. The product also promised to provide equal or better activation to service time (the speed at which the software talks to the discs) and to ensure high availability and redundancy, so “disc space is no longer an issue with getting work done.” Since installing the product, HAP hasn’t had a single failure because of IT’s inability to provision for disc space.

Policy-based alerts enable Trim’s staff to set multiple threshold parameters on network activities and capacities and to receive warnings when they are exceeded. When a disc reaches a predetermined capacity, the product transmits an alert report to the IT department, which decides how and when to respond. Another feature is loader directories that are programmed to load data onto the servers each day on a scheduled basis. When the load completes, IT receives a warning message that the disc space threshold has been exceeded and by how much. When that threshold is consistently exceeded by a certain amount, Trim can project how much he will need to grow disc space for the next year. He uses the same feature to identify when an expected process doesn’t occur. When Trim doesn’t receive the expected warning message, he knows a process has failed and can investigate.

Trim also runs a single backup and recovery system for all of HAP’s Novell, Microsoft, Windows 2000 and Sun Solaris system servers. He receives daily reports on the status of the previous night’s backups and how many copies were created, if any. The reports also list copy locations and the status of any failures. Trim says functionalities like these are essential for effective storage management.

Looking Ahead
As the healthcare industry moves toward an Internet-based approach to information access, HAP is moving with it. Traditionally, most of the payer’s users were internal, but HAP now provides an Internet-based portal through which providers, employers and members can run applications and access information. External users can view online claims information and status, identify the payer and payees, and determine how much is paid on a claim. Eventually, members also may be able to make payments online.

The challenges that Trim continues to encounter with the IT infrastructure are no longer problems, “because I don’t have to worry about the disc space, where it’s at, who needs it and why it needs to be there,” he says. He also uses CommandCentral Storage to forecast project growth and build the disc space volumes once per year. “We re-layout a server with everything it will need for that year in place. For the most part, we never have to touch it again for that year.” He can now provision for nonarchived data growth, anticipated acquisitions and membership increases.

Trim turned a flawed, reactive system into a proactive, self-monitoring storage solution that enables him to anticipate HAP’s future needs and adapt the server environment accordingly with no interruption of service. Considering how fast the healthcare industry evolves, that flexibility may make all the difference.

Editor’s Note: As HMT went to press, VERITAS Software announced that it was merging with Symantec Corp. of Cupertino, Calif. creating one company under the Symantec name. The functionality, support and development of the products mentioned here remain the same. VERITAS and Symantec finalized their merger on July 2, 2005.

For more information about Symantec’s Veritas CommandCentral Storage,

© 2005 Nelson Publishing, Inc