Modernizing Legacy Systems

Stricter enforcement of government regulations means manufacturers must remediate older control systems. Here's some help.


Companies Mentioned
Posted on Nov 03, 2006

Jon Voss is an authority on Good Manufacturing Practices. Since the Food and Drug Administration has been enforcing regulations around production systems-which has been for about the last decade-Voss has accumulated years of experience as a consultant making sure process automation platforms meet government-imposed quality standards. He's transformed so many systems throughout his career that he's devised a methodology for achieving regulatory compliance. Voss, who joined Genzyme Corp. (Cambridge, MA) in December as the director of quality assurance, is currently applying that methodology to the manufacturing operations at a new biopharmaceutical facility in Geel, Belgium, which he is overseeing. Every system that is installed must meet government regulations. It's a daunting task, but in this case Voss gets to start from scratch when it comes to deploying the control technology-and that puts him in an enviable position. That's because he can build a system from the ground up that meets the stringent rules the Food and Drug Administration (FDA) is placing upon biotechnology companies like Genzyme, as well as pharmaceutical and biochemical companies, medical device manufacturers, and the food and beverage industry. These rules continue to be revised and updated. One addendum was the FDA ruling 21 CFR Part 11, issued in 1997. This legislation places additional burdens on manufacturers to properly maintain the electronic records of the process automation systems used to make products. This addresses everything from storage to change management to audit trails. "When the Part 11 ruling was handed down, everyone hoped it would go away," says Voss. That was followed by a consensus that it would do just that. "It was too expensive to deal with all of the existing legacy process automation systems; people thought they could just focus on new systems. But then the FDA started issuing observations against legacy systems from the automation standpoint." Translation: Major remediation. Essentially, manufacturers are faced with the chore of having to validate that their process automation systems-which are becoming hybrid systems made up of PLCs, SCADA, DCS, single-loop controllers, safety instrumentation, and plant information systems-are well documented. The documentation serves as proof that the systems, in fact, do what they are supposed to do, that production quality never waivers, that every procedure can be tracked (Part 11 specifies electronic tracking), and that the process is protected from unauthorized changes. The records that fall under the umbrella of needing attention include the recipe itself, batch reports, process trend data, user profiles, critical alarms, and events critical to quality. Vendors are responding with manufacturing execution systems (MES) and batch control applications that include historical tracking and electronic signatures to aid customers in their endeavors (see "Applications Aid the Compliance Effort," page 24). But applications alone can't solve the problem. Remediating legacy equipment (that is, production systems that were in place prior to government policing) is a resource-intensive job that requires time, money, and a dedicated team of experts who can implement a foundation for Good Manufacturing Practices that are well-documented and validated. It's an issue not to be ignored. Those companies that have not complied with the FDA's current Good Manufacturing Practices (cGMP) regulations have been slapped with large fines. Take note of Abbott Laboratories (Abbott Park, IL), which, in November 1999, signed a consent decree of permanent injunction with the FDA, agreeing to stop manufacturing and distributing many of its in-vitro diagnostic tests until it corrected deficiencies in two of its manufacturing facilities. The firm had to pay $100 million to the U.S. Treasury within 10 days after the decree was entered by the court, plus pay penalties of $15,000 per manufacturing process, per day, for failure to adhere to an FDA-approved schedule to bring processes into regulatory compliance. That was the largest fine of its time until May 2002, when Schering-Plough Corp. (Kenilworth, NJ) was issued a $500 million fine resulting from inspections of four facilities that revealed "significant violations of the cGMP regulations related to facilities, manufacturing, quality assurance, equipment, laboratories, and packaging and labeling," according to the FDA-issued statement. "The consent decree is another term the FDA uses to say [a manufacturer is] in such gross violation of good manufacturing practices that they are concerned about the safety of the product made," Voss explains. "And if the product can't be taken off the market, the FDA has to approve each release." Having the government say it doesn't trust the company causes serious harm-and not just from a financial standpoint. The psychological impact on the company's employees and shareholders is palpable, as well. "It's the worst thing to happen to a drug company, other than the FDA padlocking the doors-which they've done," says Voss. The dilemma most manufacturers have now is deciding what to do with old systems because upgrading legacy equipment is like trying to hit a moving target at this point. "These are not stand-alone systems," explains Lou Pillai, team leader for process automation and MES at Pfizer Inc. (New York, NY) and a member of the World Batch Forum (WBF, St. Louis, MO) board of directors. "They are networked to other systemsand they are unique in that they are transaction-based. Real-time control is involved." That means taking equipment down for any length of time is out of the question because it would directly impact production. Instead, manufacturers must consider three options when it comes to updating legacy systems. They can replace old systems, which is a costly solution; they can reinstall systems, which means reinstalling the old system software as if it were a brand-new system (and that requires having access to all existing code and operating status, which may have evolved over several years); or they may remediate, which means retroactively constructing a validation package for a system operating now. Remediation is the most practical solution and therefore the most common method of gaining cGMP compliance. That's where Voss and Pillai step in with a proven methodology that can alleviate some of the risk associated with a remediation project. Recently, the two presented a series of educational Web seminars through the World Batch Forum that highlighted a "best practices" approach. "Like any other project, remediation scope-creep is a problem," says Voss. "If you don't manage it properly it can get out of control and then remediation becomes a full-scale validation." To reduce the scope of remediation, Voss and Pillai offer the following tips: ·Take inventory of all the available information. "This process takes time and is harder than you might expect," notes Voss. It is a teasing out exercise that requires collecting historical files, user manuals, user requirements, specifications, standard operating procedures, change history information, and calibration records. The key to this step is engaging the right person to get the information. "The right people are generally the ones who support these systems," says Pillai. ·Focus on "critical functions," or those functions that a user requires in order to have a system perform to the cGMP. Devising a spreadsheet that includes the system, a definition of its critical function, and a space for notes will keep the process focused. "If you are dealing with multiple sites where every person has a vastly different perspective on what the critical functions of a system are, having a spreadsheet that is accessible via intranet or e-mail reduces the amount of time it takes to come to an agreement on these things," says Voss. Also, define the value of each attribute of the critical functions, he says. ·Complete a gap analysis-the process of determining what critical functions are missing. A spreadsheet or a flowchart are effective tools to break down inventory items and identify what is important to the remediation process. ·Perform remediation (carefully). Once gaps are identified, you can go back and fix the holes. Make sure a quality system is in place first. "If you go through the effort of remediation and don't have a quality system, especially for change control, you may as well throw away the money spent on remediation," says Voss. That's because even after the remediation is done, the system is constantly in a state of change. Everything must be documented. ·Test the system after remediation. Testing includes comparing the critical function list with a validation document. "Writing the test is the simple part," says Pillai. It occupies less than 10% of the overall remediation process because it is a straight-forward activity. Compile a trace matrix-a tree structure-with columns for user requirements, specifications, and tests that verify that the spec and the user requirements were achieved to serve as proof that the system works. The FDA will ask for a testing document and what the specifications were for developing the test," explains Voss. "The trace matrix ensures that for every test there is a specification, and for every spec there is a user requirement." Voss and Pillai base their suggestions on real-life experiences. "I've made the mistake of spending lots of money and resources on testing a system to death, only to find out, when the test failed and I went back to the owner, that it was not a critical function," says Voss. "Then I'm forced to come up with a justification of why it is not important." This is the kind of hiccup that can impede progress and waste resources. Bear in mind, remediation does not happen overnight. Depending upon the number of systems requiring attention and the risk factors identified, it can take from months to years just to get old systems up to the new standards. And then you're still not done. "Compliance is a full-time job," says Voss, because technology innovations keep raising the bar, which means upgrades are in a constant state of flux. All-in-all, quality projects can eat up 20% to 40% of the overall corporate budget, says Voss. But following some simple rules may keep the project from snowballing-and it will definitely keep the FDA off your back. MA

Top Enterprise Software Planning (ERP) Comparison