Good Metrics Practices--the Other GMPs for Finished Pharmaceuticals

The adage “You can't manage what you don't measure” certainly applies to the Quality Management System (QMS). Keeping a watchful eye on QMS performance metrics is essential for any compliance sustainability strategy.

Performance metrics enable the feedback necessary for a self-detecting and self-adjusting—thus sustainable—QMS. Without performance metrics there is no data-driven knowledge of whether the operation is in a state of control. Metrics also offer a company hope to get out of the "firefighting" mode and  become an anticipating organization.

Other than the seriously “lagging” annual product review required by 21CFR211.180e for pharmaceuticals and the mention of “quality data” in 21CFR820.20 for medical devices, CGMPs offer little direction on effective QMS performance measures.

The International Conference on Harmonization (ICH) Pharmaceutical Quality System Q10 offers a bit more on what to measure (complaints, deviations, CAPA, changes and inspection observations). However, all these are measures of failure. Quality Assurance needs to go beyond measuring failure to maximize value to the organization.

What is measured drives the organization’s conversation and behavior.

What is measured, analyzed, reviewed with management naturally affects decisions, allocation of resources, capital investment and department expense budgets.

I have sat though many management reviews of QMS performance metrics ranging from mind numbing detail to high-level, sketchy dashboards. They’re boring, and from what I can tell, nobody wants to attend. 


Some company reviews are all-day events without a single decision or action item that truly impacts the organization. For some, by their own admission, they simply fulfill a regulatory obligation.

I wonder if a management review of region sales performance is such a perfunctory exercise.

QMS performance metrics are meant to provide important answers. But in many instances the questions are forgotten. Reviews often degrade into a charade of color-coded dribble and “happy speak.” It is a fearful experience for the presenters, and the net effect is the perpetuation of the emperor's nakedness.

Put aside popular metric “buzzwords” for a moment, and think about the questions that need to be asked to manage each element of the QMS. For example:

·       How do I know that the system is operating well and being well managed? Are there bottlenecks or lack of execution?

·       What will help us anticipate a potential problem before it becomes a real, more serious and more costly problem?

·       What problem(s) is the system detecting? Are they indicative of “one-off” issues, or is there evidence of a systemic failure? What should we escalate to senior management?

·       What are the few, high-leverage actions that will have maximum effect and reduce exposure? What issues need to be incorporated into the short and long-term business plan?

After the right questions have been asked, then consider the appropriate sources of data and how best to present them. Remember these principles:

·       Define where the data originate and how calculations are made. This helps to make a fair comparison within and across company sites. (Keep your sister sites honest.)

·       Metrics need to be time related with “event markers” in order to be comparative, i.e., before versus after an improvement is implemented. Are we getting better?

·       Metrics have direction and scale. Pick either up or down to signify getting better and stick to it, and don’t play visual tricks by altering the scale.

·       Measure the opportunity, not the success. Ninety percent “Right First Time” sounds really upbeat, but the ten percent “Wrong First Time” is the opportunity.

·       Accompany metrics with analysis. Don’t publish colorful charts and graphs without interpretation and discussion. “Red, Yellow, Green” alone does not constitute meaningful analysis. Be sure to answer the question: “So what?”

·      Arm yourself with proposed actions and cross-functional allies before proposing recommendations to management. Make the job of management so easy that the right decision is obvious. (Most VPs would rather you do the lobbying.)

Giving your QMS performance metrics purpose and discipline adds value to the organization.  These are Good Metrics Practices—the other GMPs.

The QA Pharm