Good Metrics Practice for Quality Management Reviews

A Quality Management Review (QMR) of quality data with responsible company leadership is a CGMP requirement.  QMR practices vary, but there seems to be a struggle with presenting data from across the Quality Management System is a meaningful and consistent manner when there are multiple contributors. Here are a few suggestions:

Report the opportunity for improvement

Reporting the opportunity helps to focus where to improve. For example: Ten percent (10%) of investigations were overdue, rather that ninety percent (90%) were completed on time.

A decrease shows improvement

A downward trend means improvement toward zero problems. For example: Following root cause analysis training, recurring deviations were reduced from twenty percent (20%) to five percent (5%) in six months.

Compare versus historical performance

Comparing current performance versus previous period, or this time last year, helps to illustrate improvement over time. For example: This quarter, ninety-five percent (95%) of supplier audits conducted versus plan was accomplished compared to five percent (5%) previous quarter.

Index metrics for relative comparisons

Indexing eliminates the effect of arbitrary data sets and helps to make comparisons. For example, there were seven (7) Complaints per Billion Units Manufactured Year-to-Date versus eighteen (18) for the same period the previous year.

Report absolute numbers for critical issues

Indexing should be avoided when the issue is critical or numbers are low. For example, report that two (2) batches were recalled, rather than 0.2% batches were recalled.

Note events with markers on the timeline

When data is reported versus time, it is helpful to note significant events that had an effect on the data. For example, the trend line for environmental monitoring excursions started to increase when building construction started.

Define an unacceptable trend

Trends should be defined for run chart performance data. For example, consider statistical process control method of five (5) consecutive movements in the same direction, or seven (7) seven consecutive points on same side of average.

Report measure of variability with averages

When reporting averages, be certain that that data can be legitimately combined, and provide a measure of variability. For example, reporting an improvement with a decrease in the average number of seventeen (17) deviations per batch record for the last ten batches compared to an average of twenty-five (25) with the previous ten batches, is misleading when the range of deviations increased from five (5) to forty-five (45) compared to twenty-three (23) to twenty-eight (28).

Chart scales must be sensitive for intended purpose

The scale of a chart should sufficiently large to illustrate the range of normal variation, and small enough to include all excursions within the time frame depicted. For example, the chart scale for Percent Overdue Nonconformance Investigations of 0 to 100% is inappropriate for a 12-month performance chart with normal variation of 3-6%.   A more appropriate scale would be 0-12%. If the same time frame included an excursion of 18%, a chart scale of 0-20% would be appropriate.

And remember---Data talk; opinions walk.

The QA Pharm