04.05.2015 |

LOGFILE No. 30/2015 – Quality Metrics - Bane or Boon?

Quality Metrics - Bane or Boon?

by Thomas Peither

It isn’t easy to set up appropriate quality metrics. What objectives are being pursued? What levels have been reached? What can you achieve with metrics? What facts can be mapped? Authorities are also requiring more and more objectivity through numbers. This article is a commentary on aspects of importance for implementation.

Many of the things you can count don’t count!

This quote from Albert Einstein made the rounds way back in the days of the metrology lectures I attended as a student. Today, one hundred years after Einstein uttered his famous words, we really do count and measure a lot everywhere. After all, there is nothing wrong about measuring. The masses of numbers aren’t bad either. Then why do so many people who hear this saying agree with it?

The problem is not the large number of measurements and results. The real problem lies in defining the measurements, performing them, then interpreting and communicating their results.

The FDA requires quality metrics

What are quality metrics to the FDA? In 2013, Russel Weslyk, FDA, described quality metrics as:

  • an objective measure of the quality of a product or process
  • an objective measure of the quality of a site
  • an objective measure of the effectiveness of systems associated with the manufacture of pharmaceutical products, including the pharmaceutical quality system

Thus, the objective measure forms the basis for all of the above. Why is that so?

The German physicist Hermann von Helmholtz wrote as early as 1879: "Counting and measuring provide the basis for the most fruitful, reliable and accurate of scientific methods." This theory has obviously proved itself and become established in industrial practice. However, is counting, measuring and objectivising everything not going too far? Is anything else unreasonable? Is measuring “objectively" all that counts? The next question that comes immediately to my mind: "What does objective mean?"

Definition of objectivity
Objectivity (from Latin obiectum, the past participle, passive voice of obicere: thing cast before or opposite) refers to the state of an evaluation or description of an object, an event or a fact that is independent of the observer or the subject. The possible existence of a neutral vantage point that enables objectivity is ruled out. Objectivity is an ideal of philosophy and science. As it is assumed that every point of view is subjective, scientifically useful results are measured against defined, recognised methods and standards of research. (1)

Figure 1: Definition of objectivity

Do we need metrics?

With metrics are we really seeking the truth, or are we not looking instead for proof of our trust in human beings and in processes?

What other explanation could be given if product batches are released even though measurable deviations from the requirements exist? Does the Qualified Person trust humans and processes more than numbers? Is it not often the main task of this Qualified Person, after the numbers become available, to provide proof that everything is in order even though the "objective" numbers say something different?

Or do we seriously scrutinise the numbers and attempt to find the underlying reasons, the causes for the irrationality of the numbers?

Perhaps you are familiar with the following joke:

Exam day in Physics A brick is lying on the heater. The student enters the room. The examiner asks: "Why is the brick warmer on the side facing away from the heater?" Student: “uhhhhh [stammering], possibly because of thermal conductivity or something like that?" Examiner: "No, because I just now turned it over." (2)

  • How many times do we read justifications in deviation reports such as “… Thermal conductivity or something like that?”
  • How many times in production do we ask if someone has moved the “brick”?
  • How many times do we speak with employees about their problems with processes and machines?
  • Do we not give “poorly trained” employees the blame far too often?
  • Do we try to interpret, scrutinise and evaluate the numbers correctly?
  • Do we have the theoretical and practical knowledge we need to interpret the figures adequately?
  • Do we have time to conduct an analysis that yields results?

It seems absurd to stop at quality metrics only. The FDA shares this view and calls at the same time for a culture of quality. A quality culture includes not only the attitude towards people, processes and numbers, but also how people, processes and numbers are treated.

However, the same criteria may not be applied to the observation of people, processes and numbers (cf. Figure 2). Also, it is often very difficult to describe the quality of people in terms of numbers.

People Processes / Machines   Numbers                         
Skills Robustness Definition
Knowledge Traceability Gathering
Behaviour Variability Analysis
Willingness/Enthusiasm Susceptibility to errors Evaluation
Reliability Handling Accuracy
... ... ...

Figure 2: Evaluation criteria for people, processes/machines and numbers (examples)

As much as I would like to follow Hermann von Helmholtz, it seems difficult for me to describe the work process with numbers alone. Qualitative performance aspects in employees can hardly be measured, particularly if we are to evaluate the employees’ conduct.

Everyone can observe in himself or herself that on a day-by-day basis we are not

  • always creative,
  • always communicative,
  • always capable of concentrating well,
  • always able to take everything into consideration,
  • always able to understand the other person.

In other words: We cannot perform consistently at the same level every day.

This is due to an endless number of influences that affect us emotionally and intellectually, such as

  • a traffic jam on the way to work
  • a rail strike
  • an ill child
  • an argument with the partner
  • parents in need of nursing care
  • private financial problems
  • stock/currency crash
  • sales problems in production an illness diagnosed in yourself.

Anyone wishing to capture these influences in numbers will run aground sooner or later. For this reason, performance must be judged in ways other than with numbers. It can happen that in crisis situations certain employees rise to peak performance, while others lose their footing. On the other hand, in quieter times the exact opposite can be true – which of the two employees is now the better or the higher performing one?

Therefore, please allow me to specify:

We may describe the performance capability of processes and machines with numbers as long as we are aware that the numbers are subject to error and must be interpreted. Contrary to this, the performance capability of people depends on such a boundless number of influences that it can hardly be described with figures.

Quality metrics must be carefully thought out

I agree with the statement made by Steven Mendivil at the PDA Annual Meeting 2015 (3):

It may be possible to recognise quality behaviour during an inspection. However, things become more difficult if we want to document quality behaviour in a manner that can be replicated.

The thoughts exchanged at the PDA Annual Meeting differ from those of the usual metrics presentations. Steven Mendivil discussed the problems that arise from defining, recording, analysing and evaluating metrics. The contents were so varied and profound that it would overstep the boundaries of this article to go into detail. However, I would like to repeat his closing thoughts here:

  • There is no such thing as perfect quality metrics; compromises must be accepted.
  • First of all, unintended consequences must be considered.
  • Quality culture metrics and/or mature metrics for quality attributes may exceed GMP requirements.
  • Data trending is far more valuable than a direct comparison.
  • Can globally applicable metrics be developed at all?

These statements get right to the point. Finding the proper responses to them will require a lot of work. Especially since the quality culture often differs greatly from site to site and from company to company. If we look at the cultural differences between Asia and the USA, Europe and Africa, Norway and Portugal, then we will have an idea of the Herculean task behind our wish to establish globally comparable quality metrics.

Possible steps for quality metrics

That is why it is important not to engage the total picture all at once, but rather to approach this task one step at a time. After all, you wouldn't eat a side of beef all at once, but rather bit by bit:

1st Step: Spend some time on quality metrics, find out how they are defined, determined and analysed and how they influence the organisation. Build a foundation based on literature, conferences, workshops and exchanges of opinion. Concentrate first of all on processes and machines.

2nd Step: Each individual contributes different experiences and basic principles to the discussion. It is important that these experiences and basic principles be made transparent before common conclusions are made in the company. Often enough, individual conclusions are made before a common understanding has been worked out – unfortunately, to the detriment of the organization.

3rd Step: To what purpose and for whom are the metrics intended? Different metrics will be necessary, depending on what they are intended for. There is a difference between whether I am expected to keep pace with an increasing market supply or if I am supposed to lower costs within the organization because there is less demand for the products. Explain the benefits of metrics to the employees.

4th Step: The proof of the pudding is in the eating. Try out the metrics. But be aware that they won't be perfect, but rather they are only a model or an approximation and they must always serve the purpose. Metrics should never be a purpose unto themselves.

5th Step: Modify metrics or delete them when they are no longer necessary. As human beings, our attention span is limited and we must concentrate on a limited number of things. Therefore, less is often more. Computers do have advantages – but they, too, serve only to assist us humans in meeting our responsibilities.

Therefore, we should not set up metrics because authorities require it. Metrics must serve the people within the company in order for them to achieve their goals. That will not work without metrics for processes and machines. However, metrics are not suitable for evaluating employees.

Therefore, very strongly differentiated descriptions are necessary to draw a total picture in terms of quality culture. Since such descriptions are generally not available to us nowadays, for the time being we must live with these inadequacies and uncertainties. A professional management masters these uncertainties, assumes responsibility and leads the employees so that they can fulfil their responsibilities.


Wesdyk Russell CDER/OSP, Quality Metrics, Why are we going... Where are we going..., 2013
Mendivil Steven, Quality Metrics? PDA’s Seeking Answers, PDA Annual Meeting 2015, Las Vegas 
(1) Wikipedia, https://de.wikipedia.org/wiki/Objektivität, called up on 26th June 2015
(2) http://www.deecee.de/funny-stuff/witze-jokes/akademiker-witze.html, invoked on 26 June 2015
(3) “It is more difficult to inspect quality behavior than GMP compliance.” Steven Mendivil, PDA Annual Meeting 2015, Las Vegas


Thomas Peither
Editor in Chief
Maas & Peither AG, Schopfheim, Germany
Email: thomas.peither@gmp-publishing.com