Managing variation, uncertainty and risk

By Dr Ian Cox, JMP Marketing Manager

The FDA believes we badly need “a maximally efficient, agile, flexible pharmaceutical manufacturing sector that reliably produces high-quality drug products without extensive regulatory oversight”. What could or should we each do to help the desired state more likely?

As has been well publicised, the FDA believes we badly need “a maximally efficient, agile, flexible pharmaceutical manufacturing sector that reliably produces high-quality drug products without extensive regulatory oversight” [1]. If you need to be convinced this would be a good thing, just invert the meaning of each word or phrase to obtain “a minimally efficient, uncoordinated, rigid pharmaceutical manufacturing . . . with excessive regulatory oversight”. One of W. E. Deming’s points for management deals with the need to “eliminate slogans and exhortations” [2], and it might be tempting to accuse the FDA of falling into exactly this trap. Although this would certainly be a gross oversimplification and an injustice, the fact remains that there is a world of difference between knowing and doing (see, for instance, [3]), and also that we tend to be measured and rewarded, individually and collectively, by what we do (and the consequences this has) rather than simply for what we know. So, what could or should we each do to help the desired state more likely?

A recent Cooperative Research and Development Agreement (CRADA) conducted by Conformia Software Inc. makes for an interesting read [4], since it aims to pinpoint specific roadblocks that prevent us from reaching the desired state. Taken overall, the biggest roadblocks were identified as falling within two out of four areas (“Quality by Design Process Capabilities” and “Knowledge Management System Capabilities”, rather than “Awareness” and “Organizational Capabilities”). In the interests of space, I will focus here on just the “Quality By Design Process Capabilities”, within which two out of four sub-areas were identified as providing the largest room for improvement overall (“Process Understanding” and “Continual Improvement” rather than “Formal Development Roadmap” and “Control Strategy”).

In any situation and in any setting, acquiring new process understanding and making improvements continually is hard – But, whatever point of view you care to take, whichever quality guru you follow, or whatever improvement paradigm you work within, it’s fair to say that you cannot realistically expect to make improvements without understanding, and also that, somewhere along the line, data resulting from real-world measurements have a crucial role to play. This brings us directly to the main theme of this article, and at least one response to the question posed at the end of the opening paragraph.

Leveraging data to acquire new process understanding has many facets, including but not limited to, conceptual, technical, cultural and organizational. As we will also see, these different aspects are also somewhat interdependent. To deal with the organizational aspect first, there is often a division of labour between analysts on the one hand and scientists, engineers and technicians on the other. Unfortunately, this division is at odds with the obvious point that all data is contextual, so that the chances of discovering relevant and important features in the data are much reduced if the person interacting with it does not “know” the data intimately. This brings us to a manifestation of a cultural influence (or rather, the lack of it), namely the failure of the statistical community to adequately communicate the relevance and power of what is usually called “statistical thinking” [5] for real world applications.

Generally, scientists, although numerate, can have less appreciation of the importance of variation since the systems they have studied have a restricted domain within which it is not crucial. Saying it differently, they are sometimes at risk of having precise answers, but to questions that may not be of practical importance in an operational setting. This raises a further issue, namely that, unlike in science per se, the acquisition of new process understanding is not necessarily intrinsically good – It is only good to the point that it can be used to make valuable operational improvements. The judgement of what is valuable or not is exactly that - A judgement in relation to a shared set of values and commitments that express how we want to measure and run our business [6].

The technical issues relating to leveraging data can be broken down into two areas: Data availability and data analysis. Data availability features prominently in the Conformia CRADA under “Knowledge Management System Capabilities”. In essence, once the right things are being measured, the resulting data must be organized and persisted in structures that can support making discoveries, thereby sustaining improvement (or with the negative connotation, sustaining problem solving and root cause identification). So a key requirement is a flexible and extensible data model that can respect the structure of products, their components and raw materials, and which also captures the conditions under which processing occurs (including the state of equipment at that time). The topic of data analysis brings us back to another piece of unfortunate history, namely that many of the approaches to data analysis place an undue emphasis on hypothesis testing at the expense of hypothesis generation. Of course it is the latter that is most closely tied to making new, useful discoveries using data. This difference is particularly important in an operational landscape that is becoming more multi-faceted (“highly dimensional”) and complex (“nonlinear”) rather than one dimensional and simple. Analysis software should be designed with hypothesis generation in mind, and have a comprehensive repertoire of hypothesis testing and modeling techniques available for when the answer is not obvious [7].

So, returning to the question posed above, what could or should we each do to make the desired state more likely, at least with regard to the use of data? In some ways, the answer is simple and depends on your role. If you are someone who deals with data directly, try to overcome any phobias you may have about exploring data prior to formal modeling, and do this with a proper understanding of the data context. Data is, and always will be, your most valuable asset. If you are a consumer rather than a producer, when confronted with “the” answer, simply ask what is the associated uncertainty, and assess the practical implications of being at one extreme or the other. As far as practical application goes, the only safe statement about a single number is that it is guaranteed to be wrong!

References:
[1] Janet Wookcock (Director CDER), 2008 CMC Conference.
[2] http://en.wikipedia.org/wiki/W._Edwards_Deming.
[3] The Horizontal Organization: What the Organization of the Future Actually Looks Like and How it Delivers Value to Customers, Frank Ostroff (Oxford University Press, 1999).
[4] http://www.conformia.com.
[5] http://srtl.stat.auckland.ac.nz.
[6] The Process-Centered Enterprise: The Power of Commitments, Gabriel Pall (CRC Press, 1999).
[7] http://www.jmp.com.