So what’s the big deal about 3-sigma? Is it just 6-Sigma for under-achievers? Is it only for statistical geeks? Why should anyone give a hoot?
As with all powerful ideas, 3-sigma is simple, yet difficult to get your head around. Although there is no short explanation, here’s the shortest answer I can provide. It goes like this…
First of all, 3-sigma isn’t the stuff of slogans and marketing literature. There are no black-belts in 3-Sigma and it is not a magic pill for certain riches or true love. At its core, 3-sigma refers to a theory of knowledge that gives rise to a methodology for prediction. Once you get it, for whatever purposes, your ability to understand the present and foresee the future, can be dramatically improved.
Three sigma, or three “Standard Deviations”, is a statistical value that can be written using the greek letter “sigma”. It looks like this:
A Standard Deviation is a measuring stick used to describe how data are dispersed around their average.
For what is called a normal distribution, which takes the shape of a nice “bell curve, one Standard Deviation encompasses about 68% of all observation data. Two Standard Deviations includes about 95% of all observations. And three Standard Deviations encompasses a bit more than 99% of all observations. It actually doesn’t matter if the shape of a distribution is pretty, like the “bell curve” shown above. All that matters is that all measurements you make as an observer of the world, will vary, and understanding why measurements always vary and what that variation means, is the key to understanding how we know the world and how we make predictions about the future.
Three sigma stands out from 1, 2, 4, 5, 6 . . . sigma because it alone, represents the boundary point that Walter Shewhart determined can be used to signal the difference between events that are ordinary and predictable and those that are unusual and unpredictable.
At first blush, this idea of a statistical boundary between the ordinary and the extraordinary might seem rather silly and arbitrary. Our common sense tells us that we can easily discriminate between what is special and what is common in our daily experience without resorting to statistical reasoning. Why is this not the case?
Behind the concept of 3-sigma lies a theory of how we know the world. This theory asserts that we are genetically and culturally programmed to see assignable causes for every effect of interest to us. This way of knowing permits us to act upon the world. For every observation of interest, we seek out some button to push or some lever to actuate that will bend the world to our purposes. These buttons and levers are the theories we construct about how the world works.
Our minds tell us that for every problem we experience and every challenge we face, there is always a cause that can be acted upon. The Sun’s light and gravitation are causes. My neighbor’s barking dog is a cause. Temperature is a cause. The “economy” is a cause. God is a cause. Whistling in the wind is a cause. Things we observe and things we imagine can be designated as causes. We seek and see causes in all things that interest us. This is how we make predictions about the world and how we guide our actions in the world. We are prediction machines and button-pushers, par excellence.
But do our intuitions about assignable causes serve us well or do they misguide us in our predictions about the future?
In his book, “Statistical Method from the Viewpoint of Quality Control“, Shewhart shows us that most of what a system (or process) does is a product of interactions that cannot be reduced to one or more assignable causes. Shewhart’s un-assignable cause is also called, “common cause“. In a sense, he is telling us that we cannot identify a cause that is in the system because everything in the system is a cause. A system, he says, is irreducible. This means that we when we start pushing buttons and pulling levers for observations that are not clearly produced by assignable causes, we only make a system increasingly unpredictable . If we fail to understand the nature of variation we will more likely than not, make a royal mess of things.
Based on a theory of knowledge, Shewhart created a statistical tool called a “control chart“, for monitoring what a system or process is doing as a whole, over time, and will likely continue doing into the future, barring the introduction of some assignable cause (sometimes called “special cause”). Some people like to say that Shewhart’s control charts are a way to listen to the “voice of the process”. It is a variation detector that tells us what variation is in the nature of the system we are observing and what variation is “special”. When assignable causes do start influencing the system, it will leave it’s state of control and become unpredictable. He assigns 3-sigma as the signaling point for differentiating cause that can and cannot be assigned.
When an assignable cause is signaled, Shewhart tells us that we can search for that cause in order to remove it and restore the system to its former state of predictability, which is a very good thing, OR we can act to change the system as a whole in ways that make it even more predictable and more suited to our purposes, which is sometimes, an even better thing.
So where did Shewhart get the 3-sigma limit from? He got it from lots and lots of empirical observation. He says it has no “truth” to it. It is just a value that works to minimize the consequences of the mistakes our minds trick us into making. 3-sigma is a tipping point that minimizes the two mistakes we can make—confusing common cause with assignable cause OR confusing assignable cause with common cause. In his own words,
“…knowledge…is a method of approximating a practical ideal of a minimum number of false predictions.”
You can learn a lot more about theory and techniques for using control charts by visiting Don Wheeler’s SPC Press Website and reading his books, but you still have to ask yourself….
Why would Shewhart’s ideas have importance in our daily lives?
Well, the variation that is everywhere in our experience of the world is invisible to us because most of our behavior in day-to-day life is habitual and common-sensical. Most of us simply take for granted all of the predictions we make when we get out of bed, get dressed, and drive to work. To our way of knowing, most of the world is in a steady and predictable state. It is only when we are first learning our every-day behaviors that we are forced to consider everyday causes and effects. Do you remember when the simple act of tying your shoes required a theory and a flow chart?
Our intentional behavior on the other hand, is geared toward solving problems and attaining goals and in these activities we put a great deal of effort toward identifying causes. The way we see it, if we can just push the right buttons and pull the right levers, we can make the things we want to happen, happen. But, like tying our shoes, our analyses remain rooted in our common-sense beliefs about cause and this habitual way of thinking makes us assign causes that cannot be deduced from the system. In other words, there is no “right” button to push. When we make the mistake of assigning causes that are actually common, we do more than just make incorrect predictions and push the wrong buttons, we actually make the system itself, increasingly unpredictable.
So what can we learn from Walter?
Before you start pushing buttons and pulling levers, listen to the “voice of the process” to see what it is actually doing! If the variation it produces does not show control—is not predictable, you have a theory that is not very useful and you need to rethink that theory. If on the other hand, the voice tells you that the process is predictable within some limits, you can work on the process to increase its predictability AND you can hear assignable causes that threaten to upset your apple cart in order to get things back on track. This new approach to knowing can do more than make better widgets. It can help prevent events like global warming, save marriages, fix cars, cross oceans, and help us do better in every enterprise we chose to undertake.
Note: There are also ways to detect assignable cause signals (non-random events) within 3-sigma limits, but the principle of differentiating between assignable and common cause, remains the aim of this predictive methodology.)
Shewhart produced his control charts as a means for improving the manufacture of product, but in doing so he reached deep into a theory of knowledge. The implications of his discovery pertain to all human enterprise. Dr. W. Edwards Deming did much to draw out these implications for business enterprises, but even his profoundly important work did not fully explore just how important Shewhart’s ideas were to the success of the human enterprise as a whole.
Once you begin to understand the theory of knowledge that underlies Shewhart’s control charts, you begin to see how what you can know about the world is actually shaped by your aims and intentions, the observations you chose to make, and the measurement methods you have at your disposal. Benefiting from such an understanding does not require control charts or complicated statistical calculations. It is a way of seeing that fundamentally changes how you experience the world.
Walter Shewhart captured the essence of this transformation by quoting C. I. Lewis:
“…knowing begins and ends in experience, but it does not end in the experience in which it begins.”
Still confused? In various ways, all of the entries in this blog reflect my continuing efforts at understanding and exploring the implications of 3-sigma as theory and method. In future entries I will discuss in more specific terms how a theory of knowledge and methods based in that theory, can help us to do better in all of our enterprises. I hope you will drop in from time to time to see how I am doing.
Also see: Where Systems Come From