When do you know you’ve tested enough?
When do you stop looking for a better price?
When do you stop looking for a better candidate to hire?
When do you stop researching other solutions?
When do you know you have enough data?
Analysis Paralysis
describes a situation when someone, because of concern about discovering something new, continues to look for and analyze information. They don’t know when to stop.
Here are three steps
to help you determine if you are done. Classify Discoveries; Measure Frequency and Set Thresholds.
Classify Discoveries
: A discovery is uncovering something that you didn’t know. Classify Discoveries into three categories; (A) Game Changer (B) Worthy and (C) Noted
(A) Game Changer.
A discovery that significantly changes a conclusion. For example, if you’re price shopping and you see a similar item at one-half the price, you might completely change your mind about which item to buy and where to buy it. If you’re in the manufacturing business and you discover a defect in a crucial part, or you create software and you discover a defect that will wipe out a customer’s data, you’ll push out a release date or re-engineer something
.
(B) Worthy.
Not a game changer, but somewhat impacting. Could affect customer satisfaction, or a schedule, or quality of product. You might end up addresses this by disabling some functionality, or requiring maintenance, or resetting expectations. It might be discovering that your candidate isn’t exactly what you thought you were getting or a forecast or a price that is 10% different. You probably won’t change course, but you’ll want to know if this is an anomaly, or just the tip of the iceberg.
(C) Noted
. Nice to know, but little to no consequences. Possibly require a work around, or a little, but not needed.
Measure Frequency:
How long does it take to discover something in category A, B and C. Measuring frequency is a way to determine how much effort it takes to discover an issue. At the beginning of an analysis, research or test, you might discover several category A’s that alter the course of a decision. You’ll discover category B’s that require further review, and lots of category C’s. Over time you’ll discover fewer A’s , B’s, and C’s. Your discoveries will take longer and longer, and over a period of time, the majority of discoveries will shift into category B’s and C’s, and then eventually, mostly just category C, and far fewer as well.
Set Thresholds:
When the Frequency of each of the categories of Discovery falls below a certain value, i.e. Threshold, you are done. Here is where Risk comes into play. For example, what’s the risk if a category A or B is discovered. In some cases, such as software, the risk is pretty low because you can just download the new version from the internet. In the case of safety, a category “A” might end up being a risk of a car crash. Based on the risk, you set a Threshold for Frequency per Category found. When the Frequency falls below the Threshold, for each Category, you are done.
Guarantees:
Despite all the testing, analysis, or research in the world, there is no guarantee that a defect, perhaps severe, might still exist and be discovered. However, following the above methodology; by thinking about the categories, tracking the issues discovered, tracking the frequency to find these, and setting thresholds, you will greatly increase the probability that a decision to move forward is the right course to take, and prevent analysis paralysis.
The Takeaway: Create Categories of Discoveries, Track the Frequency (time and effort) to uncover a discovery, set a threshold related to risk. When you meet the threshold, you’re done, it’s time to stop.
