The Expert Intuition Validity Test
Determine when to trust gut feelings and when to demand data
Through his adversarial collaboration with Gary Klein, Kahneman identified the two conditions under which expert intuition can be trusted: a sufficiently regular environment to be predictable, and an adequate opportunity to learn those regularities through prolonged practice with rapid feedback. When both conditions are met, experts develop genuine pattern recognition skills. A chess master reads a complex position at a glance; a firefighter senses danger before articulating why; an experienced nurse detects a patient's deterioration from subtle cues.
However, many domains that produce confident experts fail one or both conditions. Stock pickers, political pundits, and long-term economic forecasters operate in low-validity or zero-validity environments where the feedback is delayed, ambiguous, or nonexistent. In these environments, intuitive confidence is driven by narrative coherence (WYSIATI) rather than genuine pattern recognition. The confidence feels identical from the inside, which is why Kahneman and Klein agreed on a critical principle: the confidence that people have in their intuitions is not a reliable guide to their validity.
The framework provides a diagnostic protocol for evaluating when expert intuition is likely to be genuine and when it is likely to be illusory. The evaluation focuses on the environment and the expert's learning history, not on the expert's confidence or credentials.
- Intuition is recognition: the situation provides a cue, the cue accesses information in memory, and the information provides the answer
- Valid intuition requires a regular environment with stable patterns and rapid, unambiguous feedback
- Subjective confidence is not a reliable indicator of intuitive validity; it reflects narrative coherence, not predictive accuracy
- Expertise is not a single skill; the same professional may have genuine intuition for some tasks and none for others
- In low-validity environments, simple algorithms consistently outperform expert judgment
- Assess the environment's regularityAsk whether the domain has stable, recurring patterns that allow prediction. Chess, firefighting, medicine (some specialties), and weather forecasting have high regularity. Stock markets, geopolitical events, and long-term economic forecasts have low regularity. The validity of intuition cannot exceed the predictability of the environment.
- Evaluate the feedback loopAsk whether the expert receives rapid, clear feedback on the accuracy of their judgments. Anesthesiologists get immediate feedback; radiologists often do not. Psychotherapists get feedback on in-session dynamics but not on long-term treatment outcomes. The quality and speed of feedback determine whether genuine expertise can develop.
- Check the expert's learning historyHas the expert had sufficient practice in this specific task? Expertise requires thousands of hours of deliberate practice with high-quality feedback. An expert in one area of a domain may be a novice in another. A surgeon brilliant at one operation may have no special skill at another.
- Separate confidence from calibrationDo not ask whether the expert is confident; ask whether their confidence has been calibrated against outcomes. Have they tracked their predictions? What is their hit rate? The most overconfident experts are often the most celebrated, but calibration, not confidence, predicts accuracy.
- Default to algorithms when availableIn low-validity environments or when expert judgment is poorly calibrated, prefer simple algorithms, checklists, or statistical models. Meehl demonstrated across 136 studies that mechanical prediction equals or exceeds clinical prediction. Even imperfect algorithms provide consistency that human judgment cannot match in noisy environments.
In Klein's research, a fireground commander led his team into a burning house and suddenly ordered everyone out without being able to explain why. Moments later, the floor collapsed. Subsequent analysis revealed that the commander had unconsciously registered that the fire was unusually quiet and the room was unusually hot, cues that indicated the main fire was below them in the basement rather than in the room they were addressing.
This framework emerged from a seven-year adversarial collaboration between Kahneman and Gary Klein, published as 'Conditions for Intuitive Expertise: A Failure to Disagree.' Klein, the champion of naturalistic decision making, had studied expert firefighters and nurses who demonstrated remarkable intuitive skill. Kahneman, skeptical of expert intuition due to his work on biases, had focused on overconfident clinicians and stock pickers. They discovered that their disagreement was partly about which experts they had in mind, and they converged on conditions that distinguish genuine expertise from confident ignorance. Herbert Simon's definition of intuition as recognition provided the theoretical bridge.