Calibration, Not Certainty

The forecasters who beat the CIA weren't more certain than the analysts. They were better calibrated.

You've been waiting to feel sure before you make the call -- the pivot, the hire, the bet. More data, one more conversation, one more pass at the numbers, and still, somehow, not sure enough. It feels like a research problem. It's a different problem wearing research's clothes.

The study everyone half-remembers

A popular version of this story goes around every so often: a 20-year study found ordinary people, trained the right way, out-forecast intelligence analysts with classified access. That version conflates two separate pieces of research, and the honest version is more useful than the viral one. The real 20-year study, Philip Tetlock's Expert Political Judgment project (1984-2003), found the average expert barely beat chance. The "superforecasters" came from a different, later project entirely -- the Good Judgment Project (2011-2015), a four-year IARPA tournament led by Tetlock and Barbara Mellers, where amateur forecasters with no classified access outperformed intelligence analysts who had it, by roughly 25-30%.

Here's the detail that actually matters: those superforecasters weren't unusually certain. Their average IQ ran around the 80th percentile -- smart, not genius-tier -- and subject-matter credentials didn't predict who won. What predicted it was a specific, trainable habit: making granular probability estimates instead of vague hunches, and updating them often as new information arrived. A short training module measurably improved ordinary forecasters' accuracy. The skill wasn't a gift. It was reps.

What calibration actually means

Calibration isn't confidence, and it isn't certainty. You're calibrated if, across every time you say "I'm 70% sure," the thing happens about 70% of the time -- not 95%, not 40%. It's accuracy about your own uncertainty, which is a completely different skill from feeling sure. You can be extremely confident and badly calibrated. You can be only 60% sure and perfectly calibrated, and that 60% is real, usable information -- if you're willing to move on it instead of waiting for it to become 95.

That reframes the whole stall. The big move keeps waiting for a certainty that was never coming, because certainty was never the actual requirement. The requirement was an honest number, and a willingness to act on the number you actually have instead of the one you wish you had.

The drill: score it, don't just have it

The mechanism behind the superforecasters' edge wasn't smarter guessing. It was scored guessing -- a real prediction, given a real number, dated, and checked against what actually happened. A judgment you never score can't get better, the same way reflection that never resolves into a move can't turn into a decision. Pick the move you keep circling. Put a number on it -- not "I should probably do this" but "I'm 65% sure this works if I move by Friday." Write the date. Check it when it resolves.

You'll be wrong sometimes. That's not the failure mode -- an uncalibrated 95% that turns out to be 60% is the failure mode, because it's the one that keeps you waiting for a feeling that was never going to arrive on its own.

Sources

• On the Good Judgment Project (2011-2015, Tetlock & Mellers) finding trained forecasters without classified access outperforming intelligence analysts by ~25-30%, and calibration/updating-frequency (not IQ or credentials) as the key trainable differentiator: Tetlock, P. E. & Gardner, D. (2015), Superforecasting: The Art and Science of Prediction; primary tournament findings via the Good Judgment Project.
• On the separate, earlier Expert Political Judgment study (1984-2003) finding the average expert barely beat chance -- the study popular retellings conflate with the Good Judgment Project: Tetlock, P. E. (2005), Expert Political Judgment: How Good Is It? How Can We Know?