Why Winners Can Look More Skilled Than They Are
Search among one hundred candidates and keep the winner. In the synthetic experiment below, the winning score averages 88.1 across repetitions of that process. A fresh test of each selected candidate averages 75.6. That gap could suggest a failed replication or a dishonest report. Neither is necessary here. Every candidate was evaluated under the same rules. The problem was that we remembered only the best first result. Success is evidence, but the process that makes success visible is part of that evidence. A leaderboard, a collection of surviving companies, and a shelf of famous records are all filtered views. Reading the visible outcome as a transparent measure of underlying quality asks it to do more work than it can support. ...