A Statistical Framework for Luck in AI Evaluation

An agent finds a supported answer on its third attempt. Its trace shows a useful result, a successful tool call, and a judge’s approval. What did the team improve: the memory policy, the opportunity to try again, or the conditions in which this run happened? This question becomes practical when a deployment decision rests on the answer. A retry budget is something we can intervene on. A temporary tool outage is something we can record and test. A permissive judge can make a poor policy appear reliable. Calling the remaining uncertainty luck helps only if it leads us to an experiment capable of separating these mechanisms. ...

October 19, 2026 · 17 min

Why Winners Can Look More Skilled Than They Are

Search among one hundred candidates and keep the winner. In the synthetic experiment below, the winning score averages 88.1 across repetitions of that process. A fresh test of each selected candidate averages 75.6. That gap could suggest a failed replication or a dishonest report. Neither is necessary here. Every candidate was evaluated under the same rules. The problem was that we remembered only the best first result. Success is evidence, but the process that makes success visible is part of that evidence. A leaderboard, a collection of surviving companies, and a shelf of famous records are all filtered views. Reading the visible outcome as a transparent measure of underlying quality asks it to do more work than it can support. ...

October 12, 2026 · 10 min

What Can Statistics Mean by Luck?

Fifteen successes in twenty attempts. Before knowing anything else, it is tempting to admire the person who achieved them. After learning that the usual success rate is one half, we may begin to speak of good fortune. After learning that it is seven tenths, the same record seems considerably less exceptional. Nothing about the observed count has changed. What changed was the story we were prepared to tell before seeing it. ...

October 5, 2026 · 15 min · Updated October 5, 2026