A Statistical Framework for Luck in AI Evaluation

An agent finds a supported answer on its third attempt. Its trace shows a useful result, a successful tool call, and a judge’s approval. What did the team improve: the memory policy, the opportunity to try again, or the conditions in which this run happened? This question becomes practical when a deployment decision rests on the answer. A retry budget is something we can intervene on. A temporary tool outage is something we can record and test. A permissive judge can make a poor policy appear reliable. Calling the remaining uncertainty luck helps only if it leads us to an experiment capable of separating these mechanisms. ...

October 19, 2026 · 17 min

What Can Statistics Mean by Luck?

Fifteen successes in twenty attempts. Before knowing anything else, it is tempting to admire the person who achieved them. After learning that the usual success rate is one half, we may begin to speak of good fortune. After learning that it is seven tenths, the same record seems considerably less exceptional. Nothing about the observed count has changed. What changed was the story we were prepared to tell before seeing it. ...

October 5, 2026 · 15 min · Updated October 5, 2026