A Statistical Framework for Luck in AI Evaluation
An agent finds a supported answer on its third attempt. Its trace shows a useful result, a successful tool call, and a judge’s approval. What did the team improve: the memory policy, the opportunity to try again, or the conditions in which this run happened? This question becomes practical when a deployment decision rests on the answer. A retry budget is something we can intervene on. A temporary tool outage is something we can record and test. A permissive judge can make a poor policy appear reliable. Calling the remaining uncertainty luck helps only if it leads us to an experiment capable of separating these mechanisms. ...