Statisticians Re-Fit the METR Time-Horizon Plot and Find Difficulty Is Not Linear in Log Time

The Three-Question Framework

What Happened?

In an arXiv paper submitted 8 October 2026 (2610.12466), Drew T. Nguyen and William Fithian re-estimate METR's 50% time horizon for 26 AI systems across 228 tasks. They drop METR's assumption that task difficulty scales linearly with the log of human completion time and use splines and item-response theory instead.

Why It Matters

The 50% time horizon is the human completion time of tasks an AI solves half the time, and it is widely quoted as a measure of progress. The authors find the fitted curve is nearly flat from about 2 to 30 minutes and roughly linear elsewhere. So a tenfold step from 3 to 30 minutes is easier for an AI than one from 30 minutes to 5 hours. Equal multiples do not mean equal difficulty.

What Could Happen Next

The authors say their estimates score better under cross-validated proper scoring rules and urge readers to read time horizons together with diagnostic plots, especially as longer-task benchmarks arrive. Expect METR and others to respond. The paper is a v1 preprint and has not been peer reviewed.

Executive Key Takeaways

Re-analysis of 26 AI systems on 228 tasks.

Fitted difficulty curve is nearly flat between about 2 and 30 minutes and close to linear elsewhere.

Equal 10x changes in time horizon are not equal increases in difficulty.

Authors report better cross-validated proper-scoring-rule performance than the linear log-time assumption.

They recommend reading horizons with diagnostic plots; the paper is an unreviewed v1 preprint.

The source is the arXiv abstract for 2610.12466 (cs.AI, v1, submitted 8 October 2026). We read the abstract, not the full paper, so details of the method are limited to what the abstract states.

METR expresses AI capability as a 50% time horizon: the human completion time of software tasks that an AI system solves half the time. Fitting that number requires a model of how success probability falls as tasks get longer. METR's approach assumes difficulty is linear in the log of human time.

Nguyen and Fithian relax that assumption with splines and item-response theory. Their fitted spline is almost flat between roughly 2 and 30 minutes and near linear outside that range. A practical consequence: moving from 3 minutes to 30 minutes is easier for an AI than moving from 30 minutes to 5 hours, even though each is a factor of ten.

That matters for trend lines. A plot showing horizon doubling every fixed period implicitly treats each doubling as equal progress. If difficulty is not uniform in log time, the same doubling can hide very different gains.

The authors say their point estimates score better under cross-validated proper scoring rules. They also supply diagnostic plots to check whether time horizons measure what they claim to, and recommend using those plots as longer-task benchmarks appear.

Limits: this is one preprint, not reviewed, and we have not checked its code or data. Independent critiques of METR's statistics already exist, but this paper is the newest and uses a different method.

The Three-Question Framework

What Happened?

Why It Matters

What Could Happen Next

Executive Key Takeaways

Primary Sources & Citations