The recent findings from METR reveal a striking disparity: advanced models like Claude Opus 4.6 can sustain effort for roughly twelve hours before hitting a 50% success rate, yet their reliable performance—defined as an 80% chance of correctness—drops to just over an hour. This gap is not a mere statistical curiosity; it signals a fundamental shift in how we should evaluate AI’s readiness for real‑world work. Traditional benchmarks that celebrate peak capability often mask the fragility that emerges when tasks stretch beyond short bursts. For decision‑makers, the lesson is clear: intelligence without dependable consistency translates into limited economic value, especially in environments where errors propagate quickly.
To understand the numbers, METR constructs a “time horizon” by measuring how long a skilled human would need for a variety of genuine software tasks, then records the proportion an AI model completes correctly at each duration. The resulting curve plots success probability against task length, and two key points are extracted: the length at which success hits 50% (the 50% horizon) and the length at which it reaches 80% (the 80% horizon). For Opus 4.6 these are about twelve hours and seventy minutes respectively. The ratio between them—roughly ten‑to‑one—deviates sharply from what a simple failure‑per‑unit‑time model would predict, hinting at deeper structural limits rather than mere fatigue.
The naive model assumes a constant probability of a fatal mistake each minute, akin to radioactive decay. Under that assumption, success probability would fall exponentially, and the 80% horizon would sit at about one third of the 50% horizon—a 3‑to‑1 ratio. The observed ratios of 10‑to‑1 for Opus 4.6 and perhaps 6‑to‑1 for the newer Mythos Preview are far worse, indicating that failure does not accumulate uniformly. Instead, the data show a flat‑ish curve: the model’s odds of success do not deteriorate steadily with time, which contradicts the idea of a steady‑state loss of focus or memory.
Examining the raw task‑by‑task results resolves the paradox. Beyond roughly one hour of effort, outcomes do not smear smoothly across difficulty levels; they cluster into two distinct groups. One cluster comprises tasks the model solves correctly virtually every time, while another cluster contains tasks it fails almost invariably, with relatively few cases landing in the middle. This bimodal distribution means the smooth curve often quoted is merely a line drawn through a mixed population, obscuring the existence of hard boundaries that the model simply cannot cross.
These boundaries, or “walls,” represent categories of tasks that lie outside the model’s current competence, regardless of how long it is allowed to work. If, say, 20 % of the tasks in the benchmark are inherently unsolvable for the architecture, then even an infinite amount of time cannot push the success rate above 80 %, because that fraction will always drag the average down. Consequently, the 80% horizon collapses to a short duration, while the 50% horizon—reflecting the median of a mixture that includes many solvable tasks—can stretch much further, explaining the observed ten‑to‑one spread.
Plotting the actual success curve against the constant‑hazard prediction uncovers a crossover point around six to eight hours. Below this point, the model underperforms relative to the simple decay theory, precisely because it encounters those unsolvable walls early on. Above the crossover, however, the real model outperforms the naïve expectation: tasks that it can do at all remain doable even as they grow longer, and a non‑trivial fraction (roughly one in five) still succeeds at extreme lengths. This reverses the popular narrative that AI agents inevitably drift or lose coherence over time; instead, length itself is not the enemy—only the presence of insurmountable task types caps reliability.
The critical gate for deploying AI at scale is not raw capability but the ability to verify whether the model’s output is correct. In formal mathematics, a proof checker like Lean can mechanically validate every logical step, turning a low success rate into a tractable problem: run the model many times, keep the rare correct proof, and discard the rest. The cost of attempts is low, and verification is unambiguous, so even a 5 % success rate can yield valuable results, as demonstrated by OpenAI’s recent solutions to long‑standing conjectures.
Contrast that with asking a model to add a feature to a production codebase. The model may generate code that passes existing unit tests, yet those tests only capture what the programmer thought to check. Subtle misunderstandings of intent, edge cases, or hidden requirements can slip through, leaving defects that surface later in production. Here, verification is weak and expensive, because each false positive necessitates human review, eroding the automation advantage. The success rate now matters immensely, as every undetected error becomes a costly downstream issue.
This dichotomy maps neatly onto a framework borrowed from fault‑tolerant computing: there exists an error‑rate threshold below which long computations remain feasible with modest overhead, and above which costs explode. For AI agents, the quality of the verification mechanism plays the role of the error‑correcting code. If your checker mistakenly approves 5 % of bad work, then no amount of model improvement can push the reliable ceiling above 95 %, because the verification itself introduces an irreducible noise floor.
Applying this lens to the economy yields a useful map. Domains with strong, cheap verification—formal theorem proving, SAT solving, chip verification, compiler optimization—are already in the “automation is here” quadrant; low success rates are irrelevant because you can brute‑force attempts. Areas with strong but expensive verification—chip fabrication, structural engineering, clinical trials with hard endpoints—are limited by cycle time, not raw AI power. The large swath of the economy where verification is cheap but weak—most software development, data analysis, content creation—is where low reliability hurts the most, explaining why customer‑support bots still falter despite impressive demo performances. Finally, sectors with weak and expensive verification—medicine, wet‑lab biology, high‑level strategy, law—remain largely assistant‑only zones for the foreseeable future.
Where we stand today shows the curve shifting. Formal mathematics has largely moved from AI‑assisted‑human to AI‑does‑work‑plus‑checker‑verifies‑it, with most new code at frontier labs being AI‑generated and human‑reviewed. The 50% horizon has exceeded a full human workday for leading models. The ratio between the two horizons is tightening—from roughly 10‑to‑1 for Opus 4.6 toward perhaps 5‑to‑1 for Mythos Preview—suggesting incremental gains in error correction. However, Mythos’s seventeen‑hour figure brushes against the measurement ceiling of METR’s current suite, so some of the apparent advance may be an artifact of limited task length rather than a genuine leap.
Looking ahead, the next generation of hardware‑driven models is expected to shrink the pool of tasks that the model fails on outright, because increased raw capability can breach more of those hard walls. We may see reliable horizons stretching into the eight‑to‑twenty‑four‑hour range, necessitating a redesign of benchmarks that now top out at shorter durations. Pure scale, however, is unlikely to fix the ratio problem; closing the gap toward the ideal 3‑to‑1 will require models that learn from their own mistakes within a single extended attempt, rather than resetting after each step. Such continual learning remains elusive today because updating weights on unverified outcomes risks compounding errors, potentially turning the flat failure curve into a steep one.
The generation after that could unlock step‑changes if continual learning proves viable. In that scenario, long autonomous work becomes more trustworthy even when verification is imperfect, expanding the envelope of economically impactful automation beyond strictly checked domains. If continual learning fails, we will instead see extraordinarily capable systems that excel in well‑checked niches—mathematics, formal verification, simulation—but remain brittle on open‑ended, poorly specified work, leaving many organizations unable to hand over full projects to AI without intensive oversight.
What should leaders watch closely? First, track the ratio between the 50% and 80% horizons; a movement toward the 3‑to‑1 benchmark signals that internal error‑correction mechanisms are maturing. Second, monitor the proportion of tasks that the model fails on 100 % of the time across successive releases—the size of the “always‑fails” wall is a direct measure of structural limits. Third, note when the 80% horizon crosses the eight‑hour mark, as that denotes the point at which a model can be trusted for a full workday in reliably checked settings. Fourth, in verified domains, consider the honest metric of attempts‑per‑solve rather than the marketed cost of the winning run, to gauge real efficiency. Finally, observe whether any of OpenAI’s recent machine‑checked mathematical proofs survive scrutiny over the next six months; retraction‑free longevity would cement the role of verification‑based automation as a durable competitive advantage.