The stark contrast between AI’s ability to crack longstanding mathematical conjectures and its repeated stumbles in everyday customer‑service workflows reveals a deeper truth about where today’s models excel and where they falter. It is not a matter of raw intelligence alone; the decisive factor is the reliability with which an AI can sustain correct behavior over extended periods. When a system can produce a correct answer only half the time, businesses cannot entrust it with unattended work, no matter how impressive the occasional success appears. This gap between occasional brilliance and dependable performance shapes investment decisions, product roadmaps, and the realistic timeline for autonomous AI agents in the enterprise.
To quantify this reliability, researchers at METR introduced the notion of a “time horizon.” They assemble a diverse set of real‑world software tasks, estimate how long a skilled human would need to complete each, then run an AI model on the suite and record the fraction of tasks solved correctly. Plotting success rate against task length yields a declining curve. The point at which the curve crosses the 50 % mark defines the 50 % time horizon – essentially the length of work the model can complete as often as not. For Claude Opus 4.6 this horizon sits near twelve hours, while a stricter 80 % horizon – the length at which the model succeeds eight times out of ten – collapses to roughly seventy minutes. The same model, same tasks, only the confidence threshold changes.
If failures occurred at a constant, independent rate – like radioactive decay – the success probability would drop exponentially with time, and the 80 % horizon would sit at about one‑third of the 50 % horizon. Observed ratios, however, are far larger: Opus 4.6 shows a 10‑to‑1 split, and Mythos Preview hovers around 6‑to‑1. This deviation tells us that failure is not a steady Poisson process. Instead, the model’s error pattern is far more structured, indicating that certain categories of tasks are intrinsically beyond its reach, regardless of how much time is allotted.
Examining the raw data clarifies the shape of the underlying curve. Beyond roughly an hour of task length, outcomes do not smear smoothly; they cluster into two distinct groups. One cluster consists of tasks the model solves correctly almost every time; the adjacent cluster contains tasks it fails almost every time, with few middle‑ground results. This bimodal distribution means the widely quoted smooth curve is merely a line drawn through a cloud of two populations. Consequently, even if the model improves its stamina, the presence of a hard “always‑fails” subset caps the achievable success rate, pulling down the 80 % horizon while the 50 % horizon remains buoyed by the solvable cluster.
When the empirical curve is plotted alongside the constant‑hazard prediction, the two intersect at roughly six to eight hours. Below this crossover, the model underperforms relative to the simple decay expectation – these are the “walls” where specific task types defeat the AI despite ample time. Above the crossover, the model outperforms the naive exponential decay: a hundred‑hour task still yields a roughly one‑in‑five chance of success. This overturns the popular narrative that length erodes performance; rather, once a task type is within the model’s capability, extending its duration does not appreciably harm success. The true barrier is the existence of immutable task categories, not the passage of time.
The decisive gate for deploying AI at scale is not whether the model can produce an answer, but whether we can reliably tell if that answer is correct. In formal mathematics, a proof checker such as Lean can mechanically verify every logical step. If the compiled proof passes, correctness is guaranteed, eliminating ambiguity. Under these conditions, a low success rate becomes economically viable: one can run the model many times, discard the failures, and keep the rare correct output. The compute cost of discovering ten decade‑old mathematical proofs was only about two thousand dollars precisely because verification is cheap and unambiguous.
Contrast this with asking an AI to add a new feature to a software product. The model writes code that passes the existing unit tests, yet passing those tests does not guarantee the feature fulfills the intended specification, handles overlooked edge cases, or aligns with stakeholder goals. Here, the verification layer is weak and expensive; any mistaken acceptance propagates into production, requiring costly human review, debugging, and potential rework. Consequently, the overall success rate must be high for automation to retain its advantage, otherwise the benefit evaporates under the weight of manual oversight.
This dichotomy maps cleanly onto a threshold result from fault‑tolerant computing: if the effective error rate after correction falls below a critical level, arbitrarily long computations can be performed with modest overhead; above that level, costs explode and long runs become infeasible regardless of budget. AI agents behave identically. The quality of the verification mechanism – the “checker” – determines whether lengthy autonomous work is economically viable. Importantly, a imperfect checker imposes an immutable ceiling on achievable correctness; no amount of model scaling can bypass this limit if the checker consistently misclassifies a fraction of bad outputs as good.
Plotting industries along two axes – checker quality (strong vs. weak) and attempt cost (cheap vs. expensive) – yields a useful quadrant model. In the strong‑checker, cheap‑attempt quadrant lie formal mathematics, theorem proving, SAT solving, chip verification, compiler optimization, and competitive programming. Here, low success rates are irrelevant because verification is inexpensive and definitive, explaining why OpenAI could solve deep‑standing problems with modest compute. The strong‑checker, expensive‑attempt quadrant includes chip fabrication, structural engineering, physical simulation, and clinical trials with hard endpoints; progress is limited by cycle time and resource cost, not by model intelligence.
The weak‑checker, cheap‑attempt quadrant encompasses most production software development, machine‑learning research, data analysis, technical writing, and generic agentic workflows. Tests may pass, yet specifications remain ambiguous or incomplete, making low reliability especially damaging. This is where the majority of enterprises experience frustration: AI can generate plausible code or analysis, but the lack of a trustworthy oracle means errors slip through, creating technical debt and operational risk. Finally, the weak‑checker, expensive‑attempt quadrant covers clinical medicine, wet‑lab biology, strategic planning, law, and policy – domains where feedback loops are slow, costly, and often subjective, relegating AI to an assistant role for the foreseeable future.
Current trends show the frontier shifting. Formal mathematics has already moved from AI‑assisted‑human to AI‑executed‑with‑checker‑verified. In leading labs and large tech firms, a majority of newly authored code is AI‑generated with human review, pushing the 50 % horizon beyond a full workday. The ratio between the 50 % and 80 % horizons is tightening – from roughly 10‑to‑1 for Opus 4.6 toward perhaps 5‑to‑1 for Mythos Preview – signaling improved error handling. However, Mythos’s seventeen‑hour 50 % figure brushes the measurement ceiling of METR’s test suite, so part of the gain may be an artifact of limited task length rather than a true capability leap. The 80 % horizon, meanwhile, is inching toward a typical workday, with projections placing it around mid‑2027 if present trajectories hold.
Looking further ahead, the next generation of hardware‑driven models is expected to primarily diminish the “always‑fails” cluster by expanding raw capability, thereby pushing reliable horizons into the eight‑to‑twenty‑four‑hour range. Yet closing the ratio gap – moving from the observed 10‑to‑1 toward the ideal 3‑to‑1 – will likely require models that can learn from their own mistakes within a single extended attempt, rather than starting each trial with a blank slate. Without such intra‑trial adaptation, systems may become extraordinarily capable in well‑checked niches while remaining brittle in open‑ended, weakly specified work, preserving a bifurcated economic impact.
For practitioners and investors, the most informative leading indicators are the ratio of the 50 % to 80 % horizons and the proportion of tasks that the model fails 100 % of the time. A declining ratio toward three indicates that internal error correction is maturing, while a falling failure‑only fraction reveals that the hard walls are receding. The threshold at which the 80 % horizon exceeds eight hours marks the point where businesses can begin to consider delegating a full workday to autonomous agents without constant supervision. Complementary metrics such as attempts‑per‑solve in verified domains and the retraction rate of machine‑checked mathematical proofs provide ground‑truth checks on claimed progress.
Actionable guidance: First, invest in or develop strong verification layers tailored to your domain – whether formal proof checkers, exhaustive test suites, or formal specifications – because the checker’s quality ultimately caps ROI. Second, treat AI as a high‑variance tool in weakly checked environments; use it for idea generation or drafting, but retain rigorous human validation for anything that ships to customers. Third, monitor the published METR horizon data and the bimodal failure distribution as early warnings of shifting capability. Finally, pilot low‑risk, high‑repetition tasks where verification is cheap and fast, using the resulting data to calibrate attempts‑per‑solve and build confidence before scaling to more ambitious, long‑horizon projects.