Case Study

Agentic Systems: What Autonomous AI Workflows Actually Deliver — and Why 95 Percent of Pilots Fail

This is not a hype summary. 9 primary sources — from METR time-horizon studies to GAIA, SWE-bench and AgentBench to MIT, Gartner and McKinsey research — evaluated directly, every number linked to its source.

02.06.2026 · Research Case Study
Industry: AI & Automation Read time: 12 min Evidence: High (primary sources, benchmarks & field studies) 9 sources reviewed
Quick-Check — Executive Summary
  • Core finding: The task length AI agents can complete with a 50% success rate has doubled roughly every 7 months since 2019, per METR — currently around one hour for frontier models like Claude 3.7 Sonnet.
  • The benchmark reality: On GAIA, humans score 92%, GPT-4 with plugins only 15% — on tasks deliberately designed to be "easy for humans."
  • The productivity paradox: In a randomized METR field study with 16 experienced open-source developers, using AI tools made them 19% slower on 246 real issues — yet they believed afterward they had been 20% faster.
  • The enterprise reality per MIT: 95% of internal GenAI pilots show no measurable P&L effect according to the MIT NANDA study; buying from specialized vendors is roughly twice as successful as building in-house.
  • The risk forecast: Gartner expects more than 40% of agentic AI projects to be canceled by end of 2027 — of thousands of vendors, Gartner considers only around 130 to offer genuine agent technology, the rest being "agent washing."
  • The technical cost: Anthropic's own multi-agent system performs 90.2% better than a single agent — but burns roughly 15x as many tokens as a simple chat.

Case Study — AI & Automation

Agentic systems: between exponential progress and enterprise reality

"Agentic AI" has become the industry's most-used buzzword of 2025/26: AI systems that no longer just respond to individual prompts but independently plan multi-step tasks, call tools, evaluate intermediate results and adjust their plans. Between the marketing promises and actual capability sits a gap that can only be closed with independent measurement — not product announcements.

The research question for this case study is therefore not "how good does agentic AI sound in the keynote," but: what do independent benchmarks actually show about the capabilities and limits of current agents? How high is real enterprise adoption according to reliable surveys? What failure modes are documented? And where is there proven ROI versus failed pilots?

Methodologically, nine primary sources across three categories were checked directly for this analysis: (1) academic agent benchmarks on arXiv (GAIA, SWE-bench, AgentBench), (2) field studies measuring real-world capability and productivity (METR), and (3) enterprise adoption data from analyst and research institutions (MIT NANDA, Gartner, McKinsey) plus a technical disclosure of a production multi-agent system (Anthropic). All numbers come from the original documents themselves, not from secondary reporting without a cross-check.

01
METR: agent time horizons double every 7 months — but stay weak on long tasks
Nonprofit research organization METR defines "time horizon" as the task length (measured by how long it takes human experts) an agent can complete autonomously with a 50% success probability. Across six years and a test suite of software, cybersecurity and reasoning tasks, an exponential trend appears: the solvable task length roughly doubles every 7 months. Current frontier models like Claude 3.7 Sonnet reach around one hour at 50% success — for tasks under 4 minutes, success rates are near 100%, while for tasks over 4 hours, they fall below 10%.
Source: METR — Measuring AI Ability to Complete Long Tasks
Technical Deep-Dive: how "time horizon" is measured

METR first assigns each of its 170 test tasks a human reference time (seconds to dozens of hours), then measures at what task length a model still achieves a 50% or 80% success probability — analogous to a dose-response curve in pharmacology. The authors themselves list several limitations: methodological choices in time estimation influence the result, trend breaks since 2024 are possible, and different benchmarks (e.g. SWE-bench without onboarding time) are only partially comparable. A tenfold measurement error would still only shift the forecast by roughly two years — METR considers the underlying exponential trend robust, but its continuation into the future an extrapolation, not a measurement.

02
GAIA: 92% human success rate vs. 15% for GPT-4 with tool access
The GAIA benchmark (Meta FAIR, Meta GenAI, Hugging Face and others) comprises 466 questions deliberately designed to be conceptually easy for humans but demanding for agents — requiring reasoning, multimodality, web browsing and general tool use, such as parsing PDFs, tables, images or audio files. The result inverts the usual trend: human testers solve 92% of tasks correctly, GPT-4 with plugin access only 15%. That's a 77-point gap — on tasks that require no expert-level skill, just robust everyday competence.
Source: GAIA: a Benchmark for General AI Assistants (arXiv 2311.12983)
03
SWE-bench: from 1.96% success to benchmark saturation — and back to earth with SWE-Bench Pro
SWE-bench tests agents against 2,294 real GitHub issues from 12 popular Python repositories. On release in 2023, even the strongest model at the time, Claude 2, solved only 1.96% of tasks. By 2025, scores on the heavily-trained "Verified" subset rose sharply — combinations of Claude models and specialized agent harnesses at times exceeded 60–79%. That jump raised contamination concerns: the follow-up study SWE-Bench Pro uses 1,865 new, contamination-resistant problems from actively maintained enterprise repositories — there, even Claude Sonnet 4.5 reaches only 43.6% and Claude Sonnet 4 only 42.7%. Real progress turns out to be considerably smaller than the saturated numbers of the original benchmark suggest.
Source: SWE-bench: Can Language Models Resolve Real-World GitHub Issues? (arXiv 2310.06770)
04
AgentBench: top commercial models stumble on long-horizon reasoning and decision-making
AgentBench (Tsinghua/OSU/UC Berkeley, ICLR 2024) tests LLM agents across 8 different environments — from OS and database interaction through digital card games to web shopping and web browsing. The central finding: even strong commercial APIs show a clear performance gap against smaller open-source models (up to 70B parameters), and the authors identify three recurring failure modes as the main obstacle to practical agents: weak long-horizon reasoning, faulty decision-making, and poor instruction-following across multiple steps. Code training also shows "ambivalent" rather than uniformly positive effects across the various agent tasks.
Source: AgentBench: Evaluating LLMs as Agents (arXiv 2308.03688)
05
The METR field study: experienced developers got 19% slower with AI tools — and didn't notice
In the methodologically strictest field study to date on real productivity impact, METR had 16 experienced open-source developers (averaging 5 years of project experience) work through 246 real issues in their own mature projects — randomized with or without access to current AI tools (primarily Cursor Pro with Claude 3.5/3.7 Sonnet). Result: with AI use allowed, developers took 19% longer. The perception gap is the real alarm signal here: before the study, participants expected a 24% speedup; afterward, they still estimated they had been 20% faster — when in fact they were slower. METR cites possible explanations including overestimation from anecdotal experience and benchmark comparisons, plus the study's own methodological limits.
Source: METR — Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity

"The core barrier to scaling [GenAI] is not infrastructure, regulation, or talent. It is learning."

MIT NANDA — The GenAI Divide: State of AI in Business 2025
06
MIT NANDA: 95% of enterprise GenAI pilots show no measurable P&L effect
The widely cited MIT study "The GenAI Divide" (Project NANDA, August 2025) analyzed over 300 publicly documented enterprise deployments, 52 in-depth case studies and 153 leadership interviews. Core finding: 95% of individually built GenAI pilot projects show no measurable effect on profit or revenue — only 5% generate provable financial value. At the same time, 90% of employees privately use GenAI tools despite the lack of enterprise solutions, per the study, while only around 40% of companies have signed official enterprise subscriptions. One key structural difference: buying tailored solutions from specialized vendors (buy/partner) is roughly twice as successful as internal in-house development (build) — because external tools tend to absorb feedback and adapt to context more readily.
Source: MIT NANDA — The GenAI Divide: State of AI in Business 2025 (PDF)
07
Gartner: over 40% of agentic AI projects will be canceled by 2027
Gartner forecasts that more than 40% of all agentic AI projects will be abandoned by the end of 2027 — citing escalating costs, unclear business value and insufficient risk controls. An accompanying Gartner survey of 3,412 webinar attendees (January 2025) shows how early-stage the market still is: only 19% of organizations have invested "significantly" in agentic AI, 42% invest conservatively, 8% not at all, and the rest are undecided. Gartner puts the "agent washing" problem especially bluntly: of the thousands of vendors positioning themselves as agentic AI providers, Gartner considers only around 130 to offer genuinely substantial agent technology — the rest are relabeled chatbots, RPA tools or classic assistants.
Source: Gartner Newsroom — Over 40% of Agentic AI Projects Will Be Canceled by End of 2027
08
McKinsey: 23% actively scale AI agents — but no function exceeds 10% full-scale
McKinsey's 2025 "State of AI" survey shows a clear gap between experimenting and genuine scaling: 23% of organizations scale active agentic AI systems in at least one business function, another 39% are still experimenting, while around 63% have not yet begun enterprise-wide scaling. Even where scaling happens, penetration stays thin: in no single business function do more than 10% of respondents report having agents fully scaled there. Only 39% of companies report any EBIT effect at the overall company level — despite broad individual deployments. Overall AI usage did rise to 88% of all organizations (from 78% the year before), yet only around 6% qualify as "high performers" with more than 5% EBIT impact from AI.
Source: Forbes — 10% of Enterprise Functions Use AI Agents, McKinsey Finds
09
Anthropic: multi-agent systems perform 90.2% better — but cost 15x more tokens
Anthropic disclosed the internals of its production multi-agent system (an orchestrator with Claude Opus 4 as the lead agent, coordinating multiple Claude Sonnet 4 subagents). In internal research evaluations, the multi-agent setup outperformed a single-agent system by 90.2% — for example on the task of correctly identifying every board member of S&P 500 IT companies, where the single agent failed. The price: single-agent systems already consume roughly 4x as many tokens as a normal chat, multi-agent systems roughly 15x — token usage alone explains 80% of the performance variance between runs, per Anthropic.
Source: Anthropic Engineering — How We Built Our Multi-Agent Research System
Technical Deep-Dive: failure modes and countermeasures in production multi-agent systems

Anthropic documents concrete failure patterns from production: orchestrator agents spinning up as many as 50 subagents simultaneously for trivial requests; subagents endlessly searching the web for sources that don't exist; and duplicated work when task descriptions between subagents aren't clearly enough separated. Especially critical: small system errors can snowball into catastrophic behavior changes, because agents keep processing their own faulty state as context. As countermeasures, Anthropic relies on resuming from checkpoints instead of full restarts, explicitly telling an agent when a tool call fails (so it can adapt), and "rainbow deployments" that don't abruptly interrupt running agent sessions during updates. The lesson: robustness doesn't come from better prompts alone, but from deterministic technical safety nets wrapped around the non-deterministic model component.

Capability, productivity and enterprise reality of agentic AI compared
ParameterBaselineObserved valueEffectSource
Task time horizon (50% success rate)Doubling since 2019roughly every 7 months, currently ~1 hour (Claude 3.7 Sonnet)Exponential, but from a low base[1]
GAIA benchmark (tool use, everyday tasks)Human: 92%GPT-4 + plugins: 15%77-point gap[2]
SWE-bench original (2023) vs. SWE-Bench Pro (2025)Claude 2: 1.96% solvedClaude Sonnet 4.5 on Pro set: 43.6%Old benchmark saturated, new one shows the real gap[3]
Real developer productivity (RCT, n=16, 246 issues)Expected speedup: 24%Actual: 19% slower43-point perception gap[5]
Enterprise GenAI pilots (MIT, >300 deployments)Expectation: fast ROI95% with no measurable P&L effectLearning/feedback mechanism usually missing[6]
Agentic AI projects (Gartner forecast to 2027)Currently 19% "significant" investment>40% will be canceledCost/risk/benefit mismatch[7]

Learnings & Actionable Insights

What follows in practice from nine primary sources

For companies

Take the MIT NANDA finding seriously before any agent pilot: buying from specialized vendors instead of building in-house roughly doubles the probability of success. Plan for clear error budgets and feedback loops from the start — the goal isn't avoiding friction, it's treating it as the learning mechanism.

For agencies

Actively manage expectations: the real, reliably solvable time horizon for agent tasks currently sits around one hour (50% success rate) — client promises of multi-day autonomous workflows aren't backed by current models empirically.

For developers & creators

The METR field study shows perceived speed and real productivity can diverge by 40 percentage points. Actually measure your own time savings with AI tools instead of just estimating, and constrain agents to clearly scoped, short subtasks.

For investors & analysts

Price in Gartner's "agent washing" warning: of thousands of self-described agentic AI vendors, only around 130 are considered substantial. The forecast 40%+ cancellation rate by 2027 should factor into any valuation of agent startups.

Open Questions
  • Does the 7-month doubling of time horizon that METR measured hold, or does the curve flatten once tasks stretch beyond a day?
  • Why does the real RCT productivity measurement (−19%) contradict positive benchmark trends so sharply — is it generalization problems in the benchmarks, or developers' inexperience with the tools?
  • What share of the current 19% "significant" agentic AI investments will survive Gartner's forecast cancellation wave by 2027?
  • Does the MIT 95% figure shift meaningfully as more companies consistently switch from in-house builds to specialized buy solutions?
  • Can the 15x token overhead of multi-agent systems be reduced in the medium term without losing the 90%+ performance advantage over single agents?
Verified Sources & Studies
  1. Measuring AI Ability to Complete Long TasksMETR, March 19, 2025The primary study behind the "time horizon" metric and the exponential capability growth of agents.
  2. GAIA: a Benchmark for General AI AssistantsarXiv 2311.12983, Mialon et al. (Meta FAIR/Hugging Face), Nov. 2023Shows the gap between everyday tasks that are easy for humans and actual agent performance.
  3. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?arXiv 2310.06770, Jiménez et al., Oct. 2023The reference benchmark for autonomous coding agents; starting point for the saturation/contamination debate.
  4. AgentBench: Evaluating LLMs as AgentsarXiv 2308.03688, Liu et al., ICLR 2024Documents concrete failure modes like weak long-horizon reasoning and decision-making across 8 environments.
  5. Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer ProductivityMETR, July 10, 2025The only randomized field study on the real productivity effect of AI coding tools on experienced developers.
  6. The GenAI Divide: State of AI in Business 2025MIT NANDA / Project NANDA, Aug. 2025The largest known enterprise survey on GenAI pilot success rates (95% fail).
  7. Gartner Predicts Over 40% of Agentic AI Projects Will Be Canceled by End of 2027Gartner Newsroom, June 25, 2025Analyst forecast on investment reality, cancellation rate, and "agent washing."
  8. 10% of Enterprise Functions Use AI Agents, McKinsey FindsForbes, March 22, 2026 — reporting on McKinsey's "State of AI" 2025The most current adoption and scaling figures from McKinsey's global enterprise survey.
  9. How We Built Our Multi-Agent Research SystemAnthropic Engineering, June 2025The only detailed technical disclosure of a production multi-agent system, including failure modes and token cost.

← News