The Year of the Great Disconnect
In 2025, companies spent $37 billion on generative AI—AI systems that produce text, code, or answers rather than just crunching numbers—more than triple the $11.5 billion spent the year before. Yet a July study from the research group MIT NANDA found that 95 percent of these AI projects produced no measurable financial return.
Vendors, meanwhile, reported explosive growth in usage. OpenAI said the number of complex, multi-step "reasoning" tasks its models performed per customer organization grew 320-fold over the year. So AI was being used more than ever, but for most companies that usage never turned into savings or revenue. Researchers call this gap the "GenAI Divide": a small group of companies—about 5 percent—found real financial value from AI, while the rest got stuck experimenting without ever seeing a return.
This isn't a story about bad models. It's about bad strategy. The successful 5% weren't using better technology than everyone else. They were better at solving a different problem entirely: moving AI out of the realm of individual employees using it as a personal productivity tool, and into the actual workflows—the step-by-step processes—that run the business. They understood that a successful pilot project isn't one that simply works in a demo; it's one that gets adopted, scales up, and ultimately makes or saves the company money. The lesson of 2025 is that the hardest part of enterprise AI has almost nothing to do with AI itself.
The Anatomy of a Failed Pilot
The path to the 95% is paved with good intentions and eerily similar mistakes. While headlines focused on model releases and benchmark scores, the real story unfolded in stalled projects, budget overruns, and pilots that never found a permanent home. According to the MIT NANDA report, of all the enterprise-grade AI systems evaluated by companies, only 20% even made it to a pilot. A mere 5% ever reached production.
These failures aren't random; they follow predictable patterns. Understanding them helps explain why so many projects that look promising in a demo end up as expensive dead ends.
The Solution in Search of a Problem
The most common failure begins in the innovation lab. A team gets access to a powerful new multimodal model or a novel agentic framework. The mandate is to "find use cases for AI." They build a dazzling demo—perhaps an agent that can browse internal wikis and file expense reports—and present it to leadership. Everyone is impressed. The pilot is approved.
And then it stalls. The project has a champion in the technology department but no owner in the business. The finance department, which actually manages expense reporting, was never consulted. It already has a process, and while that process isn't perfect, it's reliable and integrated with the accounting software. The AI solution, for all its technical elegance, doesn't solve a problem the business owner actually feels. It's a vitamin, not a painkiller.
Successful projects start with the pain. They originate not from a technologist asking "What can I build with this?" but from a business leader asking "Why does this process take three days?" or "Why is our customer churn so high in this segment?" The technology is then pulled in to solve a specific, quantified business need, with a committed owner accountable for the result.
The Chatbot That Should Have Been a Form
A close cousin to the "solution in search of a problem" is the misapplication of conversational AI. Generative models excel at handling ambiguity and unstructured language, making them well suited to open-ended queries. But many business processes are not open-ended: they are structured, rule-based workflows that require specific inputs to produce a deterministic output.
Consider an employee trying to find out how much dental coverage they have. A team eager to use the latest tech might build a sophisticated chatbot. The employee asks, "What's my dental coverage?" The bot, using a Retrieval-Augmented Generation (RAG) pipeline, searches through hundreds of benefits documents and synthesizes an answer—but that answer might be slightly wrong, pulling from an outdated policy document or hallucinating a detail. The interaction is slow, expensive in terms of token usage, and carries a risk of error.
The better solution is often a simple form. The employee navigates to the benefits portal, selects "Dental Plan," and their coverage details are displayed instantly. This workflow is cheap, fast, and 100% accurate, because it pulls structured data directly from a database.
The mistake is confusing a user's desire for an answer with a need for a conversation. Before building a chatbot, ask:
Is the user's goal exploratory or transactional?
Can the required information be captured with a few structured inputs?
What is the business cost of a wrong or ambiguous answer?
If the task is transactional and the cost of error is high, a chatbot is often a complex, expensive, and brittle solution to a problem that traditional software already solved.
The Month Three Cost Surprise
Many pilots die a quiet death not from technical failure, but from sticker shock. A project is developed using a small, clean dataset and tested by a handful of enthusiastic users. The cost per query seems manageable. The team gets the green light to scale.
In month three of the rollout, the CFO calls. The AI bill is 100 times higher than projected. What happened?
Scale Happened: The pilot's 1,000 queries a day became 100,000 queries a day in production. The linear cost increase became an existential threat to the project's ROI.
Complexity Happened: Real-world users don't ask clean, simple questions. They submit long, messy documents. They upload images. They have meandering conversations. This dramatically increases the number of tokens—the basic unit of data processed by the model—and therefore the cost. The 320x year-over-year growth in API reasoning tokens per organization reported by OpenAI is a testament to this escalating complexity.
The 80/20 Rule Happened: The team discovers that 80% of user queries are simple, repetitive questions that could be handled by a much smaller, cheaper model, or even a cached database response. Instead, every query is being sent to the most powerful, expensive model available—like using a sledgehammer to crack a nut.
Cost is not a detail to be figured out later; it is a core architectural constraint. A successful AI strategy requires a multi-tiered approach, routing queries to the cheapest possible resource that can satisfactorily answer them. This "mixture of experts" approach—using a cascade of small, fast, cheap models and escalating to the large, powerful ones only when necessary—is a hallmark of the 5% of companies achieving real financial returns.
Evaluation as a Gate, Not a Practice
Perhaps the most subtle but pervasive failure of 2025 was the misunderstanding of evaluation. Too many teams treated "evals" as a final exam—a one-time benchmark test to be passed before launch. They would test their model against an academic dataset, get a good score, and declare it "ready."
This is like a car manufacturer testing a vehicle on a pristine track and then shipping it worldwide without ever testing it in snow, on hills, or in city traffic. The real world is messy: data drifts, user behavior changes, and new edge cases emerge daily. Evaluation isn't a gate you pass through once; it's a continuous practice that must be woven into the entire lifecycle of the AI system.
The Tyranny of the Benchmark
Academic benchmarks are essential for research, but they are a dangerously misleading proxy for business performance. A model might score 90% on a generic question-answering benchmark, but that score tells you nothing about its ability to handle your company's specific product names, your industry's jargon, or your unique internal policies.
Over-reliance on these benchmarks creates a false sense of security. Teams optimize for a test that doesn't reflect their actual use case, and when the system goes live, it fails in ways the benchmark could never have predicted: giving incorrect pricing information, misunderstanding a critical term of art, or failing to follow a multi-step instruction that is common in the company's real workflow. These failures erode user trust and can have direct financial consequences.
The successful 5% build their own benchmarks. They create high-quality evaluation sets from their own data, representing the hard, ambiguous, and business-critical cases their system will face in the wild.
A Worked Example: Building a Custom Eval
Let's walk through how a team building a customer support bot moves from a generic benchmark to a meaningful, custom evaluation practice.
Phase 1: The Generic Benchmark
The team starts by building a support bot using a powerful off-the-shelf model. To test it, they use a popular open-domain Q&A benchmark. The bot scores well, correctly answering questions about world capitals and scientific principles. The team concludes the model is "smart" and ready for a pilot.
Phase 2: The Production Failure
The bot is piloted with a small group of real customers. The results are disastrous.
A customer asks, "Can I use the Continuum Transfunctioner discount with my Series-B plan?" The bot, having never seen these proprietary product names, hallucinates an answer, promising a discount that doesn't exist.
A user asks, "My bill is higher than last month, why?" This requires checking the user's account, comparing two invoices, and identifying a specific new line item—a multi-step reasoning process. The bot provides a generic answer about price changes.
A customer asks for a refund, a "red-flag" query that should be immediately escalated to a human agent. The bot tries to handle it and fails, frustrating the customer.
Phase 3: The Custom Evaluation Suite
The team realizes their evaluation process was testing for the wrong thing. They go back to the drawing board and build a custom eval suite. This isn't a massive dataset; it's a curated collection of 200-300 examples representing their most important challenges.
They structure their test cases in a simple format, perhaps a JSON or CSV file, that includes:
[
{
"query_id": "PROD_JARGON_001",
"query": "Can I use the Continuum Transfunctioner discount with my Series-B plan?",
"category": "product_compatibility",
"ideal_answer_keywords": ["not compatible", "Series-C plan", "upgrade"],
"failure_mode_if_wrong": "Financial loss (invalid discount)",
"action_if_detected": "escalate_to_human"
},
{
"query_id": "BILL_ANALYSIS_001",
"query": "My bill from last month is higher, can you tell me why?",
"category": "multi_step_reasoning",
"ideal_answer_keywords": ["usage increase", "new service fee", "one-time charge"],
"failure_mode_if_wrong": "Customer frustration",
"action_if_detected": "log_for_review"
}
]This simple structure transforms their process. Now, every time they want to try a new model, a new prompt, or a change to their RAG system, they can automatically run it against this suite. They are no longer measuring "general intelligence"; they are measuring whether a change fixes the specific business problems they care about. They have moved from evaluation as a gate to evaluation as a practice.
Crossing the Divide
The GenAI Divide that defined 2025 isn't a technology gap—it's an organizational one. The billions companies poured into AI weren't wasted: they built real capabilities and delivered genuine personal productivity gains. OpenAI's report found that individual users saved 40 to 60 minutes a day using these tools. But for most companies, those saved minutes simply evaporated into general organizational busyness rather than showing up as a measurable line on the P&L—the profit-and-loss statement that tracks whether a business is actually making or losing money.
The 5% of firms that captured real value did so by treating AI as a business transformation project, not a technology project. Five habits set them apart:
They start with the P&L. They identify a specific, measurable business metric they want to move and work backward to the technology, ensuring every pilot has a clear path to value.
They solve for the workflow, not the task. They look beyond individual task completion and focus on deeply integrating AI into the multi-step, cross-functional processes where the real value lies. This is the "deeper workflow integration" OpenAI identified as a key driver of success.
They find a business owner. Every successful project has an executive sponsor outside of the IT department who owns the outcome and is accountable for its success or failure.
They build for cost. They design systems with cost as a primary constraint, using a cascade of models and tools to ensure the project remains profitable at scale.
They evaluate for reality. They abandon generic benchmarks and invest in building and maintaining custom evaluation suites that reflect the unique, messy reality of their business.
The story of 2025 is a necessary correction. The initial euphoria has been replaced by a pragmatic understanding of what it truly takes to make AI work in the enterprise. The technology is here. Now, the organizational work begins.