Why AI ROI Studies Contradict Each Other and What to Believe Before You Invest

Between November 2024 and the middle of 2025, three heavyweight research groups published findings on what generative AI actually returns, and their answers point in three different directions. IDC's Business Opportunity of AI study, sponsored by Microsoft, reported that companies realize an average of $3.70 for every $1 they invest in generative AI. IBM's 2025 CEO study surveyed 2,000 chief executives and found that only 25% of their AI initiatives had delivered the ROI they expected. And MIT Project NANDA's report on the GenAI Divide concluded that roughly 95% of enterprise GenAI pilots produce no measurable P&L impact at all.

So within nine months, executives were told that generative AI nearly quadruples your money, that it disappoints three times out of four, and that it almost never shows up in the P&L. I've sat in pilot reviews where the same deck quoted two of these figures a few slides apart and nobody asked how both could be true. If you're about to sign off on an AI budget, that question deserves ten minutes of your attention.

The short answer is that all three findings can be true at once, about the same underlying reality, because each study counts something different. Once you see the mechanics, the studies stop contradicting each other and start telling you something genuinely useful about how to plan.

What each study actually measured

IDC and Microsoft: dollars, averaged, self-reported

The IDC study, published in November 2024, drew on interviews and surveys with more than 4,000 business leaders and AI decision makers, who were asked to estimate the returns their organizations were seeing. The headline is an average return per dollar invested, and the same study reports that the top tier of adopters realizes $10.30 per dollar. Hold onto that second figure, because it explains most of what follows.

IBM: initiatives, judged against expectations

IBM's Institute for Business Value surveyed 2,000 CEOs between February and April 2025 across 33 countries. Its unit of analysis is the initiative, and its bar is the CEO's own expectation. Only 25% of AI initiatives cleared that bar over the preceding few years, and IBM found just 16% had scaled enterprise-wide. The same CEOs are still leaning in, though. IBM reports that 85% of them expect their scaled AI efficiency investments to show positive ROI by 2027.

MIT: pilots, held to the P&L

MIT's NANDA group set the hardest test. The GenAI Divide: State of AI in Business 2025 drew on 52 structured interviews with enterprise leaders, a review of more than 300 public AI initiatives, and 153 survey responses, and it asked whether each pilot produced measurable P&L movement. About 5% did, despite what the report estimates as $30 to $40 billion in enterprise GenAI spend. The report also found that buying tools from external vendors succeeded around 67% of the time, while internally built tools succeeded about a third as often.

For what it's worth, McKinsey's State of AI survey from March 2025 lands much closer to MIT than to IDC. More than 80% of its respondents said their organizations were seeing no tangible impact on enterprise-level EBIT from generative AI.

The denominator is doing most of the work

IDC divides by dollars, IBM divides by initiatives, and MIT divides by pilots. That choice alone does more to set each headline than anything happening inside the companies they studied.

A dollar-weighted average is dominated by whatever the biggest winners did, because large deployments carry large spend and their returns swamp the mean. An initiative judged against expectations really measures disappointment, which is a strange metric once you notice it. A project that returned real money but less than the sponsor promised counts as a failure, while a modest project that met modest goals counts as a success. And pilots sit at the earliest, cheapest, most numerous stage of the funnel, so pilot statistics will always look bloodier than initiative statistics, the same way seed-stage startups fail more often than companies raising a Series C.

It's also worth noticing what a dying pilot does inside each study. In MIT's data, it lands squarely in the 95%. In IBM's survey it may never register, because a CEO asked about initiatives probably has no idea a regional team quietly shut down its six-week experiment. And in IDC's per-dollar average it shows up as a small amount of spend that barely dents a mean built on much larger deployments.

Self-reported returns and the survivorship problem

IDC's 3.7x is what leaders say they're getting. MIT's 95% is what researchers could actually find in the financials. Those are different instruments pointed at the same building. I ran product engineering and AI teams at American Express for years, and self-assessed ROI estimates were reliably the softest numbers that crossed my desk. Nobody lowballs the return on a project they sponsored. And most productivity ROI gets built by multiplying minutes saved by loaded salary, which produces a figure that never reaches any line finance recognizes unless hours, headcount, or output actually change.

Then there's the sample. A vendor-sponsored study surveys organizations that adopted the technology, kept using it long enough to form a view, and agreed to talk about it. Companies that shut everything down in month four are thin on the ground in that dataset. The resulting average can be perfectly honest and still describe only the survivors. It also matters, plainly, that the sponsor sells the thing being measured. I don't think anyone fabricated anything. Selection does the work on its own.

IBM's figure carries the opposite bias. It measures projects against expectations that were often set at peak enthusiasm, sometimes by vendors quoting studies like IDC's, so inflated expectations manufacture disappointment even where real returns exist. MIT's bar of measurable P&L impact is the strictest of the three, which is exactly why I weight it most heavily when I'm planning budgets rather than defending them.

A few big winners carry the average

Here's the mechanism that lets a 3.7x mean coexist with a 95% failure rate, in the form of a deliberately made-up portfolio. Say a company runs ten AI initiatives at $300,000 each, $3 million in total. Seven produce nothing measurable. Two return about $550,000 apiece. One, maybe an intake automation that eliminates an entire processing queue, returns $10 million over its life. Total return comes to $11.1 million on $3 million invested, which is exactly a 3.7x average return per dollar. It's also a portfolio where the median initiative returned zero and seven out of ten pilots failed. Every headline above describes this one portfolio honestly.

IDC's own top-cohort figure hints that the real distribution has this shape. You don't get a 3.7x mean sitting alongside a $10.30 top tier unless the bottom of the distribution is wide and flat. None of this is unusual for technology bets, and venture capital has run on exactly this shape for decades. But an executive budgeting a single project against the mean of a skewed distribution is making a specific, avoidable mistake.

How to read the next AI ROI headline

Six questions sort nearly every study I've seen, and none of them require a statistics background.

• Who paid for the research, and do they sell what's being measured? Sponsorship rarely means fabrication. It reliably means selection.
• What's the denominator? Dollars, initiatives, pilots, or companies. Same reality, different numbers.
• Is the return self-reported or measured? An executive's estimate and an audited cost line are different species of evidence.
• What counts as a return? Estimated time savings inflate easily. P&L movement doesn't.
• Who is in the sample? Adopters and survivors only, or everyone who tried?
• Does it report a mean or a median? In a field this skewed, a mean mostly tells you about the winners.

Plan against the median, and name the line

Two planning rules fall straight out of the arithmetic above.

First, plan against the median initiative, not the mean return. By MIT's count and McKinsey's, the most likely outcome for any single project is no measurable P&L movement. That argues for funding a portfolio instead of a flagship: several small, time-boxed bets with explicit kill criteria, sized so the write-offs don't hurt. Expect the return to come from one or two of them, and treat anything like 3.7x as a result you report afterward, never an assumption you plan with. When we review pilot lists at FirmAdapt, swapping mean-case budgeting for median-case budgeting changes more funding decisions than any technical finding does.

Second, pre-define the P&L line the project is supposed to move. Before money moves, the sponsor writes down five things: the specific line item, the direction, the size, the date, and the person in finance who owns the baseline and will confirm the result. Say your AP team processes 6,000 invoices a month at roughly $8 of touch labor per invoice. The commitment reads something like: cost per invoice on the finance operations line, from $8 to under $5 within two quarters, confirmed by the controller. Now the pilot has a real test, and finance holds a baseline captured before the tool arrived, so nobody argues about attribution later.

If the team can't name the line, treat the project as research. Fund it from an R&D envelope, cap it at R&D size, and put a revisit date on it. Plenty of good work starts that way. The failure mode I keep running into is R&D-shaped projects wearing ROI-shaped budgets.

Two smaller expectation-setters from the MIT data are worth folding into your planning. Buying beat building by a wide margin, roughly 67% success for purchased tools against about a third of that for internal builds, which for a mid-market company without a research bench is a strong prior toward buy-and-integrate. And the pilots that crossed into P&L impact tended to be wired into an actual workflow, with feedback loops and an outcome defined before the build, rather than a chat window bolted on next to the real work. McKinsey's survey points the same way. It found workflow redesign had the biggest effect on whether organizations saw EBIT impact from generative AI.

What to do Monday morning

1. Pull the list of every live AI pilot and initiative. Ask each owner which P&L line the project moves and who in finance holds the baseline. Accept two-sentence answers only.
2. Reclassify anything without an answer as R&D. Cap the spend and set a decision date. Ninety days is usually about right.
3. Require the next proposal to carry a median-case business case. Assume nothing measurable lands for two quarters and check that the budget survives it. The upside case goes in a footnote.
4. When a funding memo cites an ROI study, ask for the sponsor and the denominator before the discussion continues. It takes thirty seconds and filters evidence better than an hour of debate.

None of this argues against investing. Every study above, including the grimmest, found a cohort making real money, and that cohort behaves in recognizable ways. It buys more than it builds, it wires tools into actual workflows, and it names the outcome before the work starts. So treat IDC's 3.7x as a description of what winning portfolios looked like after the fact, and budget as if most of your pilots will land in MIT's 95%. If one winner can pay for the rest, and every project has a named line with a named owner in finance, you've already extracted the only advice all three studies agree on.