The Productivity Paradox of AI Coding

Ask almost any engineer who has spent a week with an AI coding assistant whether it made them faster, and the answer comes back instantly: obviously. The autocomplete finishes the thought before you do. The boilerplate writes itself. The Stack Overflow tab you used to keep open all day is gone. It feels like a different job.

Then you go looking for the productivity gain in the numbers, and it gets slippery. Cycle time barely moves. The backlog doesn’t shrink the way the vibe suggests it should. Some teams ship more; others ship the same amount with more rework. And in at least one carefully run experiment, the developers who were certain they’d been sped up had actually been slowed down — and didn’t notice.

This is the productivity paradox of AI coding: a persistent, sometimes large gap between how fast AI-assisted development feels and what measurement actually shows. The gap isn’t proof that AI doesn’t help. In plenty of contexts it clearly does. The point is narrower and more uncomfortable: feeling faster is not the same as being faster, and if you want to know which one you’re getting, you have to measure rather than assume.

A study where everyone was wrong about themselves

The cleanest illustration comes from a 2025 randomized controlled trial by METR. The setup was unusually rigorous for this field. Sixteen experienced open-source developers worked on real issues in repositories they knew intimately — mature projects averaging more than 22,000 GitHub stars and over a million lines of code, where each contributor had years of history. Across 246 tasks, the researchers randomly assigned whether each developer was allowed to use AI tools (primarily Cursor Pro with Claude 3.5/3.7 Sonnet) or had to work without them, then measured how long each task took.

The developers expected AI to speed them up by 24%. That’s a reasonable prior; it matches the general mood of the industry. After the study was over — after they’d lived through every task — they estimated AI had sped them up by about 20%.

Measured completion time went the other direction. Tasks done with AI allowed took 19% longer. The tool slowed them down, and the people using it believed the opposite, both before and after. That’s not a small calibration error. The sign was wrong. They were confident about the direction of an effect they had personally just experienced, and the direction was inverted.

It is worth being scrupulously fair about what this study does and does not show, because it has been wildly over-quoted in both directions. The METR authors are explicit that their result does not demonstrate that AI fails to speed up most developers, that it fails in domains outside software, or that there’s no way to use these tools more effectively in their exact setting. They flag plausible reasons the result might not generalize — chief among them that mastering a tool like Cursor may take hundreds of hours, and their participants were experienced engineers but not necessarily power users of that specific workflow. The headline isn’t “AI makes developers slower.” The headline is that on familiar codebases, with skilled engineers, the perceived effect and the measured effect pointed in opposite directions. The perception gap is the finding.

Why familiar codebases are the hard case

The result becomes less mysterious when you think about where AI assistance actually earns its keep. The friction it removes most reliably is the friction of not knowing: unfamiliar syntax, an API you haven’t touched, a language you’re rusty in, the blank-page problem of a greenfield file. An engineer who has spent five years in a codebase has very little of that friction to remove. They already know where everything is. What AI adds in that situation is a stream of suggestions that have to be read, evaluated, and frequently corrected — and reading and judging someone else’s code is slower than writing code you already have fully formed in your head.

So the same tool that is a genuine accelerant for a developer dropped into a strange repo can be a quiet tax for the expert in their home territory. Both experiences are real. Averaging them into a single “AI makes you X% faster” number erases the thing that actually matters, which is context.

There’s a second, subtler tax hiding in the same dynamic. When you write code yourself, you hold a continuous mental model of the system — you know why each piece is there because you put it there. When you delegate generation, you trade that continuous model for a series of review decisions, and review is a fundamentally different and more interruptive cognitive mode than authorship. You stop, read, judge, accept or reject, then try to recover your train of thought. METR’s participants reported spending real time prompting, waiting on generations, and reviewing output that didn’t quite fit — and in a codebase you know cold, every one of those steps is pure overhead against the baseline of just typing what you already intended to write. The feeling of momentum (“the code is appearing!”) and the reality of the clock (“I keep stopping to evaluate code”) can diverge precisely because the brain registers the appearance of progress more vividly than the accumulation of small interruptions.

The other studies — and why they don’t simply contradict it

If you only read the METR result, you’d conclude AI coding is oversold. If you only read GitHub’s research, you’d conclude the opposite. Both exist, and a fair treatment holds them at the same time.

In a controlled experiment GitHub ran, 95 professional developers were randomly split into two groups and asked to write an HTTP server in JavaScript. The group with Copilot finished in about 1 hour 11 minutes; the group without took about 2 hours 41 minutes — roughly 55% faster, a statistically significant result (P=.0017) with a 95% confidence interval on the speedup running from 21% to 89%. That is a large, real effect. The accompanying survey of thousands of developers found majorities reporting more focus, more fulfillment, and staying in flow.

These findings don’t cancel out. Look at what’s different. GitHub’s task was a self-contained, greenfield, well-trodden problem — exactly the regime where AI assistance is strongest, and exactly the kind of task on which AI has seen enormous amounts of training data. METR’s tasks were idiosyncratic issues buried in large, mature, unfamiliar-to-the-model codebases — the regime where assistance is weakest. The two studies aren’t arguing; they’re measuring different points on the same curve. The synthesis is not “AI helps” or “AI doesn’t.” It’s: the size and even the sign of the effect depend heavily on the task and the context, which is precisely why a single anecdote — or a single benchmark — tells you almost nothing about your team.

It also explains why the glowing testimonials feel so universal. The developer writing a delighted thread about how AI 10x’d their weekend project is telling the truth — about that project. The greenfield side-project, the unfamiliar framework, the prototype thrown together in an afternoon: these are the highest-uplift scenarios, and they’re also the most fun and the most shareable. The grinding maintenance task where AI added 19% of overhead doesn’t get a thread. That’s survivorship bias operating on your timeline: you see the wins because wins get posted, and you infer a uniform effect from a heavily filtered sample. (If you’re trying to read acceptance and uplift signals honestly rather than from vibes, we’ve written separately about why acceptance rate is a noisier metric than it looks.)

We’ve been here before: the Solow paradox

None of this is new, which should be oddly reassuring. In 1987 the economist Robert Solow delivered the line that named an entire field of inquiry: “You can see the computer age everywhere but in the productivity statistics.” Through the late 1970s and 1980s, companies poured money into computers while measured productivity growth stayed stubbornly flat. The technology was visibly transformative and statistically invisible. Erik Brynjolfsson formalized the puzzle in his 1993 paper “The Productivity Paradox of Information Technology,” and the resolution that eventually emerged is the part worth carrying forward.

The gains were real but lagged by years, for reasons that had little to do with the chips. Realizing them required reorganizing how work was done — new processes, new skills, new org structures — not just dropping a faster machine onto an old workflow. Productivity at the level that shows up in statistics is a property of systems, not of tools. A faster individual step doesn’t automatically produce a faster overall outcome, because the bottleneck usually lives somewhere the tool never touched: in review, in coordination, in waiting, in rework.

AI coding sits squarely in this tradition. The individual keystroke-to-working-code step has genuinely accelerated. Whether that converts into a team shipping more value depends entirely on whether the rest of the system can absorb the new pace — or whether the saved time just relocates downstream into review queues and bug triage.

It’s also worth noting that the Solow paradox eventually resolved. Productivity statistics did pick up in the late 1990s, once organizations had spent the better part of a decade rebuilding their processes around what computers were actually good at. That history cuts in two directions for AI coding. The optimistic reading is that today’s flat or negative measurements may simply be early — that the systemic gains are coming once teams figure out the new division of labor between human and model. The cautionary reading is that those gains were never automatic; they took years of deliberate reorganization, and the firms that just bought computers and changed nothing else got the cost without the benefit. The lesson isn’t “be patient” or “be skeptical.” It’s “the reorganization is the work,” and no amount of tooling does it for you.

Where the saved time goes

The cleanest evidence that the time doesn’t vanish — it moves — comes from delivery and code-quality data.

The 2024 DORA report (Google’s long-running Accelerate State of DevOps research) found a now-familiar split personality. AI adoption was associated with higher individual productivity, more flow, and better job satisfaction — the felt experience holds up. But at the delivery level, as reported in coverage of the findings, AI adoption correlated with an estimated 1.5% drop in delivery throughput and a 7.2% drop in delivery stability. The individual feels faster; the system delivers a little slower and breaks a little more. DORA’s own read is that AI tends to increase batch size — bigger changesets land per commit — and larger batches are riskier, a relationship their research has documented for years. Notably, 39.2% of respondents reported little to no trust in the AI-generated code they were shipping — even though documentation was flagged as one of the areas where AI showed the most promise.

GitClear’s analysis points at one mechanism behind the instability. Examining a large corpus of commits, their 2025 code-quality research found that the share of changed lines attributable to refactoring fell from around 25% in 2021 to under 10% in 2024, while copy-pasted (cloned) code rose from 8.3% to 12.3% over the same window. Their churn metric — the share of code reverted or reworked soon after it was committed — climbed from 3.1% in 2020 to 5.7% in 2024. Read together, that’s a picture of more code being generated and less of it being consolidated: easier to produce, harder to maintain. The keystrokes got faster and the cleanup got bigger.

This is the offsetting-cost structure the perception gap hides. When you write a line yourself, you’ve already paid the cognitive cost of understanding it; the understanding and the typing happen together. When you accept a generated line, the typing is free but the understanding is deferred — to review, to the first bug, to the engineer six months from now who has to modify code nobody fully reasoned through. The felt speedup is real and immediate. The cost is real and delayed, which is exactly the configuration that fools intuition. We are very good at noticing what just got faster and very bad at attributing a slow afternoon of debugging back to a fast morning of generating.

Why measurement is hard, and how not to fake it

If perception is unreliable, the obvious move is to measure. The trap is that the easiest things to measure are the things AI inflates most directly.

Lines of code, commit counts, the volume of accepted suggestions, raw token throughput — every one of these goes up with AI use almost by construction, and none of them tells you whether more value reached users. Optimize any of them and you’ll get exactly what you asked for: more lines, more commits, more accepted suggestions, and no necessary relationship to outcomes. That’s Goodhart’s Law — when a measure becomes a target, it stops being a good measure — and AI coding is unusually good at gaming output metrics because generating output is precisely what it does.

The discipline that emerged in response is the SPACE framework, published in ACM Queue in 2021 by Nicole Forsgren, Margaret-Anne Storey, and colleagues from Microsoft Research and GitHub. Its central argument is that developer productivity is irreducibly multidimensional and can’t be captured by any single number. SPACE spans five dimensions — Satisfaction and well-being, Performance, Activity, Communication and collaboration, and Efficiency and flow — and the explicit point is that you must read across them, because any one in isolation misleads.

The AI productivity paradox is, in a sense, a SPACE framework story told in miniature. The satisfaction and efficiency/flow dimensions light up green — people feel great, and they’re right to. But performance, the dimension that asks whether outcomes improved, can stay flat or slip while activity metrics scream success. A measurement program that only watches activity and satisfaction will conclude AI is an unambiguous win and miss the throughput and stability erosion entirely. We’ve made the case at more length for choosing metrics that actually correlate with outcomes rather than the ones that are merely easy to collect, and for the related habit of measuring your own coding productivity honestly before extrapolating to a team.

The practical version is unglamorous. Watch effort-to-outcome, not output volume: cycle time from idea to merged-and-stable, change failure rate, the share of merged code that survives without rapid rework. Pair quantitative signals with the qualitative — periodic, structured check-ins on where AI genuinely helped and where it added review burden, captured close enough to the work that memory hasn’t smoothed it into a uniformly positive blur. And, given the METR result, treat self-reported speedup as a feeling to be validated, not a measurement to be trusted. The whole reason the paradox exists is that the feeling and the fact can disagree, so don’t promote the feeling to the status of evidence. (None of this requires surveilling individuals or ranking people; the useful unit is the team and the trend, and it can be done in a privacy-respecting way.)

The Takeaway

AI coding tools are not a scam and they are not a panacea, and the productivity paradox is what it looks like when a technology is genuinely useful and routinely overestimated by the people using it. The METR trial showed experienced developers confidently wrong about the sign of their own speedup. GitHub’s trial showed a large, real acceleration on the right kind of task. The Solow paradox showed us, decades ago, that individual-level speedups don’t become system-level productivity until the surrounding work is reorganized to capture them. DORA and GitClear show where the unreorganized time goes — into bigger, riskier batches and into rework.

The honest position holds all of that at once. AI clearly helps in many contexts, especially unfamiliar territory and well-trodden greenfield work. It clearly helps less, and can even cost you, on familiar code where you were already fast. And crucially, you cannot tell which case you’re in from how it feels, because the feeling is the one thing the research consistently shows to be unreliable.

So the conclusion is not a verdict on AI. It’s a discipline. Stop asking your team whether AI made them faster and start measuring whether your system got faster — outcomes over output, trends over anecdotes, the boring DORA-style signals over the dopamine of a full autocomplete. The teams that win the next few years won’t be the ones who adopted AI hardest. They’ll be the ones who were honest enough to find out what it was actually doing.