How to Actually Benchmark AI in a Dev Team

Most teams “measure” AI’s impact by asking people how it feels. They run a survey, get a warm number back — “developers say they’re 30% faster” — and put it in a board deck. That number is almost certainly wrong, and the most rigorous evidence we have says it is wrong in a specific, predictable direction: people overestimate the help.

In a 2025 randomized controlled trial, METR had experienced open-source developers complete real tasks on their own mature repositories, with AI tooling randomly allowed or disallowed. Before starting, the developers forecast that AI would make them about 24% faster. After finishing, they still believed AI had sped them up by roughly 20%. The measured result was the opposite: tasks took 19% longer with AI allowed. The perception gap didn’t close even after they lived through it.

That is the whole problem in one study. If your benchmarking strategy rests on self-reported time savings, you are measuring a feeling that doesn’t track reality. This piece is about how to do better — how to set up an evaluation an engineering leader can actually run, with a real baseline, the right indicators, and enough statistical hygiene to survive a skeptical CFO.

Start with the question you’re actually asking

“Is AI helping?” is not a measurable question. It’s a bundle of at least four separate ones, and conflating them is how evaluations go sideways:

• Are we shipping faster? (throughput, cycle time)
• Are we shipping worse? (rework, change failure rate, review burden)
• Are developers having a better time? (satisfaction, friction, flow)
• Is it worth the cost? (tokens, licenses, time spent prompting and reviewing)

You need to answer all four together, because each one alone is gameable. Speed without a quality check rewards slop. A satisfaction survey without delivery data rewards tools that feel good and produce churn. The frameworks worth building on were designed precisely to resist single-number thinking. We’ve made the broader case for this in why the right AI metrics matter — here the focus is the experimental design.

Build the baseline before you touch anything

You cannot measure a change you never characterized. The single most common evaluation failure is starting to track metrics the same week you roll out the tool, then comparing “now” to a half-remembered “before.” Establish at least four to eight weeks of pre-AI baseline data first, and capture the distribution, not just the average. Cycle time of “3 days on average” hides a world of difference between a tight team and one with a long tail of stuck pull requests.

Baseline these before rollout:

• Cycle time — first commit to merged-and-deployed, broken down by stage (coding, review wait, CI, deploy).
• Review latency — time a PR waits for first review and time to approval.
• Rework rate / code churn — the share of code rewritten or reverted shortly after it lands.
• Change failure rate — the percentage of deployments that cause a production incident, rollback, or hotfix.
• Throughput — PRs or changes merged per engineer per week, with size noted.
• Developer sentiment — a short, consistent pulse survey you’ll repeat unchanged.

The point of the baseline isn’t a vanity scorecard. It’s the counterfactual. Everything after rollout is measured against this, and against a control group, which we’ll get to.

The frameworks: DORA, SPACE, DevEx

Don’t invent your metric taxonomy from scratch. Three well-validated frameworks already cover the ground, and using them keeps your evaluation legible to people who’ve seen this before.

DORA: the four keys

The DevOps Research and Assessment program established four metrics that, taken together, predict software delivery performance. Per Google Cloud’s DORA program, the four keys are:

1. Deployment frequency — how often you successfully release to production.
2. Lead time for changes — how long a commit takes to reach production.
3. Change failure rate — the percentage of deployments causing a failure in production.
4. Time to restore service — how long it takes to recover from a failure.

The first two measure velocity; the last two measure stability. That pairing is the whole point: DORA is built so you cannot improve one half by quietly wrecking the other. This is exactly the discipline an AI evaluation needs, because AI’s failure mode is shipping more, faster, with more defects.

DORA’s own research makes the warning concrete. In the 2024 Accelerate State of DevOps report, the team estimated that a 25% increase in AI adoption was associated with an estimated 1.5% decrease in delivery throughput and a 7.2% decrease in delivery stability — even as roughly 75% of respondents reported that AI made them more productive. The report’s read is that AI tends to increase batch size, and larger changesets carry more risk. If you only tracked the productivity feeling, you’d have missed the stability cost entirely.

SPACE: productivity is multidimensional

The SPACE framework, introduced by Nicole Forsgren and colleagues from Microsoft Research, GitHub, and the University of Victoria, exists to kill the one-metric habit. Its five dimensions are:

• Satisfaction and well-being
• Performance (outcomes and quality of the work)
• Activity (counts of outputs — commits, PRs, reviews)
• Communication and collaboration
• Efficiency and flow

SPACE’s central rule: never measure productivity with a single metric, and never with activity alone. Activity counts are the easiest to game and the least meaningful. An AI tool can trivially inflate commits and lines of code while degrading everything that matters. A credible benchmark samples from at least three SPACE dimensions at once.

The reason this matters for AI specifically is that AI tools push hard on the activity dimension while leaving the others ambiguous. More code gets generated, more pull requests open, more lines change — all the countable outputs go up. Whether performance (does the work hold up?), satisfaction (do developers actually prefer this?), and efficiency (is the end-to-end path smoother?) also went up is a separate question that activity counts cannot answer and will actively obscure. A benchmark that leans on activity is the one most likely to mistake motion for progress, which is precisely the mistake AI tooling makes easy to commit.

DevEx: where the friction actually lives

The DevEx framework — by Abi Noda, Margaret-Anne Storey, Nicole Forsgren, and Michaela Greiler — distills developer experience to three dimensions: feedback loops (how fast you learn whether your work is good), cognitive load (how hard it is to get things done), and flow state (whether you can work without interruption). This matters for AI specifically because the supposed win is less cognitive load and tighter feedback loops — and that’s a hypothesis you can test directly rather than assume. The Stack Overflow 2025 Developer Survey found the dominant frustration, cited by 66% of developers, was “AI solutions that are almost right, but not quite,” which then forces extra debugging time. That’s a feedback-loop and cognitive-load regression hiding inside a tool sold as the opposite.

Leading vs lagging indicators

Split your metrics into two buckets and read them differently.

Leading indicators move fast and tell you whether adoption is even happening: AI tool usage rate, share of sessions with AI assistance, suggestion acceptance rate, prompts per task. These are diagnostic, not outcomes. High acceptance with rising rework is a red flag, not a win. We dig into the nuance of read-rate metrics in the acceptance-rate guide — the short version is that acceptance is an input you watch, never a goal you chase.

Lagging indicators move slowly and tell you whether anything actually got better: cycle time, change failure rate, rework rate, throughput per engineer, retention, satisfaction. These are your verdict. They lag by weeks because real delivery effects take real time to show up, which is exactly why you need that pre-rollout baseline and a patient measurement window.

The trap is declaring victory on leading indicators. “80% of the team uses AI daily and acceptance is high” is an adoption story, not an impact story. The Stack Overflow survey is a useful reality check here: 84% of developers now use or plan to use AI tools, yet trust in AI accuracy fell to around 33%, with more developers actively distrusting output than trusting it. Adoption and benefit are different variables. Measure both.

There’s a subtler reason to keep the buckets separate: they update on different clocks, and mixing them produces false confidence. Leading indicators spike the week a tool lands — everyone tries it, acceptance looks great, the Slack channel fills with enthusiasm. Lagging indicators haven’t moved yet because the code those sessions produced hasn’t been reviewed, merged, deployed, or had a chance to break in production. If you report in week two, you’ll see only the leading spike and conclude the rollout is a triumph. The honest read requires waiting for the lagging signal to catch up, then checking whether the early enthusiasm translated into anything a customer or a CFO would notice. When the two diverge — adoption high, delivery flat or worse — that divergence is itself the most important finding, and it’s the one a survey will never surface.

Stop doing naive before-and-after

Here’s where most evaluations quietly break. You measure cycle time in March (pre-AI), roll out the tool, measure again in June, see an improvement, and attribute it to AI. The problem: a dozen other things changed between March and June. You hired two seniors. A gnarly legacy migration finished. You’re now mid-quarter instead of post-holiday. The team got more familiar with the codebase. Any of these could move cycle time more than the tool did.

A simple before/after comparison cannot separate the AI effect from everything else that drifted. To get a defensible answer, borrow two ideas from applied econometrics.

Use a control group (cohort design)

Roll AI out to one group of teams and not another, matched as closely as you can on size, seniority mix, and the kind of work they do. Both cohorts experience the same hiring waves, the same incident-heavy weeks, the same seasonal slumps. If the AI cohort’s cycle time improves relative to the control cohort, you have something. If both improve identically, your “AI win” was just the company getting better at shipping.

This isn’t always clean — teams differ, and you can’t always randomize. But even an imperfect control is dramatically better than no control. The METR study got its surprising result precisely because it randomized AI on a per-task basis, holding the developer and the repository constant. Without that control, the developers’ own confident perception would have been the only “data,” and it pointed the wrong way.

Think in difference-in-differences

The cohort design naturally leads to a difference-in-differences mindset: measure the change in the treated group, measure the change in the control group, and the AI effect is the difference between those two differences. It strips out anything that affected both groups equally — seasonality, company-wide process changes, macro conditions. You don’t need a statistics PhD to apply the logic; you need two comparable groups and the discipline to compare trends, not levels. We walk through assembling these views in the team dashboard guide.

Control for the confounders you can’t randomize away

Even with a control group, name and watch the variables that distort comparisons:

• Team composition. Seniority changes everything. The METR finding involved experienced developers on familiar codebases — the population where AI’s advantage is smallest because the human already knows the answer. A team of juniors on an unfamiliar stack may show a very different result. Don’t generalize one cohort’s number to the whole org.
• Project phase. Greenfield work, steady feature delivery, and legacy maintenance have wildly different baseline cycle times. Comparing a team starting a new service against one untangling a five-year-old monolith tells you about the work, not the tool. Match cohorts on phase or segment your analysis by it.
• Seasonality and calendar effects. Velocity sags around holidays, end of quarter, and big incidents. A four-week window that happens to straddle a freeze will lie to you. Use windows long enough to absorb noise, and align them across cohorts.
• The Hawthorne effect. People behave differently when they know they’re measured. A fresh tool plus visible scrutiny produces a temporary bump that fades. Plan for a measurement window long enough to see whether early gains hold.
• Goodhart’s Law. “When a measure becomes a target, it ceases to be a good measure.” If you tell the team you’re grading them on AI acceptance rate or commit count, you’ll get more of exactly that — and learn nothing. Measure for understanding, not for ranking individuals.

What to measure, and how to read it together

No single metric is the verdict. The signal is in the pattern across them. Track these five as a panel:

• Cycle time — is the work moving faster end to end? Watch the stages; AI often shifts time from coding into review.
• Review latency and review burden — are PRs getting bigger or harder to review? AI can quietly push cost downstream onto reviewers.
• Rework rate / churn — is code being rewritten or duplicated rather than refactored? GitClear’s analysis of 211 million changed lines found copy/pasted lines rose from 8.3% in 2021 to 12.3% in 2024 while the share of changed lines tied to refactoring fell from around 25% to under 10% — and the number of code blocks with five or more duplicated lines jumped roughly 8x during 2024. Rising duplication and falling refactoring is the classic signature of speed bought with quality.
• Change failure rate — are more deploys breaking? This is your stability backstop, and DORA’s data says it’s the metric most at risk under AI.
• Throughput per engineer — more changes shipped, normalized for size. Throughput that rises only because changesets got bigger is not a win; it’s risk accumulating.

Read them as a constellation. Faster cycle time with flat-or-falling churn and a stable change failure rate is a genuine win. Faster cycle time with rising churn and more failed deploys is the slop pattern — you’re shipping faster and paying for it twice. Satisfaction up while delivery is flat might still be worth it for retention, but call it what it is. The combinations matter more than any single arrow. For turning this panel into an ongoing operating rhythm rather than a one-off study, see our work on the KPIs worth tracking.

A 90-day evaluation you can actually run

1. Weeks 1–4: Baseline. Instrument the five-metric panel and a short sentiment pulse. Define cohorts: a treated group and a matched control. Pre-register what “success” means before you see results, so you can’t move the goalposts.
2. Weeks 5–8: Rollout. Give the treated cohort the tool and real enablement. Track leading indicators (usage, acceptance) to confirm adoption is happening — a tool nobody uses can’t be evaluated.
3. Weeks 9–12: Measure and compare. Compute the change in each lagging metric for both cohorts. Look at the difference between those changes. Pair the quantitative panel with qualitative signal from the DevEx dimensions — where did friction actually move?
4. Decide honestly. If the treated cohort improved relative to control across the panel without quality regressions, expand. If gains were illusory or quality slipped, adjust the workflow — not just the verdict — and re-run. The output is a decision with evidence behind it, not a number for a slide.

The Takeaway

Benchmarking AI in a dev team is not hard because the metrics are exotic — DORA, SPACE, and DevEx have been stable and public for years. It’s hard because the honest version requires you to resist the easy answer. The easy answer is a survey that tells you what you want to hear, and the best evidence we have says that answer is systematically wrong: developers in the METR trial felt 20% faster while measurably running 19% slower.

Do the harder thing. Set a real baseline. Run a control group. Compare trends, not levels. Watch velocity and stability together so you can’t fool yourself by shipping more slop faster. Treat acceptance rates as a thermometer, never a target. The teams that get real, durable value from AI tooling aren’t the ones with the loudest enthusiasm — they’re the ones who measured honestly enough to know which workflows actually paid off, and quietly killed the ones that didn’t.