Token Leaderboards Are a Trap

Picture the dashboard. A glowing leaderboard, refreshed every hour, ranking your engineers by tokens consumed this week. Maya is at the top with 4.2 million. Devin is dead last with 180,000. The implication is obvious to everyone who glances at it: Maya is crushing it with AI, and Devin is a laggard who needs a talking-to.

There is one problem. Maya might be the laggard. She could be the developer who pastes an entire 2,000-line file into the model for every trivial question, lets an agent loop on a problem it solved twenty minutes ago, and ships verbose generated code she never reads. Devin might be the one who writes a tight twelve-line prompt, gets exactly what he needs, edits it, and moves on. The leaderboard cannot tell the difference. It was never built to.

This is the quiet disaster waiting inside a lot of AI-adoption dashboards. The instinct to count something is healthy. AI-assisted development has been invisible for too long, and visibility is genuinely useful. But the moment you take the easiest thing to count — tokens, accepted suggestions, “AI usage” — and turn it into a ranking that people are judged by, you have built a machine that manufactures exactly the behavior you do not want.

The two laws you are about to break

There are two well-worn principles from social science that explain precisely how this goes wrong, and they are worth knowing by name because they will save you from a year of expensive mistakes.

The first is Goodhart’s Law, usually stated as “when a measure becomes a target, it ceases to be a good measure.” The economist Charles Goodhart’s original 1975 formulation was drier and more precise: “Any observed statistical regularity will tend to collapse once pressure is placed upon it for control purposes.” Token count is an observed statistical regularity. It correlates, loosely, with people doing AI-assisted work. The instant you put pressure on it — rank people by it, reward it, shame the bottom — the correlation collapses, because people optimize the number instead of the thing the number was standing in for.

The second is Campbell’s Law, articulated by the social psychologist Donald T. Campbell in the 1970s: “The more any quantitative social indicator is used for social decision-making, the more subject it will be to corruption pressures and the more apt it will be to distort and corrupt the social processes it is intended to monitor.” Campbell was writing about standardized testing and crime statistics, but he could have been writing about your engineering org. The distortion is not a risk. It is the predicted, default outcome of using a single number to make decisions about people.

Both laws point at the same trap. A metric that is merely informative becomes corrupting the moment it becomes consequential. And a public ranking is about as consequential as a metric gets.

What token-maxxing actually looks like

Give a team of clever engineers a leaderboard to climb and they will climb it. They are professionally good at reverse-engineering systems, and your scoring function is just another system. Here is the menu of behaviors a token-or-usage leaderboard reliably produces.

Prompt padding. Why ask a focused question when a bloated one scores higher? Dump the whole repo into context. Restate the problem three ways. Paste logs nobody will read. Every extra token is “engagement.”

Accepting verbose output you would never write by hand. The fastest way to rack up accepted-lines or token credit is to stop reading the output and start rubber-stamping it. This is the single most dangerous behavior on the list, because it is invisible until it ships.

Letting agents idle and loop. An autonomous agent left running burns tokens whether or not it is making progress. On a token leaderboard, a stuck agent spinning in circles for an hour looks identical to productive work — better, even, because it consumed more.

Choosing the chatty workflow over the right one. Some tasks are best done with a quick model call. Some are best done in your head, or with a search, or by reading the docs. A leaderboard quietly biases every one of those decisions toward whichever path generates the most tokens.

The unifying property of all four behaviors is that they are cheaper to fake than to earn. Climbing the board does not require shipping anything; it requires generating volume, and volume is the one thing a language model will hand you for free, all day, in unlimited supply. You have built a contest whose winning move is to point a fire hose at the scoreboard. The engineers who refuse to play — who keep prompting tightly and reading carefully — don’t just fail to win. They visibly lose, in public, to colleagues producing more slop. That is the precise inversion you should fear: the leaderboard does not merely fail to reward good work, it actively penalizes it.

None of these make better software. Several of them make demonstrably worse software. And we have data on what “worse” looks like.

GitClear’s 2025 AI Copilot Code Quality study, which analyzed 211 million changed lines of code authored between January 2020 and December 2024, found that the share of “copy/pasted” cloned lines rose from 8.3% in 2021 to 12.3% in 2024, while lines associated with refactoring sank from 25% of changed lines in 2021 to under 10% in 2024. GitClear flagged 2024 as the first year in its dataset where copy/pasted code outnumbered “moved” lines — its proxy for code reuse. That is the macro signature of accepting output instead of integrating it: more duplication, less reuse. A leaderboard that rewards raw AI volume pours fuel directly onto that fire.

The deeper irony: more usage is not more productivity

Here is the part that should stop you cold before you ship that ranking. Even if you could measure AI usage perfectly and honestly, with zero gaming, it still would not be a proxy for productivity. The link you are assuming exists may not exist at all.

In 2025, METR ran a randomized controlled trial on 16 experienced open-source developers working on their own large repositories — 246 real tasks, randomized to allow or disallow AI tools. The developers expected AI to make them about 24% faster. After doing the work, they believed it had made them about 20% faster. The measured result: they were 19% slower when allowed to use AI. They were not just wrong about the size of the effect. They were wrong about the direction, and they stayed wrong even after living through it.

Sit with what that means for a usage leaderboard. The developers in METR’s study could not accurately perceive their own AI productivity from the inside, in a controlled setting, on tasks they understood deeply. A manager reading a tokens-consumed ranking from the outside has far less information and a far stronger incentive to draw the convenient conclusion. The number on the board and the value delivered are not weakly correlated. In at least one careful study, they pointed opposite directions.

This is not an argument that AI tools are bad — METR is explicit that it captures a specific snapshot of early-2025 tooling in a specific setting, and capability is moving fast. It is an argument that “used the AI more” and “was more productive” are different claims, and a leaderboard silently asserts they are the same. (We dug deeper into which signals actually track outcomes in the AI metrics that actually matter.)

The people problem is worse than the data problem

Suppose you do not care about any of this. Suppose you say, “I know the metric is noisy, but a little competition is healthy and it gets people using the tools.” That instinct is not crazy — gamification can genuinely build habits, which is exactly why we have written about streaks and gamification done well. The problem is what a forced ranking by a proxy metric does to a team specifically.

The DORA research program — the most rigorous long-running body of work on software delivery performance — issued a direct caution in October 2023 against using its own metrics to compare teams. Its official guidance is that the goal is to improve your team’s performance over time, not to compete against other teams, because creating league tables leads to unhealthy comparison and counterproductive competition, and ranking teams ignores context. If the people who built the gold-standard delivery metrics tell you not to build league tables out of their carefully validated four metrics, you should be extremely suspicious of building one out of token count.

Context is the killer word there. A platform engineer untangling a flaky deployment pipeline and a feature developer scaffolding CRUD endpoints will have wildly different AI footprints, and neither tells you who is more valuable that week. A senior reviewing and rejecting bad AI output — arguably the highest-leverage work in an AI-assisted org — registers as low usage. A junior happily accepting everything registers as high usage. The leaderboard inverts the value hierarchy and broadcasts the inversion to the whole company.

And there is a trust cost that compounds. When developers realize they are being ranked by a number they can game, two things happen: the conscientious ones feel surveilled and resentful, and the strategic ones start gaming. You end up training your best engineers to produce theater for the dashboard instead of value for the codebase. We have seen how badly metric-driven mandates can backfire when they collide with developer judgment, which is its own resistance dynamic worth understanding.

Healthy curiosity versus toxic ranking

None of this means measurement is the enemy. The distinction that matters is between visibility and ranking — between giving people a mirror and forcing them onto a podium.

Healthy curiosity sounds like: “Huh, I used three times the tokens this week — oh right, I was doing that big migration.” It is private, self-directed, and diagnostic. You look at your own trend, you notice something, you adjust. Nobody is judging you and you are not optimizing for anyone’s gaze.

Toxic ranking sounds like: “I’m 14th out of 30 and my manager can see it.” It is public, comparative, and consequential. It tells you nothing diagnostic — 14th at what? — but it tells you exactly one actionable thing: produce a bigger number. That is the Goodhart trap, fully sprung.

The same underlying data — your token usage, your acceptance patterns, your model mix — can power either experience. What flips it from useful to corrosive is not the data. It is whether the comparison is imposed or chosen, and whether the thing being compared is an input you can pad or an outcome you have to actually achieve.

What to measure instead

If the goal is genuinely to understand and improve AI-assisted development — not to surveil it — there is a better playbook, and most of it follows from taking Goodhart, Campbell, and DORA seriously. The throughline is simple: stop ranking individuals by inputs they can pad, and start watching trends and outcomes that only move when real work happens. Four principles get you most of the way there.

Move the unit from the individual to the team

Almost every gaming incentive evaporates when the unit of comparison stops being a person. Team-level signals — how is AI adoption trending across the squad, are we shipping at a healthy cadence, is review throughput keeping up with generation — answer the questions leadership actually has without putting any single developer’s name next to a rankable number. The DORA guidance is unambiguous here: track your own trajectory over time, not a head-to-head against your neighbor. A trend line for the team is a thermometer. A ranking of teammates is a weapon.

The shift also changes what a number even means. An individual’s token count begs to be read as a verdict on that person. A team’s adoption curve reads as a description of a system you are all responsible for improving together. The first invites blame; the second invites the question that actually helps — what is getting in the way, and what would unblock it. You want the metric to start conversations, not end careers.

Make individual comparison opt-in, private, and self-directed

People are genuinely curious about their own patterns, and that curiosity is worth feeding. The fix is consent and privacy: let a developer see their own data, compare against their own past, and choose whether to surface anything. Opt-in self-comparison scratches the competitive itch for the people who enjoy it — there is a real, fun version of this, which is partly why the Strava-for-developers analogy resonates — without conscripting the people who would find a forced ranking demoralizing. Strava works because you choose to share your run. It would be a nightmare if your employer posted everyone’s pace by default.

Anchor on outcomes, not inputs

Tokens, lines, and suggestions are inputs. They are trivially paddable, which is exactly why they make terrible targets. Outcomes are harder to game because gaming them requires actually doing the work: did the change ship, did it stay shipped, did defects go up or down, did review latency hold. This is the whole spirit of the SPACE framework, published in ACM Queue in 2021 by Nicole Forsgren and colleagues, whose central claim is that productivity is multidimensional and cannot be captured in any single number. SPACE deliberately spans satisfaction, performance, activity, communication, and efficiency precisely so that no one dimension can be gamed without the others exposing it. A token count is pure “activity” — one fifth of one framework, and the easiest fifth to fake.

Hold every velocity-flavored number in tension with a quality number

If you are going to look at how much AI-generated code is flowing, look at it next to churn and duplication, not in isolation. GitClear’s data is the cautionary tale: volume went up while reuse went down. A single metric pointed at speed will always drift toward producing more, faster, worse. Pairing it with a counterbalancing signal is how you keep the system honest, which is the same logic that makes a code-quality gate in CI/CD more useful than any acceptance-rate dashboard.

The Takeaway

Token leaderboards fail not because counting is wrong, but because ranking people by a paddable proxy is a near-perfect recipe for the failure modes that Goodhart and Campbell described decades ago. The measure becomes a target, the target gets gamed, and you end up with bloated prompts, idle agents, rubber-stamped output, and a top performer who is quietly your worst contributor. Worse, the underlying premise is shaky to begin with: METR’s randomized trial found experienced developers were 19% slower with AI even as they believed they were faster, so “more usage” is not even reliably “more productivity.” More usage is just more usage.

The good news is that the better path is not harder, only humbler. Move comparison to the team level. Make individual reflection opt-in and private. Anchor on outcomes you cannot fake, and always hold a velocity number in tension with a quality one. Give people a mirror, not a podium.

Visibility into AI-assisted development is worth having — that is the entire reason to instrument it. Just don’t take the most visible number and turn it into a contest. The leaderboard that ranks your developers by tokens is not measuring who is best at the work. It is measuring who is best at climbing the leaderboard. Those are different skills, and only one of them ships software.