Ask whether AI coding tools make developers more productive and you’ll get two opposite numbers. One study says experienced developers got 19% slower with AI. Another says engineers who adopted AI agents merged 24% more pull requests. Both are serious work. So which one’s right?

We lined up four studies: METR’s two randomized trials, Microsoft’s internal rollout study, and a telemetry report from engineering analytics company Faros. Short version: most of the disagreement comes down to what each one measured. And once you overlap them, a fairly clear pattern shows up.

What the “19% slower” trial actually measured

The starting point is METR’s July 2025 trial. Sixteen open-source developers worked through 246 real issues in large repositories they already contributed to, with AI randomly allowed or disallowed per issue. The repos averaged about 10 years old and over 1.1 million lines of code, and the developers had roughly five years on their projects. They mostly used Cursor Pro with Claude 3.5 and 3.7 Sonnet.

Allowing AI made tasks take 19% longer (95% CI +2% to +39%). The perception gap is the wild part. Before starting, developers expected AI to cut their time by 24%. Afterward, they still thought it had cut their time by 20%. They got slower and felt faster.

The paper digs into why. Developers accepted fewer than 44% of AI generations. In a subset of screen-recorded tasks, about 9% of their time went to reviewing and cleaning up AI output and about 4% to waiting on it. AI helped less when developers knew the repo really well and when a lot of the context was undocumented. So what this trial measured is how long it takes an expert to finish one issue in a huge codebase they know inside out.

The follow-up points to “faster,” but METR doesn’t trust it

METR ran a bigger round starting in August 2025 and published results in February 2026: 57 developers, 143 repos, more than 800 tasks. The 10 returning developers saw task time drop 18% (CI −38% to +9%), and the 47 new recruits saw a 4% drop (CI −15% to +9%). Both intervals include zero, so “no effect” isn’t ruled out.

METR’s bigger point was about measurement. Between 30% and 50% of surveyed developers said they skipped submitting some tasks because they didn’t want to do them without AI. That removes exactly the tasks where AI helps most, so METR calls its estimate a lower bound and says the results are “likely a bad proxy” for real productivity impact. It’s redesigning the study.

Some coverage read this result backwards. METR wrote it as “a speedup of −18%,” so outlets like Particula reported it as 18% slower. But METR’s own text says “our raw results show some evidence for speedup” and that developers are likely more sped up now. So we read it as 18% faster. When one post can produce opposite headlines, it’s worth checking the source yourself.

What the “24% more PRs” study measured

Three Microsoft researchers posted a paper to arXiv in July 2026 on the company’s internal rollout of Claude Code and GitHub Copilot CLI, covering about four months (Jan. 5 to Apr. 29, 2026) and tens of thousands of engineers. It compares adopters against a synthetic control built from similar non-adopters.

Adopters merged 24.0% more PRs per engineer per day (CI +14.5% to +33.7%). A placebo test at a fake earlier date showed −1.1%, essentially nothing, and the lift held up over time: +29.4% in February, +20.0% in March and April. More use meant more lift, from +15.0% in weeks with three days of tool use to +50.1% with five or more.

The authors are clear about the limits. They counted merged PRs and nothing else: no reverts, review time or quality. Small, frequent PRs might get rewarded. Engineers chose whether to adopt. The authors are Microsoft employees, and Microsoft owns GitHub and sells these tools, which the paper itself notes. Among single-tool users, Copilot CLI’s lift (+24.9%) was about 2.2 times Claude Code’s (+11.4%), but given all that, it’s shaky ground for picking a winner.

The report that measured quality: review and bugs grew faster than output

Faros’s April 2026 report fills in the missing piece. It covers two years of telemetry from 22,000 developers across more than 4,000 teams and compares each organization’s lowest- and highest-AI-adoption periods.

Output clearly went up: task throughput per developer rose 33.7% and PR merge rate rose 16.2%. Over the same comparison, bugs per developer rose 54%, the incidents-to-PR ratio rose 242.7%, and median time in review rose 441.5%. PRs merged with no review at all were up 31.3%.

Running the numbers ourselves, PR merge rate went to about 1.16 times its baseline while median time in review went to about 5.4 times. Code comes out a bit faster, and checking it got a lot slower. Grain of salt, though: Faros sells engineering analytics, the sample is its own customers, and the report gives percentage changes without baselines or detailed methodology. Take the direction seriously and the exact numbers less so.

Putting the four together

Each study measured something different. METR measured how long experts take on single tasks in big, familiar repos. Microsoft measured PR counts across a company. Faros measured team-level throughput and quality. Call all of that “productivity” and of course it looks contradictory.

Keep only the overlap and it’s pretty consistent. Output volume is up: Microsoft’s +24% PRs and Faros’s +16.2% merge rate point the same way, and METR’s follow-up leans toward faster, if uncertainly. The cost piles up at review. Even in METR’s 2025 trial, 9% of time went to reviewing AI output, and in Faros’s data review time and bugs grew much faster than throughput. Developers’ own impressions don’t track any of this well. People who got slower believed they’d gotten faster.

So here’s our conclusion. AI coding tools speed up writing code and push the bottleneck to review and verification. Whether they make you “faster” depends on where you draw the line. If you only count writing code, yes. If you count the quality of what ships, the honest answer is closer to “we don’t know yet.”

How to actually judge it on your team

The first step is changing what you measure. If you only track PR counts or “it feels faster,” you repeat the Microsoft paper’s blind spot and the METR participants’ illusion. Track review wait times, reverted commits and post-release bugs or incidents before and after rollout, and you’ll know whether the gain is real.

It’s also worth splitting by kind of work. In METR’s first trial, AI helped least in huge repos the developer knew deeply, full of undocumented context. Unfamiliar code or repetitive work may see bigger gains. That’s our read, not something these studies compared head to head. But with the data we have now, widening AI use only as fast as your reviewers can keep up looks like the safest bet.