DEV Community

Manohar Papasani
Manohar Papasani

Posted on

Agent pass rates fell from 91% to 15% as codebases grew past 50,000 lines.

#ai #lowcode #genai #techlearning

The pilot that looks fine on a small repository can tell you little about the large one.

Fast prototypes are not the same as production-ready code.
Here is what the sources say to measure before release.

  • Does it still work when the codebase or dataset is large?
  • Are generated data access and permissions reviewed?
  • Does the AI output need manual refinement?

📌 Scale breaks the pilot
Agent pass rates by codebase size (Sonar)

  • 91% on codebases under 10,000 lines (RepoMod-Bench, 2026, code translation)
  • 67% between 10,000 and 50,000 lines
  • 15% above 50,000 lines

📌 Benchmarks alone do not decide
Pilot before you shortlist (Sonar)

  • Benchmark rankings alone cannot make the decision, because small isolated task scores rarely predict large codebase behavior.
  • Shortlist two or three options and pilot them on representative work.

📌 Measure failures, not just wins
Acceptance criteria for AI builders (ToolJet)

  • Record failures and manual repairs as well as successful generation.
  • Use the same acceptance criteria and non-production dataset on every shortlisted platform.

📌 Check access before release
Generated data access and writes (ToolJet)

  • Review generated data access, writes and permissions before release.
  • Test the current model and builder with the same representative task, recording consumption.

📌 Pricing can shift fast
Microsoft Power Apps

  • Per-app plan discontinued; entry price jumped from $5 to $20/user/month.
  • Copilot generates apps from natural language, with over 1,000 pre-built connectors.

What this means for your team:

  1. Pilot two or three platforms on a representative, non-production task.
  2. Compare acceptance rate, rework, cost and quality, not only build speed.
  3. Test on a large codebase or dataset, since small-scale results may not hold.

Pick one real task this week and time how much of the generated output you have to fix.

P.S. Per-user pricing can scale quickly in large teams, so check current pricing before you commit.

Originally shared on LinkedIn.

An AI interview question every day, leadership for tech people and hands-on AI learning:

Top comments (0)