Roman Agaev | Creator of LLMGen | AI Platform Architect | Benchmark Methodology Author
Why I Built This
I am a platform engineer. Not a researcher, not a prompt engineer — someone who ships production systems. Over the past 117 days, I built LLMGen: an AI-driven platform engineering system that orchestrates the full software delivery lifecycle across multiple parallel projects.
The output: 44 completed features (25 greenfield, 19 brownfield), 1,350 commits, roughly 6.8 million lines of code. Every feature includes requirements, design documents, implementation, tests, CI/CD pipelines, Helm charts, and deployment configurations.
When I went to benchmark these results against the industry, I hit a wall. There is no benchmark for what LLMGen does.
The Benchmark Hierarchy I Discovered
AI engineering benchmarks form a 5-level hierarchy. The industry has built evaluation frameworks for levels 1 through 4 — and left level 5 completely unmeasured:
LLMGen operates at Level 5. Each "feature" produces ~150,000 LOC — a complete vertical slice from requirements through deployment configs. A SWE-bench task produces 50-500 LOC. That is a 60x scope difference per feature, and no benchmark accounts for it.
What Makes LLMGen Different
Two things separate LLMGen from every other tool I have seen:
Two-Tier Architecture
• Tier 1: IDE Extension — Step-by-step interactive workflows with explicit approval gates. Integrates with your existing IDE.
• Tier 2: K8s Multi-Agentic Cluster — 24 autonomous agents on Kubernetes, executing custom templates in parallel.
This is why LLMGen can orchestrate 44 projects simultaneously. Kiro, Cursor, and Devin are all single-tier tools. LLMGen has a local interactive layer and a cloud-scale autonomous layer.
It Orchestrates, Not Competes
LLMGen does not compete with Cursor. It orchestrates Cursor as a component within its Tier 1 workflows. It does not compete with Kiro — Kiro operates at Level 3 (single-feature spec-driven development). LLMGen operates at Level 5 (multi-project platform engineering). Different levels, different problems.
Methodology Walkthrough
I needed a way to make LLMGen's output comparable to existing benchmarks without misrepresenting what either system does. Here is the approach.
Step 1: Map to Each Benchmark Level
I scored LLMGen against every level of the hierarchy:
• Level 2 (SWE-bench): 489 requirements identified, 100% of projects that entered the workflow produced complete output
• Level 3 (Ship-Bench): 91/100 SDLC quality across Planning (93), Architecture (93), Implementation (89), QA (88)
• Level 4 (SWE-AGI): 44/44 systems completed, ranging from 15K to 1.5M LOC each
Step 2: Compute Composite Score
94/100 weighted normalized — combining feature completion, SDLC quality, system-scale delivery, and cost efficiency.
Step 3: Four-Tier Verification
Validation is structured into tiers:
• Tier 1 (completed): Build + mocks, unit tests, coverage gating
• Tier 1.5 (completed): Static analysis, zero-violation enforcement
• Tier 2 (completed): Project-level E2E testing on Kind clusters, 100% pass rate
• Tier 3 (completed): System DevOps E2E, multi-project integration, 100% pass rate
Results
Token Efficiency (The Underrated Metric)
LLMGen's step-segregated architecture delivers 40-55% token reduction compared to prompt-based development. Each workflow step starts with fresh context — no conversational drift, no context accumulation, no re-explaining what you already told the AI three prompts ago.
The breakdown:
• Structured templates: -20% tokens (eliminates ad-hoc explanations)
• Step isolation: -25% tokens (prevents context accumulation)
• Policy validation: -15% tokens (rejects invalid outputs early)
• Prompt archiving: -10% tokens (enables replay without re-prompting)
This is not a model improvement. It is an architectural improvement. Any tool could adopt step-segregated prompting. Most do not.
At Tier 2 Scale
1,000 parallel projects. ~2 hours. ~$250K cost vs $150M+ traditional. The 24-agent K8s cluster that ran 44 projects is the same architecture that scales to 1,000. The constraint is budget, not design.
What Developers Should Care About
If you are evaluating AI coding tools, understand the 5-level hierarchy. A tool that scores 93.9% on SWE-bench (Level 2) and a tool that scores 95% on Ship-Bench (Level 3) are measuring different things. Neither measures platform engineering (Level 5). Comparing them without level context is misleading.
If you are building AI engineering systems, consider measuring:
• Lifecycle completeness (not just code generation)
• Multi-project orchestration (not just single-repo performance)
• Token efficiency (architectural, not just model-level)
• Cost per unit of deployment-ready output (not just speed)
If you are benchmarking AI systems, Level 5 is the gap. Build the benchmark.
Check the Methodology
The full benchmark methodology, raw data, and verification approach are available for review. I welcome scrutiny — the numbers are real and the methodology is transparent.
GitHub: https://github.com/romanagaev/llmgen-benchmark
Methodology Paper: https://github.com/romanagaev/llmgen-benchmark/blob/main/docs/paper.md
If you are working on AI engineering benchmarks or have built systems with comparable scope, I want to hear from you.
Roman Agaev is the creator and architect of LLMGen and the author of the normalized benchmark methodology. He designed the platform, its two-tier architecture, and the measurement framework that maps platform-level AI engineering to industry standards.
LLMGen's Tier 2 multi-agentic architecture — designed for 1000x parallelism — remains in development. Roman is seeking the right environment to bring this vision to production scale. Open to conversations with organizations interested in AI-driven platform engineering at enterprise scope.
LinkedIn: https://www.linkedin.com/in/romanagaev/
Tags: #ai #platformengineering #benchmarks #llm #softwaredevelop
Top comments (0)