TL;DR
- PLE is a benchmark from Google Cloud AI Research that asks AI agents to do professional physical-engineering work and grades the files they hand back, starting with requests a colleague would send on an active job: a roof deck to size, a site layout to fit, a switchboard to load, a cooling loop to choose.
- It holds 85 long-horizon tasks across structural, civil, electrical and mechanical engineering, built around a realistic project file system of PDFs, spreadsheets, CAD/BIM drawings, correspondence and logs, a file mix an agent has to read before it designs anything.
- PLE has 44 expert-authored tasks with 941 rubric items at 21.4 per task and 41 co-authored tasks; 37 of the co-authored tasks are graded against constraints as well as an objective score.
- The best configuration, Claude Opus 5.5 in Claude Code, reaches a mean score of 50.6% at a cost of $17.18 per task, with 27M tokens and 63 minutes per task on average, and the results on model and harness choice separate very differently: a 50-point spread across models against 0.7 to 12.1 points across harnesses for a fixed model.
- The numbers get soft per task: the mean range of one task-harness cell over three repetitions is 15.4 points, and 105 of the 595 cells span 30 points or more.
- A public subset of 29 tasks ships with its graders and harness; Netics recommends using it as a starting point before adding representative deliverables from the firm's own work.
The PLE record on arXiv (arxiv.org/abs/2610.30537, submitted 21 October 2026, retrieved 2026-10-24), the citable version of the benchmark's scope and results.
What PLE asks an agent to produce
A task opens with a request written the way a client or a senior engineer writes one. The structural example asks for a roof deck design: size the steel decking, the open-web joists and the lateral diaphragm framing so member demand stays inside code capacity under gravity, snow-drift and shear loads. The electrical example asks for a low-voltage switchboard that carries 32 feeders and branch circuits inside ampacity and voltage-drop limits. The mechanical example asks for a cooling loop selected to meet thermal load inside hydraulic limits while keeping the 25-year life-cycle cost down.
Behind that request sits a project file system: PDFs, spreadsheets, CAD and BIM drawings in DXF, DWG and IFC, correspondence and logs. The agent works with document, spreadsheet and code tools, and the answer is a package of files — an engineering memo, a calculation workbook, an equipment schedule, a 2D or 3D drawing. The paper describes these as production-format deliverables that a licensed reviewer would sign off on.
Netics visual built from the PLE paper's own figures (arXiv 2610.30537, retrieved 2026-10-24): task count, rubric density, best mean score and cost per task.
The paper explains why this shape of work is worth measuring and also why it can be graded at all. Existing engineering benchmarks test textbook questions, isolated tool calls or toy design problems whose full specification fits in the prompt, and the paper states that leading agents have already saturated most of them. A signed deliverable behaves differently: it can be checked against the governing documents and against physics, so eligibility and economic quality both become measurable quantities.
How the 85 tasks were built and graded
Two routes produced the tasks. Practising engineers from the data vendor's expert pool wrote 44 of them end to end, inside realistic project environments, and each is graded by a weighted rubric of binary, evidence-bound criteria. Those 44 tasks carry 941 rubric items, 21.4 per task on average.
Figure 1A of the PLE paper, the structural task card (arxiv.org/html/2610.30537v1, retrieved 2026-10-24): the request, the discipline and the input file count for one graded deliverable.
The Creator-Solver pipeline produced 41 co-authored tasks. Engineers wrote the task briefs, an adversarial agent pipeline built each environment and its grader from licensed engineering documents, and every task was redesigned until a frontier solver agent stopped saturating it, then passed an independent ground-truth audit before experts validated or intensified the difficulty.
Co-authored tasks add a second scoring channel. Alongside rubric constraints, an economic or performance objective is computed from the design itself. The paper gives the reason a rubric alone falls short: two wire schedules that satisfy every code check can differ by more than 20% in installation cost. Binary checks alone cannot distinguish a merely code-compliant design from an economically competitive one.
Figure 1C of the PLE paper, the electrical task card (arxiv.org/html/2610.30537v1, retrieved 2026-10-24): switchboard sizing across 32 circuits with ampacity and voltage-drop constraints.
The file mix an agent has to read
Reproducing the environment also means reproducing its file mix. Across the corpus, 51.6% of files are PDFs, 25.2% are spreadsheets and data tables, and 22.4% are text, Word documents and e-mail. In the co-authored environments the balance shifts to 31.2% PDFs and 27.5% spreadsheets and data. The grader checks the requirements listed by the agent against the references, so successful navigation of the corpus contributes to the final score.
What the results say about model and harness choice
Fourteen frontier models ran under seven publicly available agent harnesses, three runs per pair. Claude Opus 5.5 in Claude Code leads at 50.6%, GPT 6 Astra in Codex reaches 48.3%, Gemini 4 Argon in Antigravity 41.7%, Claude Fable 5 in Claude Code 34.9% and Grok 4.7 in Grok Build 32.0%. Gemini 3.8 Flash in Antigravity lands at 26.4%, while Qwen3 Coder 480B, MiniMax M2 and Llama 4 Maverick stay below 7% on their best harness, with Llama 4 Maverick in Claude Code at 0.4%.
Figure from the PLE task figures (arxiv.org/html/2610.30537v1, retrieved 2026-10-24): the input model an agent receives for the site design task, before any deliverable exists.
Read as a model ranking, that table hides the more useful comparison. Taking each model's best harness, the spread across models reaches roughly 50 points. For a fixed model, the spread across the seven harnesses runs from 0.7 to 12.1 points with a median of 3.0, and for Claude Opus 5.5 the harnesses span 47.3% to 50.6%. Model choice dominates, and harness choice still moves the result enough to matter between two engines of similar capability.
What the results cost per task
Cost moves on its own track. The median configuration in the paper costs $6.63 per task and scores 25.7%, while the leader costs $17.18. A pair that scores a few points below the leader at a fraction of that price is the interesting option for anyone planning volume.
Where the numbers get soft
Run-to-run variance is large at the level of a single task. The mean range of a task-harness cell over three runs is 15.4 points, and 105 of the 595 cells span 30 points or more. Set-level means are far steadier: for Claude Opus 5.5 each harness mean has a standard deviation of 0.3 to 1.8 points across its three runs, with Claude Code at 52.0%, 49.3% and 50.6%.
The grading was re-checked in the same section. All 1,159 LLM-judged runs of Claude Opus 5.5 across the seven harnesses were re-graded with a second pass of Claude Opus 5.5, plus Claude Opus 5 and Gemini 4 Argon, and the second Claude Opus 5.5 pass reproduces 98.7% of item verdicts.
Anyone quoting a single screenshot of an agent's score is quoting noise. The paper's own instruction is that reported per-task numbers belong with repeated runs, and a procurement conversation about agent capacity should ask for the number of runs before it asks for the percentage.
Figure from the PLE task figures (arxiv.org/html/2610.30537v1, retrieved 2026-10-24): the delivered 3D model for the site design task, the artefact a reviewer would open.
What an engineering team can do with a public 29-task subset
The paper releases a public set of 29 of the 85 tasks together with their graders and the agent harness, and holds the remaining tasks private for an open competition and leaderboard. Netics recommends taking that public subset as a starting point, then defining separate checks for the firm's own deliverable conventions.
Netics recommends running the public subset first, then putting three of the firm's most common deliverable types through the same harness and recording scores and observed failure modes. The review record should flag any rounding errors, code-edition mismatches and omitted load cases alongside the mean score.
Figure 1D of the PLE paper, the mechanical task card (arxiv.org/html/2610.30537v1, retrieved 2026-10-24): equipment and piping selected against thermal load, hydraulic limits and life-cycle cost.
The harness question is the one that survives the benchmark. An agent platform that a team runs itself has to answer the same question PLE asks: what surrounds the model, and how much of the result comes from that. For related context, the article on benchmarking an inference stack with a real load client discusses load generators and measurement design.
Originally published on the Netics blog.
Top comments (0)