Why Textbook Benchmarks Miss Real-World Engineering

Frontier language models routinely clear 90 percent on academic benchmarks these days. Answering multiple-choice questions or writing isolated Python snippets has become relatively easy for modern models. But when you move past software code into physical engineering, like designing buildings or electrical substations, the reality looks very different.

That gap prompted Google AI and Infrastructure to team up with hiring platform Mercor and release Physical-Engineering’s Last Exam, or PLE. Instead of toy puzzles, each task asks for a realistic, billable engineering deliverable. The benchmark spans 85 long-horizon tasks across four core disciplines: 30 in mechanical, 27 in electrical, 18 in civil, and 10 in structural engineering. The tasks come from two pipelines. In 44 expert-authored tasks, practicing engineers built the project folders, prompts, and grading rubrics themselves. In the other 41 co-authored tasks, engineers wrote the problem, an agent pipeline assembled the environment and grader, and engineers then validated each task and made it harder.

What sets PLE apart is the project environment. Agents do not get a neat one-line prompt. Each task runs inside a containerized project folder mirroring a genuine engineering handoff, averaging 61 files per task and 5,170 files across the entire suite. Agents receive project specification PDFs, material cost spreadsheets, correspondence emails, 2D CAD files, 3D building information models, and revision logs. From that multi-format pile, the agent must identify governing legal codes, run structural and thermodynamic calculations, and produce production-grade deliverables that an engineer could review and stamp.

How Frontier Models Fared Across 85 Billable Tasks

The researchers evaluated 14 frontier models across seven open agent harnesses: Antigravity, Claude Code, Codex, Hermes, OpenCode, OpenHands, and Pi. GPT-6 Astra ran only in Codex and Grok 4.7 only in its own Grok Build harness, which brings the total to 86 model and harness pairs. Every pair ran each task three times, with a six-hour agent budget per run.

The baseline scores reveal just how difficult autonomous engineering remains. Across the 85 tasks, only a single configuration managed to break the 50 percent mark: Claude Opus 5.5 running inside the Claude Code harness, which averaged 50.6 percent. Achieving that top score required substantial resources, chewing through an average of 27 million tokens and 63 minutes per task.

One caveat on that headline number. Google’s blog post says every frontier model scored below 50 percent, with a best of 48.3 percent, but that figure is on the private test set. The paper’s 50.6 percent is the mean over all 85 tasks and three runs. Unless noted otherwise, the numbers here come from the paper’s full 85-task results.

OpenAI’s GPT-6 Astra inside Codex came in second at 48.3 percent, trailing the leader by just 2.3 points. The real highlight for Astra was economic efficiency. While Claude Opus 5.5 cost 17.18 dollars per task on average, GPT-6 Astra landed a close second for 9.15 dollars, roughly half the expense.

Another standout on cost was Gemini 3.8 Flash. Operating in Claude Code, it managed a 28.2 percent score while burning only 2.55 dollars per task. That represents 56 percent of the top configuration’s score for less than 15 percent of its cost. On the other end of the spectrum, open-weight models like Qwen3 Coder 480B, Kimi K2 Thinking, MiniMax M2, and Llama 4 Maverick stayed below 7 percent even in their best harness.

The discipline breakdown for the top configuration (Claude Opus 5.5 in Claude Code) shows a wide spread. Civil engineering was the easiest at 69.9 percent, while structural came in at 46.6 percent and mechanical at 40.5 percent. Co-authored tasks were also harder than expert-authored ones, 48.0 versus 53.1 percent in the same setup.

The 91 Percent Paradox: Why Partial Compliance Fails in Physics

The most illuminating takeaway from the paper lies in how agents failed. Across all runs spanning 86 model and harness pairs, 23.2 percent received a flat score of zero. The top configuration rarely failed outright, though. It scored zero in only 11 of its 255 runs, so its losses came from somewhere else.

On the co-authored tasks, the top configuration’s deliverable items satisfied 91 percent of their individual constraints on average. So it got most things right. Yet only 41.3 percent of those items passed every hard safety and feasibility constraint.

In software development, getting 90 percent of your functions working still gives you something testable. You can run the code, hit an error, and patch the bug. But physical engineering does not grant partial credit for safety. If an agent designs a steel truss where nine out of ten joints hold, but one calculation underestimates the load, the bridge fails. If wire gauge calculations ignore peak electrical current, you get an electrical fire.

PLE’s co-authored graders treat these safety and feasibility limits as hard constraints. If a deliverable item breaks even one, it earns no objective credit, no matter how well it handles everything else. The site-design and MEP-coordination graders go further with all-or-nothing gates. The paper points to these hard constraints as the main place where the top configuration lost points. Getting most constraints right and leaving zero critical errors turn out to be very different skills.

Models Drive Scores While Harnesses Dictate the Bill

If you use coding agents, you’ve probably wondered how much the harness matters compared to the model. PLE gives a fairly clear answer.

When researchers held the underlying model constant and swapped harnesses, scores moved by a median of only 3.0 points. By contrast, swapping the model created performance swings of up to 50 points. The ability to parse complex engineering drawings and reconcile conflicting building codes lives in the model’s weights, not in the wrapper around it.

Cost is a different story. The eight most expensive configurations, all Claude models running in Antigravity, Claude Code, or Codex, cost between 27 and 54 dollars per task, yet scored between 33.8 and 49.3 percent. Every one of them spent more than the top setup’s 17.18 dollars and still scored lower. That’s why the paper recommends always reporting PLE results with both the model and the harness.

Google and Mercor released 29 public tasks, grading scripts, and an evaluation container on GitHub, and kept the remaining 56 tasks private for an open competition at ple-bench.com. You can submit your own agent setup for evaluation on the private set, and Google says it will run 13 sponsored models free on its servers. Prize details haven’t been announced yet.