Researchers have built an agent for writing industrial controller code that is structurally forbidden from marking its own work complete. The system, SemaPLC, submitted to arXiv on August 19, 2026, blocks task completion until three external checks independently confirm the code matches the specification, compiles in the real toolchain, and behaves correctly when actually executed. The results expose an uncomfortably wide gap between code that builds and code that works: on project-context tasks, integrated compilation succeeded 89.4% of the time while dynamic behaviour scored far lower.
Key facts
- Full title: SemaPLC: A Project-Grounded, Verification-Gated Agent Harness for PLC Code Generation, submitted August 19, 2026.
- 72.6% mean strict verified pass rate across 117 independent program-unit tasks; 89.4% integrated compilation on 65 project-context tasks, with dynamic behaviour scoring 52.2.
- Ranked #3 paper of the day on Hugging Face with 111 upvotes.
- Ships as a browser-based development environment that generates, edits, verifies and simulates controller programs.
A programmable logic controller is the computer that runs physical machinery -- the conveyor, the pump, the safety interlock. Programming one is a specialist trade, and it is exactly the kind of repetitive, well-documented, template-heavy work that language models are good at. It is also work where being wrong has consequences a text editor cannot undo.
The design decision at the centre of SemaPLC is a refusal. The agent can write whatever it likes. It cannot say "done." Completion is gated behind three separate external checks, each of which the model has no ability to influence: does the code satisfy the stated specification, does it compile in the actual vendor toolchain, and does it do the right thing when the program is genuinely executed. The model's assessment of its own output is not an input to that decision.
An analogy: the difference between a student grading their own exam and a student whose exam is graded by three people who have never met them, one of whom makes them perform the experiment in front of a class.
The number that carries the paper is the gap between two figures on the same set of tasks. Integrated compilation succeeded 89.4% of the time. Dynamic behaviour -- what the program actually did when run -- scored 52.2. Those are measurements of the same code by different judges, and the spread between them is the entire argument. Code compiling means the syntax is legal and the types line up. It says close to nothing about whether the valve opens at the right moment.
This matters far beyond factory automation, because "it compiles" is a proxy metric used all over AI code generation, and so is "the model said it was confident." Both are measurements of the wrong thing, and both are popular because they are cheap. SemaPLC's contribution is showing how far apart the cheap measurement and the real one can be in a domain where you can actually run the code and check.
The system is also project-grounded rather than snippet-based, which is a second, quieter improvement. Most code-generation evaluation hands a model an isolated function with an isolated test. Real controller programs live inside a project with existing tags, existing hardware assumptions, and existing logic that the new code has to integrate with. The 65 project-context tasks measure that integration; the 117 independent program-unit tasks measure the easier, isolated version.
The timing is uncomfortable in a way the authors did not choose. SemaPLC was published the same day five US federal agencies warned that attackers are using AI-generated exploitation scripts against Siemens S7 controllers. The two documents describe the same layer of the technology stack from opposite ends: one is AI writing controller code carelessly enough to be a threat, the other is AI writing controller code under a gate that will not let it lie about success. That the research community and the threat-intelligence community arrived at industrial controllers in the same week is a reasonable signal about where the next few years of attention go.
The honest caveats are substantial. A 72.6% strict verified pass rate is a research result, not a deployment story, and safety-critical control is the last place anyone should ship a system that fails a quarter of the time. More fundamentally, a verification gate is only as good as the specification behind it. A requirement written incorrectly passes every check in the pipeline, cleanly and confidently, because the pipeline is checking conformance rather than intent. The paper's contribution is the harness discipline; it does not claim controller programming is solved.
The reusable lesson for anyone building coding agents anywhere is simple enough to state in one sentence: make the runtime the judge. Any metric the model can influence will eventually be optimized rather than satisfied.
Related reading on this site: agent harnesses and scaffolding, reinforcement learning with verifiable rewards, reward hacking, and a runbook, not a model, hit 95 percent on Terminal-Bench.
Originally published on Ground Truth, where every claim is checked against the primary source.
Top comments (0)