Large language models have become remarkably good at writing conventional software. Frameworks like Codex, Claude Code, and other coding agents can already solve a large fraction of real-world programming tasks.
But scientific programming—especially quantum computing—is a very different challenge.
A quantum program is not simply expected to compile and produce the correct output. It must also preserve physical correctness, maintain differentiability, respect algorithmic constraints, and often achieve high computational performance. Traditional software benchmarks rarely capture these requirements.
To evaluate how AI agents perform in this setting, we developed ORBIT-Q (Open Research Benchmark for Integrated Tasks in Quantum Computing), a benchmark specifically designed for autonomous scientific programming in quantum computing.
Why Existing Benchmarks Are Not Enough
In conventional coding benchmarks, passing unit tests is often sufficient.
In scientific computing, however, an implementation may pass numerical tests while still being fundamentally wrong.
During our experiments we frequently observed behaviors such as:
- Framework bypassing. Instead of using the requested quantum framework, the agent secretly reconstructs the computation with NumPy or JAX tensor operations.
- Broken differentiability. The generated code produces correct numbers but destroys the end-to-end automatic differentiation pipeline.
- Violation of physical assumptions. The implementation changes the intended mathematical or physical problem while still appearing to "work."
These failures are difficult to detect using standard execution-based evaluation alone.
Scientific programming therefore requires evaluation beyond correctness—it requires semantic verification.
ORBIT-Q: A Dual-Axis Benchmark
ORBIT-Q consists of 12 challenging research-level quantum programming tasks, covering representative workloads in quantum simulation, quantum machine learning, tensor network algorithms, optimization, and automatic differentiation.
Its key idea is a dual-axis evaluation protocol.
Axis 1: Agent Evaluation
Keep the quantum framework fixed and compare different AI agents.
This measures how well various models (GPT, Claude, etc.) can solve scientific programming tasks under identical software environments.
Axis 2: Framework Evaluation
Keep the AI agent fixed and compare different quantum software frameworks.
This evaluates not only functionality and runtime performance, but also something increasingly important in the AI era:
How AI-friendly is a software framework?
A framework with discoverable APIs, composable abstractions, and consistent interfaces allows autonomous agents to generate substantially better solutions.
Preventing "Cheating"
To ensure generated solutions genuinely use the intended framework, ORBIT-Q employs a three-stage verification pipeline:
- Deterministic functional testing
- LLM-based source-level semantic auditing, designed to detect framework bypassing and other implementation shortcuts
- Expert manual review
This combination substantially reduces false positives that commonly appear in conventional coding benchmarks.
Results: Which Frameworks Work Best?
Using the same coding agent (Codex + GPT-5.5), we evaluated several mainstream quantum software frameworks.
TensorCircuit-NG achieved the strongest overall performance, successfully solving 10 out of 12 benchmark tasks while also delivering significantly faster execution than competing frameworks.
For comparison:
- TensorCircuit-NG: 10 / 12
- PennyLane: 8 / 12
- TorchQuantum: 4 / 12
- MindQuantum: 4 / 12
Beyond success rate, TensorCircuit-NG consistently produced solutions that executed several times faster than those generated for other frameworks.
This suggests that framework design has a substantial impact on autonomous scientific programming—not only for human developers but also for AI agents.
Results: Which AI Agents Perform Best?
Under the TensorCircuit-NG environment, the leading coding agents achieved:
- Codex + GPT-5.5: 10 / 12
- Claude Code + Opus 4.8: 9 / 12
- Claude Code + Sonnet 4.6: 7 / 12
Although these results are encouraging, a significant gap remains between AI-generated solutions and expert-written implementations.
Human experts solved all 12 tasks, while producing implementations that were typically more than twice as efficient as the strongest AI-generated solutions.
Current AI systems are becoming effective research assistants, but they are still far from replacing domain experts in scientific software development.
An Unexpected Observation: Safety False Positives
One particularly interesting finding was unrelated to quantum computing itself.
During evaluation with Claude Code + Opus 4.8, two benchmark tasks failed—not because the model lacked the necessary programming capability, but because the interaction was interrupted by Cybersecurity Refusals.
These tasks involved no networking, no external services, and no security-sensitive objectives. They consisted solely of local quantum programming and framework exploration.
This illustrates an often-overlooked issue in agent evaluation:
Product-level safety policies can significantly affect end-to-end task reliability, even when the underlying model is technically capable of solving the problem.
For autonomous scientific workflows, reliability depends not only on model intelligence but also on surrounding product behavior.
The Economics of Scientific AI
Another interesting lesson concerns inference cost.
Developers often compare models by token price alone.
Our experiments suggest this can be misleading.
Lower-cost models frequently require many more iterations because they generate incorrect implementations, encounter execution failures, or repeatedly need debugging.
Consequently, obtaining one successful scientific solution may consume substantially more time and tokens than using a stronger (but more expensive) model.
For scientific programming, a more meaningful metric may be:
Cost per successful scientific solution
rather than simply cost per token.
Looking Ahead
As AI agents become increasingly integrated into scientific research, software frameworks will need to evolve as well.
Future scientific software should not only be easy for researchers to use—it should also be easy for autonomous agents to understand, compose, and optimize.
Although ORBIT-Q focuses on quantum computing, we believe its evaluation methodology can be generalized to many areas of scientific computing where correctness, semantics, differentiability, and performance all matter.
If AI is going to become a true collaborator in scientific discovery, we need benchmarks that measure much more than whether code simply runs.
Paper
ORBIT-Q: Dual-axis Benchmarking of Autonomous Agents in Scientific Quantum Programming
Shi-Xin Zhang and Yu-Qin Chen
arXiv:2607.03105
The benchmark, evaluation framework, and source code are all open source at GitHub: https://github.com/sxzgroup/ORBIT-Q and the accompanying webpage: https://sxzgroup.github.io/ORBIT-Q/.
Top comments (0)