DEV Community

InterSystems Developer for InterSystems

Posted on Originally published at community.intersystems.com

ObjectScript LLM Benchmark: Execution-Graded Results for 14 Models

We've published the first open benchmark measuring LLM competence on InterSystems ObjectScript
and the IRIS platform: ObjectScript LLM Benchmark.

Summary

The ObjectScript LLM Benchmark evaluates 14 LLMs on 140 hand-written items covering ObjectScript and the InterSystems IRIS platform. The benchmark tests code generation, comprehension, and bug fixing. 93 items are execution-graded on a live IRIS instance, while 47 comprehension items are evaluated by a rubric-driven LLM judge. Overall accuracy ranges from 0.515 to 0.851, with the top 9 models forming a statistically overlapping cluster. Code generation is the most difficult and discriminating task family. Model cost varies substantially even among models with similar accuracy.

Why benchmark LLMs on ObjectScript and IRIS?

Established coding benchmarks — SWE-bench, LiveCodeBench, Terminal-Bench — draw from Python,
JavaScript, and shell. None cover ObjectScript, IRIS SQL, Embedded Python, or %CSP.REST.
Published coding scores don't transfer to the IRIS platform. This is a first attempt to
measure that gap directly.

What the benchmark measures

140 hand-written items across three families: code generation, code comprehension, and bug fix.
The primary grader is a live IRIS instance — model responses are compiled and executed, and
output is compared to an expected string. No subjective scoring for the 93 execution-graded
items. The 47 comprehension items use a rubric-driven LLM judge.

We ran 14 models (Claude 4.x/5, GPT-5 variants, and three self-hosted Qwen3/DeepSeek models)
at their highest available reasoning configurations, one response per item at temperature 0.

Overall results

Overall accuracy by model with 95% bootstrap confidence intervals

Overall accuracy ranged from 0.515 to 0.851. The top nine models cluster between 0.737 and
0.851 with overlapping bootstrap 95% confidence intervals — at n=140 we can't separate them
from each other, only distinguish the top cluster from the bottom five.

Results by task family

Accuracy by task family, models in overall rank order

Code generation is the hardest family (ceiling 0.745, range 0.511 across models) and the most
discriminating. Bug fix was the least discriminating (range 0.196). Every model scores higher
on comprehension than on generation — the gap runs from ~0.03 to ~0.25, though part of that
reflects different grading regimes for each family, not just capability.

A few observations worth calling out:

  • DeepSeek-v4-flash (self-hosted) ties for first on code generation (0.745) and leads on bug fix (0.913), placing 7th overall. Its comprehension score is lower (0.774 vs 0.91–0.96 for top Claude models), graded by a different judge, so interpret that comparison carefully.
  • Cost efficiency varies more than raw accuracy. Among models scoring above 0.80, claude-sonnet-5 runs at $4.61 per 1,000 correct answers vs. $17.54 for gpt-5-6-sol.

Cost vs. accuracy

Cost per request vs. overall accuracy; marker area proportional to latency

Cost per request across the 10 priced models spans $0.0009–$0.0144, a 15× range. Normalizing
by accuracy compresses that to 12× and changes the ordering.

Limitations

  • Small corpus. 140 items is a limited sample; the confidence intervals reflect that.
  • Pass\@1 only. One response per item, temperature 0. Run-to-run variance for reasoning models is not captured.
  • Self-referential judge. Comprehension is graded by a pinned claude-opus-4-8, which also records the highest comprehension score (0.960). The execution-graded items are unaffected, but the comprehension column should be read with that in mind.
  • Coverage gaps. No Interoperability (productions/HL7/DTL), no large-context tasks, no SQL-only work.
  • Contamination going forward. Items were unpublished at run time; this release publishes them, so future runs against these items carry contamination risk.

Key Takeaways

  • The ObjectScript LLM Benchmark provides an open benchmark specifically for LLM performance on ObjectScript and the InterSystems IRIS platform.
  • Overall accuracy across the 14 evaluated models ranges from 0.515 to 0.851, with the top 9 models showing overlapping 95% confidence intervals.
  • Code generation is the hardest and most discriminating task family, while bug fixing shows less variation between models.

FAQ

  1. What is the ObjectScript LLM Benchmark?

The ObjectScript LLM Benchmark is an open benchmark designed to measure how well LLMs perform on ObjectScript and the InterSystems IRIS platform. It includes 140 hand-written tasks across code generation, code comprehension, and bug fixing.

  1. How are LLM responses graded in the benchmark?

For 93 execution-graded items, model responses are compiled and executed on a live InterSystems IRIS instance and their output is compared with an expected result. The remaining 47 comprehension tasks are evaluated using a rubric-driven LLM judge.

  1. Which type of ObjectScript task is hardest for LLMs?

Code generation is the hardest and most discriminating task family in the benchmark. Every evaluated model scores higher on code comprehension than on code generation.

  1. Which LLM performed best on the ObjectScript benchmark?

Overall accuracy ranges from 0.515 to 0.851. However, the top 9 models have overlapping 95% bootstrap confidence intervals, so the benchmark does not provide enough evidence to statistically separate models within that top cluster.

  1. Does a more expensive LLM perform better on ObjectScript tasks?

Not necessarily. The benchmark shows substantial differences in cost even among models with similar overall accuracy. For models scoring above 0.80, the reported cost per 1,000 correct answers ranges from $4.61 for claude-sonnet-5 to $17.54 for gpt-5-6-sol.

Top comments (0)