When the Grader Has Bugs: Auditing Scale AI / Outlier AI's Open-Source Evaluation Harnesses & Automated Grading Logic

Author: HM Shihabuddin Jashory (@jashory) — Founder & Chief Architect @ Vision Flow Group | Principal Offensive Security Researcher

Full Multi-Platform Audit Pack & References: GitHub Gist 93de210430fea08080cedf9b8fe7252e

Disclosure & Safe Harbor Notice: All findings discussed in this article were audited exclusively against publicly licensed open-source repositories under the github.com/scaleapi organization (Apache-2.0 / MIT). Separate security advisories have been formally submitted via GitHub Private Vulnerability Reporting (GHSA), with all function-level security details withheld pending upstream patch release under the standard 90-day Coordinated Vulnerability Disclosure (CVD) policy.

---

1. Introduction: Who Audits the Automated Auditor?

Across the AI data-training and reinforcement learning from human feedback (RLHF) ecosystem, platforms such as Outlier AI (operated by Scale AI) rely on automated evaluation harnesses, trajectory parsers, and telemetry classifiers to grade both frontier AI models and the human domain experts who train them.

For thousands of software engineers, mathematicians, and security researchers contributing on Outlier AI, automated quality scores and compliance flags operate as an unappealable black box. When an automated classifier flags a contributor's session or grades a complex coding trajectory as "low quality," the contributor receives a generic deactivation notice with zero visibility into whether the failure originated in their work—or in a deterministic bug inside the grading pipeline itself.

After authoring 39 verified security advisories (GHSA) across NASA, the James Webb Space Telescope (spacetelescope), Google OSS, Solana/Firedancer, Ethereum (reth), Consensys (gnark), Sui (MystenLabs/sui), and Scale AI (scaleapi), I conducted a source-code conformance and quality audit of Scale AI's publicly released evaluation and agent infrastructure repositories:

1. scaleapi/scale-agentex (GHSA-x9wh-48x5-48mw) — Enterprise agentic execution and orchestration backend (agentex-server).
2. scaleapi/agentenv-framework (GHSA-2c7r-rh8v-mq48) — Reinforcement learning and computer-use agent environment protocol (agentenv-protocol).
3. scaleapi/mcp-atlas (GHSA-8j45-96hh-cpmq) — 1,000-task Model Context Protocol (MCP) benchmark, trajectory diagnostic suite, and LLM-as-a-Judge scoring harness.
4. scaleapi/llm-engine (GHSA-5wcc-f9qh-5x7r) — Open-source LLM inference and fine-tuning engine.
5. scaleapi/SWE-bench_Pro-os (GHSA-hm37-ccch-4jx2) — Professional software engineering benchmark evaluation suite.
6. scaleapi/SWE-Atlas (GHSA-365m-p99q-7q86) — Benchmark evaluation and scoring pipeline (qna_eval).

What the public evaluation code reveals is instructive for the entire AI evaluation industry: the grading and diagnostic harnesses themselves contain deterministic non-security software bugs and silent exception-to-penalty fallbacks that systematically distort evaluation outcomes.

---

2. Three Deterministic Grading & Evaluation Defects Inside scaleapi/mcp-atlas

When a platform evaluates an AI agent or a human-authored reference trajectory, the grading harness must satisfy three engineering invariants:

1. Accurate Failure Attribution: Distinguish between a genuine tool error and benign domain text that happens to contain the word "error".
2. Fault Isolation: If the platform's own grading API experiences a rate limit (HTTP 429) or network timeout (HTTP 503), never silently record that infrastructure outage as a 0.0 score against the participant.
3. Canonical Denominator Integrity: Compute aggregate pass rates against the full canonical task set without allowing missing rows or duplicate IDs to skew the mean.

A direct inspection of scaleapi/mcp-atlas shows how all three invariants break:

2.1 Unanchored Substring Matching Misclassifies Valid Scientific & JSON Tool Outputs as Errors (services/diagnostics/extract_enriched_trajectory.py:102)

In services/diagnostics/extract_enriched_trajectory.py (line 102), the diagnostic pipeline determines whether a tool call succeeded or failed using:

is_error = text.startswith('Error:') or text.startswith('ERROR:') or 'error' in text[:80].lower()
status = 'error' if is_error else 'ok'

Because 'error' in text[:80].lower() performs an unanchored substring check across the first 80 characters of every tool response:

• Any structured JSON API response that begins with {"status": "success", "error": null, ...} or {"error_count": 0, ...} is deterministically misclassified as status = 'error'.
• Any scientific, statistical, or engineering tool output whose opening sentence mentions "standard error of the mean", "margin of error", or "error handling" is likewise flagged as a failed tool invocation.
• Downstream in services/diagnostics/single_model_diagnostic.py (line 694), these false-positive error statuses feed directly into the automated failure-mode classifier, inflating reported tool-call error rates for completely valid trajectories.

2.2 Silent Infrastructure Exception-to-Zero-Score Fallback (services/scoring/score_claims.py:464-472)

In CoverageEvaluator.evaluate_single_claim (services/scoring/score_claims.py, lines 464-472), the scoring harness calls an external LLM judge API with a 3-attempt retry loop. When the evaluator's own backend exhausts its retries (for example, during transient HTTP 429 Too Many Requests or 503 Service Unavailable spikes on LiteLLM), the exception handler executes:

except Exception as e:
    return {
        "claim": claim,
        "coverage_outcome": "not_fulfilled",
        "justification": f"Error during evaluation after {max_retries} attempts: {str(e)}",
        "confidence_level": 0.0,
        "coverage_score": 0.0
    }

Instead of marking the claim as evaluation_error / NaN and halting aggregation until the grading service recovers, the harness silently converts its own infrastructure failure into "coverage_outcome": "not_fulfilled" and assigns coverage_score = 0.0.

2.3 Inner-Join Benchmark Distortion & Unchecked Duplicate task_id Weighting (services/scoring/score_claims.py:139-145)

In merge_gtfa_with_model_data (services/scoring/score_claims.py, lines 139-145), the scoring script joins the canonical ground-truth claims (df_gtfa) with the evaluated predictions (df_model) using:

merged_df = pd.merge(df_gtfa, df_model, left_on='TASK', right_on='task_id', how='inner')

Using how='inner' without prior task_id deduplication or canonical task-count verification means that omitted tasks vanish from the denominator rather than receiving 0.0, and duplicate task_id rows multiply the weight of individual tasks in calculate_benchmark_statistics.

---

3. Six Coordinated Security Advisories (GHSA) Across scaleapi Harnesses (Embargoed Under 90-Day CVD Policy)

Separate from the non-security grading logic defects above, our source-code audit identified six architectural security findings across Scale AI's open-source ecosystem, all formally submitted via GitHub Private Vulnerability Reporting (GHSA) with reporter credit accepted (@jashory).

90-Day Coordinated Vulnerability Disclosure (CVD) Embargo Notice: In strict compliance with industry-standard 90-day Coordinated Vulnerability Disclosure policy, all function-level locations, affected file paths, and remediation patches for the six security advisories below are withheld from public disclosure until upstream patches are released by the maintainers.

1. scaleapi/scale-agentex — GHSA-x9wh-48x5-48mw (Critical — CWE-287 / CWE-306 / CWE-863 / CWE-918): Reported privately via GitHub PVR (State: Triage, Reporter Credit Accepted). Technical details withheld pending upstream release under 90-day CVD policy.
2. scaleapi/agentenv-framework — GHSA-2c7r-rh8v-mq48 (High — CWE-22 / CWE-306 / CWE-862 / CWE-918): Reported privately via GitHub PVR (State: Triage, Reporter Credit Accepted). Technical details withheld pending upstream release under 90-day CVD policy.
3. scaleapi/mcp-atlas — GHSA-8j45-96hh-cpmq (High — CWE-250 / CWE-94): Reported privately via GitHub PVR (State: Triage, Reporter Credit Accepted). Technical details withheld pending upstream release under 90-day CVD policy.
4. scaleapi/llm-engine — GHSA-5wcc-f9qh-5x7r (High — CWE-78 / CWE-22 / CWE-918): Reported privately via GitHub PVR (State: Triage, Reporter Credit Accepted). Technical details withheld pending upstream release under 90-day CVD policy.
5. scaleapi/SWE-bench_Pro-os — GHSA-hm37-ccch-4jx2 (High — CWE-22 / CWE-59): Reported privately via GitHub PVR (State: Triage, Reporter Credit Accepted). Technical details withheld pending upstream release under 90-day CVD policy.
6. scaleapi/SWE-Atlas — GHSA-365m-p99q-7q86 (High — CWE-670 / CWE-78): Reported privately via GitHub PVR (State: Triage, Reporter Credit Accepted). Technical details withheld pending upstream release under 90-day CVD policy.

---

4. Constructive Engineering Recommendations for Scale AI & Outlier AI

1. Separate Infrastructure Exceptions from Quality Scores: Never map evaluator timeouts, rate limits (HTTP 429/503), or container crashes to score = 0.0 (not_fulfilled). Track evaluation_status = "infra_error" separately and trigger automatic re-queueing.
2. Enforce Least Privilege (CWE-250) in Automated Judges: Automated LLM judges grading untrusted model or contributor outputs should run with minimal read-only privileges inside the verifier environment.
3. Adopt Strict Conformance Unit Tests for Path, Identity & URL Boundaries: Integrate deterministic unit tests verifying authentication boundaries, canonical workspace path containment, and outbound URL validation.
4. Provide Transparent, Reproducible Audit Logs for Contributor Reviews: When an automated Trust & Safety or quality classifier flags a human contributor on Outlier AI, provide the exact timestamped telemetry event category and guarantee a human-in-the-loop engineering review for verified contributors.