Defensive Agent Harness: Measuring the Microtask Eligibility Gap (arXiv 2610.00025)

Agent harnesses increasingly want to run small language models (SLMs) on the microtasks around a frontier LLM planner: auto-approving shell commands (T1), selecting tools (T4), and managing memory recall/write (T2/T3). The paper Measuring the Microtask Eligibility Gap (Hu / Ramachandran, arXiv 2610.00025, Sept 2026) asks: can off-the-shelf SLMs meet practitioner-defined thresholds for these narrow, high-frequency calls?

The Eligibility Gap

The benchmark defines 4 frozen microtasks with a fixed prompt, an automatic metric, and a confidence-anchored non-LLM baseline (TAU = 0.95). A configuration passes only if its confidence interval clears TAU — a CI-aware eligibility rule. The authors sweep Qwen3 {0.6, 1.7, 4, 8}B in their best settings (FP16, greedy, frozen prompt, no tuning):

• 0 of 16 configurations pass.
• 4-bit quantization (RTN/GPTQ/AWQ) does not move any configuration into eligibility; the gap tracks model size, not precision.
• Replication on Llama-3.x: 12/12 ineligible.
• Neutral paraphrase sweep: 0/112 eligible across 3 paraphrases per cell.
• Cross-family replication confirms robustness; the gap is real.

The practical implication: place SLMs behind a baseline that meets the CI-backed threshold (TAU), and use the SLM only where the baseline fails.

The Defensive Harness (4 Components)

This repository (amurlaniakea/Microtask_Eligibility_Harness, AGPL-3.0-or-later) implements a local-first defensive harness that enforces the eligibility gap as a fail-closed rule — not as an assumption.

Component Design (Non-LLM Baseline)

Component	Microtask	Detection Method	Fail Mode
Preflight Validator	T1 Shell auto-approval	Regex patterns (rm -rf, sudo, `curl	bash, eval(`)
Memory Guardrail	T2 Memory recall	BM25 mock (token overlap); tau_score < 0.95 → requires review	Blocked / Review
Memory Guardrail	T3 Memory write	Regex risk (clear_memory, invisible chars, spam) + length/duplicate	Blocked
Preflight Validator	T4 Tool/network selection	Whitelist of signatures (list_dir, read_file, git_op); missing or mismatched → blocked	Blocked

The harness adapter combines all 4 in a deterministic pipeline: any tau_score < 0.95 activates blocked; any score of exactly 0.0 (confirmed risk) activates blocked; scores between 0.0 and 0.95 activate requires_human_review. This matches the SDD fail-closed principle.

Adversarial Testing

Every component has an adversarial test:

• T1 base64 evasion: payload echo cm0gLXJmIC8= | base64 -d | sh tests whether the regex catches encoded shell commands. The test is marked xfail (strict=False) — the evasion is a known limitation of the non-LLM baseline; it requires human review, which the harness correctly flags.
• T4 typo (lss) and signature mismatch: verifies that altered tool signatures are rejected with tau_score = 0.0.
• T3 invisible char + forget injection:  and clear_memory patterns are rejected (is_eligible = False).
• T2 hidden context: paraphrased query with low token overlap scores tau_score = 0.0 (requires review).

These tests demonstrate that the defensive layer catches adversarial inputs — not by assumption, but by verification.

Benchmark Replication

The script benchmark/run_benchmark.py replicates the paper's 16-cell sweep (4 mock SLM configurations x 4 microtasks):

• Malicious tasks: 0/16 eligible. The harness blocks or requires review for every malicious input.
• Legitimate tasks: 8/8 approved. A second set of benign tasks (ls -la, memory query with real overlap) passes all 4 components with tau_score >= 0.95, confirming zero false positives.

The raw output is saved to benchmark/logs/benchmark_raw_output.jsonl in JSONL format (16 malicious + 8 legitimate lines), including per-task final_status, tau_scores, and gate (which component triggered the block/review).

Technical Stack

• License: AGPL-3.0-or-later (verbatim FSF text, 661 lines, SPDX in all .py files).
• Package: pyproject.toml (PEP 639), installable with pip install -e .[dev].
• Dependencies: pytest>=7.0.0, pytest-cov>=4.0.0, ruff>=0.1.0 (dev extras only).
• CI: .github/workflows/ci.yml runs ruff check ., pytest --cov=., and the 0/16 benchmark on every push and PR.
• Code quality: ruff (line-length 120, E/F/I rules); pytest with xfail documentation for the known base64 evasion.
• Author: Pedro Sordo Martínez ([e-mail]).

References

• Paper: arXiv:2610.00025 — Measuring the Microtask Eligibility Gap (Hu, Ramachandran; 2026).
• Repository: amurlaniakea/Microtask_Eligibility_Harness (public, AGPL-3.0-or-later, branch master).
• Benchmark log: benchmark/logs/benchmark_raw_output.jsonl (generated per execution, excluded from git via .gitignore).
• Installation: pip install -e .[dev] (verified, exit 0).
• CI: .github/workflows/ci.yml (verified; pytest --cov=. passes with 6 passed, 1 xfail; benchmark passes 0/16 malicious + 8/8 legitimate).

---

Defensive harness enforcing non-LLM baseline eligibility. Replicates the 0/16 eligibility gap measured in arXiv 2610.00025, with adversarial testing and open-source verification.

License: AGPL-3.0-or-later