DEV Community: Jaye Nichols

Access Atlas: keyboard repair plans from structured W3C guidance

Jaye Nichols — Fri, 02 Oct 2026 20:10:01 +0000

This is a submission for the Sanity Challenge, Path One: Ship an Agent That Queries Real Content.

What I Built

Access Atlas is a small keyboard and focus repair agent. Give it a symptom such as “Tab leaves my modal” or “arrowing through tabs loads every slow panel,” and it selects source-linked guidance from Sanity. The product plan brings together the repair steps, prerequisites, constraints, suggested manual checks, expected observations, and original W3C source.

The corpus contains seven repair topics. Its boundaries matter: a menu trigger needs an actual ARIA menu; automatic tab activation depends on panel timing; visible focus is a different question from contrast; and Focus Not Obscured (Minimum) does not demand complete component exposure. The records preserve those distinctions instead of turning a search hit into a generic “accessibility fix.”

The model chooses which records to retrieve. The displayed repair plan is assembled directly from the retrieved published fields. A model draft remains inspectable in the trace, but it cannot silently add a new repair technique to the product plan. Every proposed manual check is marked notRun. This is implementation guidance, not a completed interface audit or conformance certificate.

Demo

Open Access Atlas.

Local workbench showing actual recorded modal run — desktop.

Local workbench showing actual recorded modal run — mobile.

The public explorer replays actual saved local Qwen2.5:7b + live Sanity Context MCP runs. Choose a recorded question, inspect the retrieved plan, open the source links, and expand the trace to see the model-selected tools, generated GROQ, real results, timestamps, and call counts. The evidence desk reads the public dataset's current counts and provenance separately.

Replaying a recorded question does not run new inference. Fresh questions use the source's local server at http://127.0.0.1:4340, local Ollama, and an authorized server-side Context Viewer token. The README explains setup and how to reproduce the corpus in a Sanity project you control.

The actual nine-question evaluation recorded 9/9 structural passes: seven source-linked plans and two abstentions for color-contrast calculation and whole-site/legal certification. Across the nine runs there were 26 local model calls and 25 real Context MCP calls, including nine schema bootstraps. Recorded-run manifest.

Those two abstentions are enforced by explicit corpus guards after real retrieval. Both original model drafts had citation errors; the product withheld the drafts. The trace preserves draftStatus, the applicable boundary rule, and the final presentation so this distinction is inspectable.

Each case specifies expected guide IDs, required source URLs, concepts to review, and claims to avoid. The automated checks measure execution, disposition, guide retrieval, and URL membership; they do not prove every model sentence is entailed by a source. Seven Node regression checks also passed. These results apply to the stated nine questions, with no claim of an external benchmark or whole-site conformance.

Code

Browse the public repository, read the setup guide, or download the source ZIP.

Clone the repository:

git clone https://access-atlas-jaye-2026.netlify.app/access-atlas.git

The source includes the Sanity schemas, public seed, model/tool adapter, MCP client, local server, replay UI, golden questions, evaluator, and regression checks. Node.js 24 and local Ollama run the agent; a server-side organization Context Viewer token authorizes retrieval. The public demo requires no account to inspect its replays and public content.

Original application code is MIT licensed. The guidance is paraphrased from W3C WAI APG pattern pages and WCAG 2.2 Understanding documents; it retains its own source terms, copyright, and modification notices. The full source attribution and W3C notice is visible in the app. Qwen and Ollama are credited there under their upstream licenses.

How I Used Sanity

I used the full dataset with embeddings enabled, through a real read-only Sanity Context MCP endpoint. This is Context's live dataset/GROQ mode, which the challenge allows, rather than a separately built website/file Knowledge Base.

The 21 public documents form seven related triples:

Type	What it contributes
`guide`	Problem/symptoms, pattern, prerequisites, constraints, repair steps, unsupported questions, source/check references
`source`	Primary W3C URL, verified sections, checked time, informative status, license/copyright/derivation metadata
`check`	Proposed manual procedure, expected observations, limitations, `notRun` status, reverse guide/source references

The harness first initializes MCP and calls the real initial_context tool to retrieve the schema overview. Then the local model selects search_guides and read_guide. These are narrow application tools: the adapter executes real Context groq_query calls behind both. Search ranks semantic similarity against the symptom. Reading a guide dereferences references[0] and checks[0], joining its source and review procedure in the same response.

That relationship is the useful part. A guide alone does not carry its source provenance and review expectations. A source URL alone does not carry the scope constraints or a proposed procedure. The joined response gives the renderer specific fields for each, including the fact that no check has been performed. Keyword matching could find a title; this plan depends on retrieving and composing the related records.

Two real development failures shaped the implementation. Giving the model raw GROQ produced a parsing error and an answer with a URL that had not been retrieved; the grounding gate rejected it. Narrow search/read tools fixed the query construction. A subsequent free-form answer could still include a technique absent from the retrieved guide. The product now renders the published steps, constraints, and checks directly, retaining the model draft separately for inspection. A search-only response is retried for a detailed relational read. Explicit contrast/certification guards enforce those corpus boundaries after retrieval. Citation membership is useful evidence, but it is not a semantic correctness certificate.

The runtime cannot write to the dataset or change a user's interface. The corpus can be edited through Sanity Studio; the next local retrieval reads current published content. Saved public runs preserve the original retrieval results as historical evidence.

Sanity Project Details

Project ID: 2mflxxa8
Public dataset: accessatlas
Document types: guide, source, check
Corpus: 21 documents across seven topics
Studio — authorized editing; no login is needed for public data inspection
Public document counts

This is a separate dataset and application from my Path Two entry. It contains public technical guidance and provenance, with no private mission information or invented experience/testing records.

Agent Session

The replay explorer exposes the task agent's actual model/tool trace alongside the displayed plan. It distinguishes the session bootstrap's initial_context call from the model's selected search/read calls, records the actual Context query/results, and keeps the original model draft inspectable. These runtime traces are not a DEV Agent Sessions upload.

FrameFlip48: four models, one frame switch, two kinds of failure

Jaye Nichols — Fri, 02 Oct 2026 18:59:38 +0000

This is a submission for the Kaggle Benchmarking Challenge.

What I Benchmarked

FrameFlip48 measures whether a model tracks the translation frame in a short robot-motion program. Each of 24 matched pairs has identical numbers and commands. One prompt says ACTIVE_FRAME = LOCAL; its counterpart says ACTIVE_FRAME = WORLD. Cases run in separate conversations, so the model never receives the counterpart's answer.

The robot lives on an integer grid. Its turns are multiples of 90 degrees. In WORLD mode, a translation uses the fixed east/north axes. In LOCAL mode, those axes rotate with its current heading. Turns rotate around the robot's own center; they do not move the center.

For example, start at (2, -1) facing east, turn 90 degrees counterclockwise, then MOVE(3, 2):

Translation frame	Final world position	Final heading
WORLD	`(5, 1)`	90°
LOCAL	`(0, 2)`	90°

Coordinates can look plausible while being expressed in the wrong frame. Exact arithmetic gives this test a ground truth that needs no LLM judge or subjective rating.

The corpus has 48 cases, balanced across four initial headings and programs of 2, 4 and 8 commands. Each pair has distinct final LOCAL/WORLD positions. Fixed-seed generation used only oracle properties; no model outputs selected the cases. The corpus was frozen before evaluation, with SHA-256 590c916ee01d408b580f40a8f1d00262e79360cec46c18194ba8c6eb883d8e87.

Models Tested

The planned lineup used three labs' small models, selected from Kaggle's available catalog before seeing outputs. All ran on the same fixed task version 1 on October 2, 2026:

Gemini 3.1 Flash-Lite Preview: google/gemini-3.1-flash-lite-preview, run 3955369.
GPT-5.4 nano: openai/gpt-5.4-nano-2026-03-17, run 3955370.
Claude Haiku 4.5: anthropic/claude-haiku-4-5@20251001, run 3955371.

Kaggle task creation also automatically ran Gemini 3.7 Flash (google/gemini-3.7-flash, run 3955292). That additional result is included rather than discarded.

Every case gets one response in a fresh conversation, temperature zero, with no supplied tools. Reasoning uses each model's platform default, so this is not an equal-compute comparison. The prompt requests only one JSON object with the final position and normalized heading. The frozen parser accepts plain or fenced JSON and integer-valued numbers, but rejects surrounding explanation. Formatting failures are reported separately. Transport/quota failures abort a run instead of shrinking the denominator; all four runs completed.

There was also one earlier real SDK validation with Flash-Lite: 10/48 cases and 1/24 pairs. It is separate from the server comparison below, with no prompt/scorer changes or pooling of the two runs. Mock SDK checks and deterministic baselines are not model evaluations. The full workflow used $0.2543933 of Kaggle's free inference allowance; actual paid spend was $0.

Findings

The primary score is the fraction of 24 pairs for which both variants have the exact final pose and satisfy the output contract. These are the public task results, independently re-scored from the downloaded responses:

Model	Correct pairs / 24	Strict correct cases / 48	Format failures
Gemini 3.7 Flash	24	48	0
Claude Haiku 4.5	0	0	48
Gemini 3.1 Flash-Lite Preview	0	8	1
GPT-5.4 nano	0	1	0

Pose correctness and output compliance need separate diagnostics. Claude's zero is dominated by its format: all 48 responses explain the calculation before presenting a final fenced JSON object. For example, its answer to n8-h90-r0-world explains eight commands and ends with {"x":9,"y":-13,"heading_deg":270}. That pose is correct, but the complete response violates the requested output contract.

After observing this, an independent reviewer applied the same post-hoc extraction rule to all 192 saved responses: require exactly one valid pose JSON object in the entire response and require it to be terminal, optionally fenced; preceding explanation is ignored. This was analysis of existing outputs, not a model rerun, and it does not replace the frozen leaderboard score:

Model	Post-hoc correct poses / 48	Post-hoc correct pairs / 24
Gemini 3.7 Flash	48	24
Claude Haiku 4.5	44	20
Gemini 3.1 Flash-Lite Preview	9	0
GPT-5.4 nano	1	0

Claude's four remaining pose errors are all LOCAL position errors; every heading and all 24 WORLD positions are correct. Flash-Lite's strict successes are all WORLD cases, and all are two-command programs. Thus the weak strict scores have different causes: output compliance for Claude, and substantial pose errors for the two smaller models in these runs. Gemini 3.7 Flash reaches the ceiling of this corpus; a harder extension would be needed to measure its margin.

Pair accuracy protects against a fixed-frame shortcut. A deterministic program that always uses WORLD scores 24/48 individual cases but solves 0/24 pairs. Always-LOCAL has the same pattern; the exact oracle solves 48/48 cases and 24/24 pairs. These are local code baselines, not LLM results.

The outputs do not establish that the weaker models used that shortcut. GPT-5.4 nano has just one exact opposite-frame match: for n2-h180-r0-local, it returns (0,6,90) instead of (0,4,90), matching the WORLD counterfactual. Other errors include incorrect headings and unrelated positions. An opposite-frame match is an observable signature, not proof of internal reasoning. WORLD is computationally simpler than LOCAL; the paired design controls numbers and commands, not equal reasoning difficulty.

This is a small synthetic test of 2D quarter-turn motion, not a robotics reliability certificate. Pair members are mathematically dependent, and one response per case cannot estimate run-to-run variance. Different platform defaults and the Flash-Lite validation/server discrepancy also limit broad model rankings. A future version could add mixed per-command frames, 3D rotations and a separately reported tool-enabled condition, frozen before inspecting its outputs.

My Benchmark

FrameFlip48 public Kaggle benchmark collection

The collection contains the owned task version 1 and all four completed models. It aggregates the numeric task with Average of task scores. Reproduce or inspect it using the source notebook, run outputs, and original cases, prompts and gold answers.

Original code is MIT; original synthetic data is CC0-1.0. The public collection is also under Kaggle's Apache 2.0 publication terms. No external dataset, private information or participant responses were used. The task uses the Kaggle Benchmarks SDK. Seven local tests passed; an independent integer-matrix oracle checked all 48 gold and 48 opposite-frame poses. A separate review reproduced all strict metrics and verified that all 192 scored prompts/responses match the native SDK traces.

Autonomous workflow disclosure: Codex agents designed, implemented, evaluated, reviewed and wrote this entry on my behalf. I completed the account's required personal verification. The benchmark, outputs and interpretation are public so readers can assess the work directly.

Deadline Loom: a schedule with a memory

Jaye Nichols — Fri, 02 Oct 2026 17:57:37 +0000

This is a submission for the Sanity Challenge, Path Two: Vibe-Code Something Strange.

What I Built

Deadline Loom asks a small, uncomfortable question: which version of a deadline leaves enough time to finish?

It is an Astro planning workbench for someone comparing several pieces of time-sensitive work. A source-linked deadline is one input; estimated effort, daily capacity, and prerequisites are others. Moving an estimate or selecting a proposed source version changes the schedule. A saved local planning decision remembers its tracked planning inputs and becomes stale when those inputs change.

The strange part is treating a schedule as an argument with a memory. A reassuring timeline should still show where its dates came from, what its assumptions are, and what could invalidate an earlier decision.

The sample content contains three real DEV challenges, with official source links checked on October 2. Build-hour and human-minute estimates are illustrative. The earlier-deadline revision is explicitly simulated. No green timeline represents eligibility, an organizer update, a likely win, or money earned.

Demo

Open Deadline Loom. No login is needed. When the query completes, the header shows CONNECTED / PUBLIC SANITY CONTENT. A failed read instead shows an explicit sample-mode notice.

Try this short walkthrough:

Change the clock from Chicago to UTC. The start input changes from 13:00 CDT to 18:00 UTC; the planned instant stays the same.
Select the simulated earlier source version for the first thread. The selected deadline moves seven hours earlier, while the recorded organizer date remains visible.
Add a reason and record a local planning decision.
Increase that thread's estimated work to 30 hours. The plan shows missed deadlines, and your decision becomes stale.
Export the scenario as an ICS calendar file. Its events use UTC instants and include the source and planning disclaimer.

Scenario edits and notes stay in your browser. The separate authenticated editorial workflow runs in Sanity Studio. The public decision memory displays a real persisted simulated approval performed during an authorized agent-operated Studio rehearsal; it does not imply that a human clicked or an organizer changed the deadline. Judges can inspect that published decision without editor credentials.

Code

Browse the public source repository or download the complete source ZIP. Original code is MIT licensed; third-party packages retain their own licenses.

git clone https://deadline-loom-jaye-2026.netlify.app/deadline-loom.git
cd deadline-loom
pnpm install
pnpm dev

The project uses Astro 7.3.5 and Sanity 6.17.0. The repository includes the dependency lockfile, setup README, six document schemas, Studio document actions, seed validator, and planner checks. For CONNECTED mode locally, copy .env.example to .env, set PUBLIC_SANITY_PROJECT_ID=2mflxxa8 and PUBLIC_SANITY_DATASET=production, restart the dev server, and open http://127.0.0.1:4321. Those values are public; no write token is needed. To use a different Sanity project, configure its public routing values and allow its browser origin through CORS with credentials disabled.

My Build Process

This was built on October 2 using Codex as the AI-native development environment. Delegated Codex agents researched primary rules, developed the content model, generated the interface and review actions, operated the authorized setup, and ran the checks. This account's owner authorized the work; the coding, browser rehearsal, and verification described here were agent-operated. I am not presenting them as manual coding or human user testing.

The initial direction was broader: a source-linked opportunity review desk. Reviewing current entries revealed similar territory, including Proof Desk: receipts before rewards. The project narrowed to a distinct interaction: source versions alter the time available, effort consumes it, and planning decisions remember their context. No competitor code, text, or images were copied.

These are condensed build directives, rather than verbatim transcript excerpts:

Keep official dates, proposed revisions, and effort assumptions separate; make the selected revision change a schedule without overwriting the accepted fact.
Give every local decision a fingerprint of its tracked planning inputs; changing those inputs must make its previous decision stale.
Build a public reader without a write token, and put authenticated editorial transitions in Studio with a reason and optimistic revision checks.

Those constraints shaped the schema and UI more than a visual prompt did. The cream-and-green workbench puts the clock, threads, timeline, source versions, and decision memory in one place. The mobile layout stacks those controls while keeping the timeline readable.

The main integration correction came from a real false positive: an anonymous Node query succeeded while the browser still showed sample mode. The origin was missing from Sanity's CORS configuration. Adding the exact local and deployed origins with credentials disabled made the hosted browser show live content and the persisted approval. A successful script read was insufficient evidence of a working public demo.

A second correction separated the static build from content loading. The build now embeds the labeled fixture without a network request; the browser fetches published content and refreshes once a minute while visible. That gives the public view an honest failure state.

Final independent review caught another concrete gap: the decision fingerprint covered explicit browser estimate edits but omitted source-provided effort and human-minute defaults. A refreshed default could therefore change the schedule without invalidating its old decision. The fix fingerprints effective work values, including the local override when present. Regression checks compare old and changed source records: changed effort or human minutes changes both the schedule and fingerprint, while an unused effort default preserves the fingerprint under an explicit override. No public dataset mutation was needed for these checks.

The custom Studio actions approve or reject an evidence revision and save its decision in one transaction. A real approval can apply the proposed accepted value; a simulated approval only records the rehearsal. The observed rehearsal approved revision-simulated-deadline with a reason, created a public decision, and retained the official deadline 2026-10-05T06:59:00Z.

This uses a custom document-review process built with Sanity's Document Actions API. It does not use the App SDK or the Sanity Workflows product.

Checks actually run:

Production Astro and Studio builds, plus TypeScript checking, passed.
Planner checks covered equivalent Chicago/Pacific/India deadline instants, a winter offset, earliest-deadline order, capacity, revision impact, UTC calendar times, text escaping, UTF-8 line folding, remote effort/minute changes, and preservation of explicit overrides.
Browser interactions confirmed the timezone-preserving start, seven-hour revision impact, local decision recording, late schedules, and stale decision memory.
Anonymous public queries and the hosted browser confirmed the Sanity content and simulated approval. The exported ICS file was downloaded and inspected.

The scheduling model remains deliberately small: serial work using average daily capacity, with human minutes added separately. It does not model hourly availability or holidays. Prerequisites are a checklist, not an eligibility validator. The estimates are assumptions, and the rejection action has not been independently rehearsed in the live dataset.

Sanity Project Details

Project ID: 2mflxxa8
Dataset: production (public)
Published document types: competition, evidenceSource, sourceClaim, evidenceRevision, reviewDecision, projectNote

sourceClaim → competition + evidenceSource
evidenceRevision → competition + evidenceSource
reviewDecision → exact evidenceRevision
projectNote → estimate, verification, or simulation context

Sources remain independent documents with publisher, URL, and checked time. Claims preserve what was reported. A revision represents a proposed change, and a decision points to the exact revision reviewed. This avoids treating a copied date as both evidence and acceptance.

The initial seed contains 28 documents. The Studio rehearsal added one persisted simulated decision, bringing the public dataset to 29. The browser reads published content without an Authorization header. Public routing values are included; write credentials, personal account details, identity documents, and financial records are excluded from the dataset and public source.

Agent Session

No public agent transcript is attached. The build account above describes the work actually completed, and the repository contains the implementation and reproducible planner checks.

Submitted by @jayenichols. AI agents were development tools; there are no additional human teammates.