DEV Community

Diya Burman
Diya Burman

Posted on

I Converted the order-api to OKF. Here's What I Found.

#ai #softwareengineering #agents #testing

Preface

I want to be upfront about something before we get into it. None of the frameworks in this article is mine. The ideas here come from two people who have been thinking about this stuff way harder and longer than I have — and they deserve full credit before I say another word.

Dan Shapiro — CEO of Glowforge, Wharton Research Fellow, and the person who gave this whole conversation a vocabulary. His blog post “The Five Levels: from Spicy Autocomplete to the Dark Factory” is the conceptual spine of everything I’m about to say. Read the original. It’s short, sharp, and will make you uncomfortable in the best way. danshapiro.com

Nate B. Jones — AI strategist, zero-hype practitioner, and the person whose YouTube channel made me realize I had been fooling myself about where I actually sat on this ladder. His video “The 5 Levels of AI Coding (Why Most of You Won’t Make It Past Level 2)” is what triggered this entire newsletter. natebjones.com — Watch the video

This newsletter — The Level 5 Engineer — is my public learning log. I’m a Senior Software Engineer and a Tech Lead, currently somewhere between Level 2 and Level 3 (in context of the title of this newsletter) on a good day. The goal is Level 5. I’m documenting the climb in real time — the frameworks, the tools, the mindset shifts, and the moments where I realize I’ve been doing it wrong. If you’re on a similar journey, pull up a chair.

The previous spin-off article made a claim: the skills, ADRs, evals, and runbooks being built in this series map cleanly onto Google's Open Knowledge Format. Same problem, different vocabulary.

Claims need testing. So I ran the test.

This article is the result of converting the order-api's docs/ directory into a conformant OKF v0.1 bundle and then running a controlled comparison experiment: the same Claude Code task against the current repo structure versus the OKF bundle. Two fresh agent contexts. Same task description. Document what each agent does differently.

The result was not what I expected.

The conversion

25 documents in scope across the docs/ directory plus CLAUDE.md at the project root.

The mapping:

Current type OKF type Count
ADR Decision 2
Eval Guardrail 3
Skill (all tiers) Methodology 6
Runbook Playbook 2
Reference docs Reference 9
Pedagogical examples Reference 3
CLAUDE.md Agent Standing Orders 1

Conversion involved two changes to each document. First, YAML frontmatter at line 1:

---
type: Guardrail
title: "Operation Scope Eval"
description: "Pre-flight check that must be answered before modifying app/main.py or any file in tests/."
tags: [eval, pre-flight, operation-scope, layer-3]
timestamp: 2026-06-28
---
Enter fullscreen mode Exit fullscreen mode

Second, a ## Related section at the bottom of every ADR, eval, skill, and runbook — cross-links to the documents most likely to be relevant when reading this one.

For eval-operation-scope.md, the related section links to ADR-001 and ADR-002 (whose invariants this eval enforces), the CLAUDE.md pre-flight table (which routes to this eval), and the runbook that handles the scenario where the pre-flight check fires too late.

For ADR-001-inventory-before-payment.md, the related section links back to the eval that enforces it at runtime, the Gherkin scenarios in tests/features/order_creation.feature that test the behavioral outcome, and the CLAUDE.md decision index entry.

Then index files at each directory level, following OKF spec §6:

docs/index.md                  — entry point for the bundle
docs/ADR/index.md              — both decisions with descriptions
docs/evals/index.md            — three evals with trigger summaries
docs/runbooks/index.md         — both runbooks with trigger scenarios
docs/skills/index.md           — three-tier structure explanation
docs/skills/tier1/index.md
docs/skills/tier2/index.md     — five tier-2 skills including deprecated
docs/skills/tier3/index.md
docs/log.md                    — bundle update history
Enter fullscreen mode Exit fullscreen mode

OKF conformance check after conversion:

Total docs: 24
Missing frontmatter: 0
All docs have frontmatter.
Enter fullscreen mode Exit fullscreen mode

All tests still passing. The conversion touched only docs/ files — no implementation, no feature files, no step definitions, no Pact files.

The experiment

One task. Two fresh agent contexts. No knowledge of what the other run did.

The task:

"Add a new endpoint to the order service: GET /orders/{order_id}/history — returns a list of status changes the order has gone through (created, confirmed, etc.) with timestamps. Write the Gherkin scenarios first. Apply the relevant skills. Run the pre-flight evals. Then implement."

Run A started with: CLAUDE.md + task description only.
Run B started with: CLAUDE.md + task description + docs/index.md as explicit starting point, with the instruction "The docs/ directory is an OKF knowledge bundle. Start by reading docs/index.md for an overview of available knowledge, then navigate from there."

Both runs documented which files were read before the first Gherkin scenario was written.

What happened

Run A — Navigation log (10 files, in order)

  1. docs/skills/tier1/output-formatting-standard.md — CLAUDE.md named it explicitly
  2. docs/skills/tier2/gherkin-scenario-quality-v2.md — CLAUDE.md named it explicitly
  3. docs/ADR/ADR-001-inventory-before-payment.md — CLAUDE.md decision index
  4. docs/ADR/ADR-002-fire-and-forget-notification.md — CLAUDE.md decision index
  5. docs/evals/eval-operation-scope.md — CLAUDE.md pre-flight table
  6. tests/features/order_creation.feature — style reference
  7. tests/features/order_status_good.feature — GET endpoint assertion style
  8. app/main.py — data model and in-memory store
  9. tests/features/notification_service.feature — count pattern style
  10. docs/evals/index.md — end-of-navigation check: confirmed nothing missed

Two tool calls to find the Gherkin skill. One call for each ADR. One call for the eval. Then code.

Run B — Navigation log (20 files, in order)

  1. docs/index.md — entry point (required by experiment)
  2. docs/skills/ directory listing — index mentioned skills/ subdirectory
  3. docs/ADR/ directory listing — index mentioned ADR/ subdirectory
  4. docs/evals/ directory listing — index mentioned evals/ subdirectory
  5. docs/skills/index.md — three-tier structure
  6. docs/ADR/index.md — ADR listing before individual ADRs
  7. docs/evals/index.md — which evals exist and what they trigger on
  8. docs/skills/tier2/index.md — find Gherkin quality skill by name
  9. docs/skills/tier2/gherkin-scenario-quality-v2.md — the skill
  10. docs/ADR/ADR-001-inventory-before-payment.md — pre-flight ADR check
  11. docs/evals/eval-operation-scope.md — pre-flight eval
  12. docs/ADR/ADR-002-fire-and-forget-notification.md — complete ADR check
  13. docs/evals/eval-environment.md — confirmed not triggered (Run A did not read this)
  14. docs/evals/eval-contract-preflight.md — confirmed not triggered (Run A did not read this)
  15. docs/skills/tier1/output-formatting-standard.md — formatting standard
  16. tests/features/order_status_good.feature — style reference
  17. tests/features/order_status_good.feature (second pass) — additional assertion style reference (Run A did not revisit this)
  18. app/main.py — data model
  19. tests/features/order_creation.feature — style reference
  20. docs/skills/tier2/step-definition-style.md — step definition conventions (Run A did not read this)

Five hops to find the Gherkin skill. Eight directory traversals before any ADR was opened.

The results

Q1: Did the agent find the relevant skill faster in Run B?

No. Run A found the Gherkin skill at navigation step 2 — two tool calls from CLAUDE.md. Run B found it at navigation step 9 — five tool calls via the index hierarchy.

OKF was slower by three navigation steps.

The reason is simple: CLAUDE.md names exact file paths. docs/skills/tier2/gherkin-scenario-quality-v2.md is in CLAUDE.md's skill table. The agent opens it directly. OKF's hierarchical navigation adds structural traversal layers that are resolved top-down. When a direct pointer already exists, hierarchical navigation is strictly overhead.

Q2: Did OKF cross-linking change which documents the agent consulted?

Yes — significantly.

Run B's agent read all three evals. Run A's agent read one (the one that fires). The docs/evals/index.md caused Run B's agent to read eval-environment.md and eval-contract-preflight.md and explicitly confirm they were not triggered. Run A's agent did not know those evals existed until step 10, when it checked docs/evals/index.md as a final verification.

The "Related" section in eval-operation-scope.md provided a third confirmation path to ADR-001 and ADR-002 — beyond CLAUDE.md's decision index and the ADR/index.md listing. Three independent paths to the same documents.

Q3: Were there documents found in one run but not the other?

Run B found, Run A did not:

  • docs/evals/eval-environment.md — proactively read and confirmed not triggered
  • docs/evals/eval-contract-preflight.md — proactively read and confirmed not triggered
  • docs/skills/tier2/step-definition-style.md — discovered via tier2/index.md; CLAUDE.md does not reference this skill in its skill table

That last item is the most important finding in the experiment.

step-definition-style.md is the skill that encodes five implicit conventions for writing step definition files — the fixture-chaining pattern, mock server state asserted via call log rather than response body, the time.sleep(0.3) timing, and the _post_order-style helper naming. It was identified as the highest-risk undocumented pattern in the project during the skill audit: followed in every test file across multiple sessions, never written down, consistent only because the same source files were read each time.

CLAUDE.md's skill table does not list it. An agent relying solely on CLAUDE.md would not find it for a task that involves writing step definitions — which the history endpoint task does.

Run B's agent found it because tier2/index.md listed it alongside the four other tier-2 skills. The OKF index is not selective — it lists everything in the directory. CLAUDE.md is selective — it lists what the author remembered to add.

Q4: Did the index.md change the order of understanding?

Yes — and this is the most structural difference between the two runs.

Run A built understanding specific-first: went directly to individual documents named in CLAUDE.md before having any overview of what existed.

Run B built understanding overview-first: read the bundle structure before reading any individual document, arriving at each document knowing what else existed in the same category.

This mattered for the eval space specifically. Run B's agent knew "there are three evals" before reading any of them. Run A's agent knew only "there is a pre-flight table in CLAUDE.md" and read only the eval that fires.

Q5: Did OKF frontmatter change any agent decision?

Not directly. The type, tags, and description fields were not cited as decision drivers in either run. The description field in index.md entries was used to confirm documents before opening them — but this was confirmation, not routing. Routing was driven by CLAUDE.md in both runs.

The frontmatter's most useful effect was indirect: the description field in index entries gave Run B's agent enough context to decide whether to open a file without opening it. A modest efficiency gain, not a qualitative change.

Q6: How many implicit decisions were made?

Run A: 10. Run B: 11. Difference of 1 — within noise.

OKF cross-linking does not reduce implicit decisions for Gherkin scenario writing. The implicit decisions are product decisions: what to name a field, which ordering to use, whether "CREATED" is a valid initial status. These come from specification gaps, not navigation gaps. No amount of infrastructure can supply them. That remains the human's job.

What OKF gives that CLAUDE.md does not

Completeness over selection. CLAUDE.md's skill table will always lag the actual document count. It lists what the author remembered to add. OKF's index never lags — the index is derived from the documents. Every file in tier2/ appears in tier2/index.md. Every eval in evals/ appears in evals/index.md. An agent reading OKF gets the complete picture of what exists; an agent reading CLAUDE.md gets the author's current mental model of what the agent needs.

Overview-first navigation. An agent starting from docs/index.md knows the shape of the knowledge bundle before it navigates into it. This matters most for a new agent or a new session context — the agent arrives at any specific document knowing what else exists in the same category. CLAUDE.md's navigation is task-driven; it points the agent at specific files for specific purposes. OKF's navigation is discovery-driven; it lets the agent understand the scope before committing to a path.

Confirmed non-applicable documents. Run B's agent confirmed that two evals were not triggered and documented that confirmation. Run A's agent may have silently assumed those evals did not exist. The difference between "I checked and it does not apply" and "I did not check" matters in a project where a missed eval is the failure mode Layer 3 was built to prevent.

What CLAUDE.md gives that OKF cannot replace

Direct routing is faster. When CLAUDE.md names an exact file path, the agent opens it in one tool call. When OKF provides a hierarchy, the agent traverses N levels. For targeted navigation on a known task, CLAUDE.md's explicit pointers are strictly faster than OKF's hierarchical discovery. The three-step speed advantage in Run A is real.

Behavioral instructions. CLAUDE.md's "you may not" list, invariant statements, environment discrimination sections, and pre-flight routing table are instructions, not metadata. "Before modifying app/main.py, run docs/evals/eval-operation-scope.md" is a behavioral instruction. OKF can express "eval-operation-scope.md exists and its description says it intercepts app/main.py modifications" — but an agent must infer from that description that it should run the eval. Inference is the failure mode. The routing instruction is what prevents inference.

Behavioral constraints. CLAUDE.md's HALT conditions and prohibition list ("you may never push directly to main," "do not add continue-on-error to pipeline jobs") cannot be represented as OKF frontmatter. They are instructions, not structured knowledge. OKF formalizes what exists; CLAUDE.md governs what must be done and what must not be done. These are different layers of the same system.

The honest synthesis

OKF and CLAUDE.md solve different problems.

CLAUDE.md is an instruction document that happens to contain a knowledge map. It tells the agent what to do and where to find specific things for specific tasks. Its strength is precision and speed for known tasks.

OKF is a knowledge map that happens to be readable by agents. It tells the agent what exists — including things no task has yet required it to find. Its strength is completeness and discovery for unknown scope.

When both exist: CLAUDE.md's direct pointers are faster for targeted navigation. OKF's index hierarchy is more complete. The combination catches what each one misses alone. CLAUDE.md's skill table misses step-definition-style.md; OKF's tier2/index.md surfaces it. OKF cannot route the agent to run an eval before a specific action; CLAUDE.md's pre-flight table does exactly that.

The finding that runs counter to the intuitive case for OKF: for a project with a well-maintained CLAUDE.md, OKF does not replace or accelerate what CLAUDE.md already does. It fills the gaps that CLAUDE.md leaves uncovered. Those gaps are real — step-definition-style.md is a high-priority skill that CLAUDE.md's skill table does not reference, meaning any agent relying solely on CLAUDE.md for navigation would miss it entirely — but they are not the primary navigation problem. They are the completeness problem.

The right architecture is both. CLAUDE.md for routing and behavioral instructions. OKF for structural completeness and cross-document relationships. Neither replaces the other. They solve different problems at the same layer of the stack.

The order-api repository is now an OKF v0.1 conformant bundle. The full conversion — frontmatter, index files, cross-links, and log.md — is in the repo.

Further reading

This article was written with the assistance of AI tools.

Top comments (0)