Bobai Kato for Ota

Posted on Jul 17 • Originally published at ota.run

Pressure-testing Ota on Bedrock: query identity as replay evidence

#pressuretesting #aievaluation #replay #sql

Overview

Bedrock, created and maintained by Vinicius Pereira, is not a
normal application test repo.

It is a natural-language-to-SQL stability harness. It records generated SQL across repeated runs,
replays the committed fixture against a frozen SQLite store, compares the results with a defended
answer key, and blocks a candidate when reliability regresses.

That made it a strong pressure target for a more specific Ota question:

when a verification lane replays prior model behavior, how should Ota distinguish the inputs that
defined the replay from observations that explain what happened?

The answer cannot be one undifferentiated receipt field.

Why Bedrock mattered

Bedrock already had the useful artifacts in its repository:

data/fixture.jsonl: one recorded SQL query per question and run
data/store.db: the frozen SQLite state
data/baseline.json: the defended regression baseline
docs/scorecard.md: the rendered stability result

Those artifacts answer different questions.

A committed fixture, store, and baseline are replay inputs. They define the deterministic lane.
A repeated SQL trace is observed behavior. It can show that one question produced different query
shapes, but it is not an input Ota used to decide whether the current command could start. In this
repo, fixture.jsonl has a valid dual role: the deterministic replay consumes it and Ota parses it
as historical query evidence. That does not make it independent corroboration; it is one captured
artifact with two explicitly separate meanings.

That distinction became the pressure boundary.

What the contract models

The deterministic gate now declares the replay inputs explicitly:

gate:
  replay_inputs:
    - id: recorded_sql
      kind: static_file
      path: data/fixture.jsonl
    - id: frozen_store
      kind: static_file
      path: data/store.db
    - id: defended_baseline
      kind: static_file
      path: data/baseline.json

Those files are captured as declared, static replay truth.

The recorded query trace is declared separately:

```yaml | Witnessed Query Trace
gate:
witnessed_observations:
query_traces:
- id: recorded_sql
path: data/fixture.jsonl




That is the important Ota boundary.

- `replay_inputs[]` says what the deterministic gate consumed
- `witnessed_observations.query_traces[]` says what behavior the fixture contains

The second must not be promoted into the first. Otherwise a receipt can make historical model
output look like a pre-execution decision input, which weakens replay and evidence semantics.

## What the replay can honestly prove

Bedrock's offline path is deliberately narrow and useful:

- replay the committed SQL fixture
- execute it against the committed SQLite store
- compare the result set with the defended answer key
- compare the candidate score with the committed baseline

The contract makes that boundary visible:



```yaml | Offline Gate
gate:
  command:
    exe: python
    args:
      - -B
      - main.py
      - gate
  safe_for_agent: true

That proves the declared fixture, store, and baseline path. It does not prove that a live model
will generate the same SQL tomorrow, or that a live database has not changed.

Those claims remain separate in the contract's record:live lane:

record:live:
  effects:
    writes:
      - data/fixture.jsonl
    network: true
    network_kind: integration_test
    external_state:
      - anthropic_api

That lane reaches a live model and rewrites the recorded fixture. It is intentionally outside the
deterministic, agent-safe replay workflow.

What Bedrock exposed in Ota

Bedrock helped close an Ota evidence-model gap.

The initial temptation was to place per-record query identity under receipt
evaluated_inputs[]. That would have been wrong. A query trace is execution evidence, not a
current-run input.

Ota now keeps the split explicit:

declared static files stay under receipt evaluated_inputs[]
historical query identities and divergence summaries live under receipt witnessed_observations.query_traces[]

That gives later comparison and correlation a cleaner foundation:

inputs can remain unchanged
observed query behavior can still diverge

Neither fact is allowed to impersonate the other.

Bedrock also made one remaining boundary clear: its live recording path depends on a generic,
unpinned pip requirements lane. Ota does not yet own that as typed dependency hydration. The
contract names it as outside the deterministic proof instead of hiding it in setup shell.

What the matrix proves

The pressure matrix is intentionally offline and cross-platform:

ota validate
ota doctor
ota tasks --use
ota tasks --safe --use
agent-mode task dry-run for the stability gate
agent-mode workflow dry-run and preparation for verify
deterministic test execution
scorecard replay against the committed SQL fixture
baseline gate execution
archived receipt for the declared workflow
native coverage on Ubuntu, macOS, and Windows
the same replay workflow in Ota's pinned Python container context

This is not a claim that Bedrock's live model lane is hermetic. It is proof that the deterministic
replay lane is explicit about its inputs, output evidence, and boundary. The contract's file checks
assert artifact presence; Ota then captures SHA-256 source identities before execution and compares
them during replay. Those observed identities are not yet contract-declared expected digests, so
they attest what the lane used rather than pinning a separate immutable contract assertion.

The released v1.6.24 pressure matrix passed across native Ubuntu, macOS, Windows, and the
container replay lane on Ota's fork: run #29572073749.

Why this repo mattered

Bedrock clarified a product rule that applies beyond AI evaluation:

do not collapse observed behavior into declared execution inputs.

For this repository, a clean replay input can narrow or acquit the input class it actually names.
A divergent query trace can explain a flap. Neither artifact should overclaim the other role.

That is the right foundation for Ota's replay and receipt model: declared truth, execution truth,
and witnessed evidence remain linked, but separate.

Top comments (1)

Vinicius Pereira • Jul 17

Author of bedrock here. You landed on the exact seam the harness is built around. replay_inputs are what I commit and feed deterministically; witnessed_observations are what the model actually did. Let the second masquerade as the first and the replay can only confirm itself, a test that lies because it encoded the model's old output as its own assumption. Binding integrity to SHA-256 identity instead of shape is the same call throughout the repo: a frozen store is proven by its digest, not by a file that happens to be named like one.