Pressure-testing Ota on Hasura: raw Kubernetes manifests and honest kubectl proof

Overview

Hasura exposed a useful execution-governance boundary:
raw Kubernetes manifests without Helm or Compose owning the operational path.

That sounds simple. It is also easy to overclaim.

kubectl can parse and render a manifest locally without contacting a cluster. A Kubernetes API
can accept those objects without proving that Hasura became ready. A green command can therefore
mean at least three different things:

• the manifests were interpreted locally;
• a cluster accepted the declared resources;
• the application started and became usable.

This pressure test asked whether Ota could govern the first two claims while refusing to imply the
third. The contract now installs released Ota v1.6.24 directly, so the version used by humans,
agents, and CI is contract truth rather than workflow-local bootstrap glue.

The pressure slice

The work stayed deliberately scoped to
install-manifests/kubernetes.
That directory contains the raw Hasura deployment and service manifests used by this test.

The Ota contract owns:

• released Ota v1.6.24 bootstrap identity;
• cross-platform kubectl acquisition;
• non-mutating local manifest tasks;
• explicit cluster-mutating apply tasks;
• a safe verification workflow;
• a separate deployment workflow;
• task and workflow dry-run coverage;
• the task and workflow truth consumed by the three-OS pressure matrix.

It did not attempt to govern all of Hasura or turn this narrow infrastructure directory into an
application-readiness contract.

What the contract says

Ota owns its own bootstrap identity

The pressure contract does not ask CI to choose an implementation branch or duplicate installer
commands. It pins the released version through the contract:

agent:
  bootstrap:
    ota:
      source:
        kind: version
        version: v1.6.24

The GitHub workflow uses ota-run/setup with source: contract, so the workflow consumes this
identity instead of becoming another source of Ota version truth.

Ota owns the tool

The contract does not hide kubectl installation in a shell preamble:

tools:
  kubectl:
    version: "*"
    acquisition:
      provider: apt
      package: kubectl
    platforms:
      macos:
        acquisition:
          provider: brew
          package: kubernetes-cli
      windows:
        acquisition:
          provider: winget
          package: Kubernetes.kubectl

That gives humans, CI, and agents the same declared tool requirement instead of three separate
installation conventions.

Local verification is non-mutating

The safe verification lane uses kubectl annotate --local. It asks kubectl to load and emit the
object without contacting a Kubernetes API:

k8s:preview:deployment:
  command:
    exe: kubectl
    args:
      - annotate
      - --local
      - -f
      - deployment.yaml
      - ota.run/preview=true
      - -o
      - yaml

The aggregate keeps both manifests in one canonical verification path:

verify:
  description: Render the focused raw Kubernetes manifest set without mutating the cluster
  aggregate:
    tasks:
      - k8s:preview:deployment
      - k8s:preview:service

This proves that the installed kubectl can interpret and render the selected manifests. It does
not prove server-side admission, policy compatibility, resource creation, or application
readiness.

Cluster mutation is not agent-safe verification

The apply tasks declare their external effect instead of disguising deployment as another check:

k8s:apply:service:
  command:
    exe: kubectl
    args:
      - apply
      - -f
      - svc.yaml
  effects:
    external_state:
      - kubernetes

Those tasks are intentionally excluded from agent.safe_tasks. The workflow layer also gives
verification and deployment different operational meanings:

workflows:
  default: verify

verify:
  intent: ci_verification
  run:
    task: verify

deploy:
  intent: deployment
  run:
    task: k8s:cluster:proof

Ota is not merely storing two commands. It is declaring which path is routine verification and
which path mutates an external system.

The proof ladder

The pressure design keeps three distinct proof levels:

Level	Evidence required	Current v1.6.24 result
Local manifest interpretation	Real kubectl annotate --local execution for both manifests	Proved locally on macOS
Kubernetes API and resource acceptance	Real kubectl apply followed by kubectl get in an ephemeral kind cluster	Hosted Ubuntu reproof pending
Hasura application readiness	Pod readiness, database connectivity, service reachability, and application behavior	Not proved

This distinction is the main engineering result. A created Deployment and Service are stronger
evidence than local YAML handling, but they are still not proof that Hasura became ready.

Current Ota 1.6.24 evidence

The upgraded contract was exercised with released Ota v1.6.24, commit c1ab23a58.

The current local proof covers:

• ota validate;
• ota doctor;
• ota tasks --use;
• ota tasks --safe --use;
• ota run verify --dry-run --json;
• ota up --workflow verify --dry-run --json;
• real ota run verify --native --agent;
• both real kubectl annotate --local tasks.

ota doctor reported the repository slice ready with no findings. The task and workflow previews
were runnable, and the real safe verification closure completed successfully.

The pressure workflow is prepared to repeat contract discovery and dry-run coverage on Ubuntu,
macOS, and Windows, with real kind cluster apply-and-observe proof on Ubuntu. That hosted matrix
must pass against the upgraded contract before this draft is published as cross-platform
v1.6.24 evidence.

Matrix design

The matrix is asymmetric by design. It does not pretend that every operating system owns a
Kubernetes runtime:

Lane	Contract and discovery gates	Dry-run coverage	Real execution	Honest boundary
Ubuntu	Validate, Doctor, task inventory, safe-task inventory, workflow inventory, execution topology	verify, cluster proof, and both workflows	Local manifest verification plus apply-and-observe against an ephemeral kind cluster	Kubernetes resource acceptance, not Hasura readiness
macOS	Validate, Doctor, task inventory, safe-task inventory, workflow inventory, execution topology	verify, cluster proof, and both workflows	No hosted cluster execution	Contract and admission compatibility only
Windows	Validate, Doctor, task inventory, safe-task inventory, workflow inventory, execution topology	verify, cluster proof, and both workflows	No hosted cluster execution	Contract and admission compatibility only

The Ubuntu lane creates the kind cluster in GitHub Actions, outside the Ota contract. Ota then
owns the declared kubectl task and workflow execution inside that provider-owned boundary. This
keeps cluster provisioning truth separate from manifest execution truth.

The refreshed matrix acceptance bar is:

1. ota-run/setup installs the contract-pinned v1.6.24 release.
2. ota --version confirms the installed release before any contract claim is evaluated.
3. ota doctor passes without a tolerated blocked state.
4. The safe verify closure executes through the real agent boundary.
5. Successful JSON, topology, and execution-receipt evidence is retained for every lane.
6. Ubuntu proves resource apply and observation without promoting that result to application readiness.

Explicit boundaries

The pressure result remains narrow:

• Contract-owned and currently proved with v1.6.24: Ota bootstrap identity, kubectl acquisition shape, task/workflow selection, safe-task separation, local validation and discovery, dry-run admission, and real local manifest execution on macOS.
• Contract-owned and awaiting hosted reproof: Ubuntu kind resource apply/observation and the Ubuntu, macOS, and Windows matrix.
• Repo-owned and outside this slice: editing deployment.yaml with the real database URL.
• Not proved: Hasura pod readiness, database connectivity, service reachability, application behavior, production-cluster policy, RBAC, storage, ingress, load balancers, and rollback.
• Not governed by this contract: creation and teardown of the CI kind cluster, which remained GitHub workflow infrastructure.
• Not repo-global: no result here should be read as proof of the wider Hasura repository.

What Hasura taught Ota

Raw manifests do not need a giant abstraction. They need precise ownership and honest evidence.

Ota adds value here by keeping four truths together:

• how the required tool is acquired;
• which command path is safe verification;
• which path mutates Kubernetes;
• what each successful lane actually proved.

The important outcome is not that Ota can run kubectl. Any shell can do that. The outcome is that
humans, CI, and agents receive the same governed execution path without being told that local
rendering, cluster acceptance, and application readiness are interchangeable.

---

Originally posted here: https://ota.run/blog/pressure-testing-ota-on-hasura-raw-kubernetes-manifests-and-honest-kubectl-proof-4m2h