A manifest can pass JSON Schema validation and still describe an AI team that cannot work safely.
Every required field may be present. The kind may be supported. The permissions field may be an array exactly where the schema expects one. The validator exits successfully, even though one permission points to a capability that does not exist, a task names an unknown owner, and a risky action has no approval gate.
That is not a failure of JSON Schema. The mistake is treating structural validation as proof that the whole model is correct.
I want that boundary to stay explicit in NexFlow. A schema checks the shape of a manifest. Reference validation checks whether a set of manifests agrees with itself. A future runtime must decide what happens when an agent attempts a real action. Calling all three layers simply "validation" encourages tools to promise more than they can verify.
What the first layer catches
NexFlow describes AI developer teams through YAML manifests. They cover the project, actors and agent definitions, tasks, workflows, handoffs, capabilities, permissions, context, memory, providers, model profiles, prompt sets, retrieval profiles, events, and extensions.
YAML is readable in reviews, but tools need a stricter contract. Each supported manifest kind therefore maps to a JSON Schema.
The current repository validator parses YAML safely, converts each document into a JSON-compatible structure, selects a schema from its kind, and reports errors with file and instance paths. At the reviewed repository checkpoint, npm run validate validates 113 manifests against 17 schemas.
Schema validation is good at catching errors that do not require knowledge of another file. A permission effect, for example, is limited to the values published by the specification:
effect: allow
effect: deny
effect: approval_required
A value such as full_auto_magic should fail immediately. The same applies to missing required fields, an unsupported specification version, an unknown kind, a value of the wrong type, or an identifier that violates the expected format.
Human reviewers should not have to spend time finding those mistakes. A validator can remove them before the meaningful review begins.
A structurally valid mistake
The harder case begins when one manifest refers to another.
An agent definition might contain this fragment:
components:
permissionRefs:
- implementation_branch_work
- delete_everything
For JSON Schema, this may be a perfectly valid array of well-formed identifiers. The schema for that file does not necessarily know whether delete_everything exists in permissions.yaml, which capabilities it grants, or whether an appropriate approval gate covers it.
This is where semantic validation begins. It treats the manifest set as a connected model and resolves references across files.
For NexFlow, that includes checking task owners, workflow dependencies, handoff artifacts, permissions, capabilities, context sources, memory scopes, events, and extensions. A reference is not correct merely because it has the right syntax. The validator must find one valid target, check its target kind, and make sure adjacent policies do not contradict it.
The repository already includes a bounded semantic reference smoke check for maintained examples. At the same checkpoint, npm run semantic-smoke passes for seven example projects. This is useful evidence of current coverage, not a claim of complete semantic validation.
Why one large green check is misleading
Imagine a tool that prints one line:
Configuration valid
The user cannot tell what happened. Perhaps the tool only parsed YAML. Perhaps it applied JSON Schemas. Perhaps it resolved a small subset of references. Or perhaps its author expects the word valid to be read as a safety guarantee.
The supported boundary should be visible in the result.
- Structural validation confirms that a document matches a published shape.
- Semantic validation confirms only the relationships and constraints that the tool explicitly lists as supported.
- Runtime enforcement applies real permissions, credentials, approval gates, isolation, and audit rules when an action is attempted.
The first two layers can prepare a configuration for execution. They do not execute it, and they do not replace runtime controls.
A valid manifest cannot stop a dangerous command
JSON Schema does not isolate credentials or inspect actual access in GitHub, Linear, or a filesystem. It does not pause execution before a risky change. It cannot revoke a previously granted permission or record a real agent action in an audit log.
Those responsibilities belong to a runtime that applies the declared policy at the moment of action.
NexFlow does not currently provide a production runtime, an official production CLI, or ready-made provider integrations. The repository contains a specification, schemas, documentation, examples, and bounded validation tooling. That is enough to improve the model and its verifiable boundaries. It is not enough to claim that an AI team is being executed.
This precision matters beyond NexFlow. Any tool claiming support for a specification should name the level it supports and show evidence. "Supports NexFlow" is too broad. Structural validation, validation of specific relationships, and policy enforcement are different promises.
What a useful validation result should say
A useful validator should report more than a red or green state. On failure, it should identify the file, field path, validation layer, and reason. When a layer was not run, that omission belongs in the result too.
An unknown permission reference should look different from a field-type error. The user should be able to see that YAML parsing succeeded, JSON Schema validation passed, and semantic reference resolution failed. Diagnostics then help repair the model instead of announcing that something somewhere is wrong.
Successful output needs the same precision. Instead of a generic valid, a tool should say which schemas it applied, which semantic checks it ran, and which guarantees remain outside the current command.
The repository documentation keeps these boundaries separate in the validation model, the schema guide, and the conformance model.
Validation should reduce uncertainty
JSON Schemas give NexFlow a first practical layer of discipline. They catch authoring mistakes, stabilize manifest shapes, and give tooling a contract it can test. The next layer checks relationships. Execution begins later, when a system applies permissions and approval gates to real actions.
A manifest that passes schema validation is therefore a good starting point, not final proof.
A strong tool does not hide that distinction behind a green check. It shows which uncertainty it has removed and where a human or future runtime still has a decision to make.
Top comments (2)
This is the boundary I keep wanting more tools to expose. Schema validation is a cheap first gate, but the dangerous bugs are usually name resolution and authority checks across files. I would almost rather see the validator print shape passed, references unproven than let that green check look final.
Schema validation is necessary, but it only proves the manifest is shaped correctly. The harder test is whether the responsibilities, escalation paths, and handoff boundaries match how the team will actually operate.