No run Command. How Do You Prove a CLI Will Not Execute an Agent?

A command line interface may have no run, execute, or start command. That still does not prove it will execute nothing.

An ordinary validation command can load a module with a side effect, fetch a remote schema, read a secret from the environment, start a child process, or overwrite a file. None of those actions has to be called agent execution. The capability can arrive inside a new dependency, a convenient helper, or another operating mode.

That boundary matters in NexFlow. The current repository CLI can discover manifests, validate their structure, expose bounded inspection, build a static graph, and create a minimal set of starter files. It is not a runtime, and it should not acquire runtime authority as its validation features grow.

Documentation alone is not enough. I wanted a testable answer to a stricter question: can the real CLI complete its useful commands when network access, process execution, and arbitrary file writes are technically denied by the test harness?

That question led to a separate system of no-runtime checks. It does not prove absolute security or turn Node.js into a sandbox. It does make one specific contract visible, and it breaks the build when the code quietly crosses that boundary.

Describe effects before commands

A command list explains what the user can see. A safety boundary needs a different answer: what may each command do to the outside world?

The NexFlow prototype stores that answer in an explicit effect budget. Every command disables:

• network access;
• process execution;
• credential access;
• provider calls;
• executable extension loading;
• runtime preflight;
• workflow execution;
• background work.

Project reads also have boundaries. discover reads selected manifests. validate, inspect, and graph additionally read local schemas. None of those commands receives a budget for modifying the project.

init has one narrow exception. It may create only a fixed set of starter files in an explicitly selected, existing directory. This distinction matters. A safe validation CLI does not have to be completely free of side effects, but every allowed effect must be named, bounded, and tested separately.

In simplified form, the budget looks like this:

const disabledRuntimeEffects = Object.freeze({
  network: false,
  processExecution: false,
  credentialAccess: false,
  providerCalls: false,
  extensionLoading: false,
  runtimePreflight: false,
  workflowExecution: false,
  backgroundWork: false
});

const CLI_EFFECT_BUDGETS = Object.freeze({
  discover: effectBudget("selected-manifests", "none"),
  validate: effectBudget("selected-manifests-and-local-schemas", "none"),
  inspect: effectBudget("selected-manifests-and-local-schemas", "none"),
  graph: effectBudget("selected-manifests-and-local-schemas", "none"),
  init: effectBudget("built-in-template-and-existing-targets", "fixed-starter-files")
});

The objects are frozen, and an unknown command receives no default budget. The parser and dispatcher use the same closed set of operations. If an execute command appears in the code without a separate reviewed decision, the CLI stops before touching the project.

This is not a complete permission system. It solves a narrower problem: the current repository tool must not accidentally acquire capabilities intended for a future runtime.

Static checks look for an unexpected path out

The budget is useless if the rest of the code can bypass it. The next layer fixes the CLI's own module graph and the allowed imports for each file.

The check knows which local modules the entry point may load and which dependencies are allowed for discovery, schema validation, inspection, graph construction, and initialization. A new import changes that graph and requires an explicit review of the allowlist.

The checked source also denies several short paths to new authority:

• reading process.env;
• CommonJS require and createRequire;
• dynamic evaluation through eval or Function;
• global fetch and WebSocket;
• native loading through process.binding or process.dlopen.

This boundary is deliberately blunt. It may need to change when the CLI gains a legitimate dependency. That friction is useful. An expanded surface should become visible in review, not arrive unnoticed with a convenient feature.

An import check still does not prove that an allowed dependency is safe. It only records a known execution path. Dependency compromise, new platform capabilities, and bypasses through APIs that the harness does not intercept remain separate risks.

A negative harness denies effects during execution

Static search is not enough. The code must run in conditions where a forbidden action fails the test.

Before launching the CLI, the negative harness replaces dangerous Node.js APIs. It blocks child_process, HTTP, HTTPS, HTTP/2, TCP, TLS and UDP clients, DNS, worker threads, server creation, and a broad set of file operations.

Global fetch and WebSocket are also replaced with functions that throw an error containing a forbidden-effect marker.

The principle fits into a few lines:

replaceFunctions(childProcess, [
  "exec", "execFile", "fork", "spawn", "spawnSync"
]);

replaceFunctions(https, ["createServer", "get", "request"]);

globalThis.fetch = () => denied("fetch");

This kind of harness is easy to overestimate. The tests therefore begin by testing the denial itself. Separate probes attempt network access, process execution, and a file write. Each attempt must fail with the expected error. If this stage does not pass, a later successful CLI run proves nothing because the guard may simply be inactive.

Only then does the harness run real commands.

--help and --version must work without reading the project. discover, validate, inspect, and graph run against a test project while runtime effects remain blocked. Every command is checked for its exit code, the absence of a denial marker, and executionAuthorized: false in its machine-readable response.

The test environment contains values that resemble provider and cloud credentials. These values act as canaries. None may appear in the output. This is not a scanner for every possible secret. It is a precise check that the CLI did not start reading ambient credentials merely because they were available to the process.

Absence of change also needs evidence

A successful exit code does not show what the command left behind.

Before every read-only run, the test takes a snapshot of the selected project: directory structure, file names, and contents. It takes another snapshot after the command. Any difference violates the boundary, even when the CLI returns a polished report and never touches the network.

This catches an accidentally rewritten manifest, a cache beside the source files, a temporary file, a log, or an automatically corrected configuration.

The test then adds command-like and remote-address fields to a manifest. For discover, they must remain inert data. The CLI does not execute the string, open the address, print the canary secrets, or modify the project.

Runtime-shaped commands are tested separately: run, execute, start, serve, deploy, provider, mcp, login, and others. They must return a usage error with exit code 2 before any side effect. The project snapshot must remain identical.

This tests dangerous user expectations as well as current features. A word that resembles a future command must not fall through to a generic handler and activate an unknown path.

An allowed write needs its own negative contract

init cannot be read-only or it would create no starter project. Instead of general write access, it receives a fixed target set:

• project.yaml;
• actors.yaml;
• agents.yaml.

The test creates an existing directory with an unrelated keep.txt, runs init, and checks the exact resulting directory contents. The unrelated file must remain unchanged. No new target other than the three allowed files may appear.

The machine-readable result also reports reviewRequired: true and executionAuthorized: false. Creating configuration does not authorize the system to execute what that configuration describes.

There is a useful general rule here: an exception to a denial should be narrower than the denial itself. "The CLI may write files" is too broad. "The init command may create three named files in an explicit directory and may not overwrite a conflict" is testable.

One large test does not replace a scenario catalog

The negative harness answers questions about forbidden effects. It does not verify every user-facing promise.

NexFlow also has a catalog of 15 reproducible scenarios. Each scenario names a command, arguments, an input directory, and an expected result. The input is copied into a temporary directory before execution, so the test uses the real CLI without risking changes to maintained fixtures.

The catalog currently executes 280 assertions. It checks:

• process and machine-response exit codes;
• one JSON object in stdout and empty stderr;
• response conformance to JSON Schema;
• completed structural checks;
• diagnostic codes;
• the number of discovered documents;
• the size of bounded inspection and the static graph;
• the absence of absolute paths and environment values in output;
• unchanged inputs for read-only commands and error scenarios;
• the exact file set after init;
• a constant executionAuthorized: false value.

The catalog separates scenario data from assertion code. A new meaningful failure or supported case can become another publicly described entry while the same test program applies the contract consistently.

The numbers 15 and 280 do not prove quality by themselves. They are a current, reproducible property of the repository. When the catalog changes, those values will change too. They should be obtained from a fresh test run before publication, not copied from an older article.

CI checks the boundary separately

The prototype has a dedicated GitHub Actions workflow. On work involving main and develop, it runs eight groups of CLI checks:

1. commands and manifest discovery;
2. structural validation;
3. machine-readable diagnostics;
4. bounded inspection;
5. static graph generation;
6. starter initialization;
7. the no-runtime boundary;
8. cataloged fixture scenarios.

The separation helps diagnosis. If the JSON response shape breaks, the team sees an output-contract failure. If code attempts network access, the runtime-boundary check fails. If one fixture changes a diagnostic code, that failure remains separate from general schema validation.

One green integration test often hides the reason for a failure. Several focused checks show which promise stopped holding.

What these tests actually prove

For the current modules and dependencies, the tests provide a specific regression guarantee: implemented commands can complete their scenarios while common network, process, and file APIs are denied by the harness. Read-only commands preserve the selected project, runtime commands are rejected, environment canaries do not enter the response, and init stays inside its fixed file set.

The proof boundary ends where the harness ends.

It is not an operating-system sandbox. It does not protect against a compromised process, native code, an unintercepted API, dependency substitution, or concurrent file changes outside the checked scenario. It does not prove the safety of a future runtime because that runtime is absent from these commands. It also does not establish conformance to a future reference CLI. Its architecture, packaging, stable contract, and supported scope have not been approved.

Those limitations do not make the check useless. They prevent a narrow engineering guarantee from turning into a broad marketing promise.

Apply the approach to another CLI

For a tool that must analyze a project without executing it, I would start with six steps.

First, list effects, not only commands: network, processes, environment, secrets, file writes, module loading, and background work.

Then give every command a closed budget. An unknown command or an unknown budget field should fail closed.

Next, build a test harness that breaks forbidden APIs. Add probes that prove the denial is actually active.

For read-only operations, compare state before and after. Use canaries in the environment so an accidental leak is easy to detect.

Limit allowed exceptions, such as starter-file generation, to exact paths, names, and conflict behavior.

Finally, move user scenarios into a catalog and run the safety boundary as a separate CI step. A new import, command, or effect then becomes an explicit contract change.

Refusal is a product feature

CLI demos usually show what a tool can do. At a specification boundary, what it consistently refuses to do matters just as much.

The absence of a run command is only an interface signal. The real boundary appears when network access, processes, credentials, and arbitrary writes receive an explicit zero budget, while real commands continue to work under technical denial of those effects.

This test does not promise safety for every future version. It records a more useful claim: today, this tool can complete its validation work without quietly acquiring runtime authority.

Comments

[Pushpendra Agrawal]

this is a sharp way to think about it. the harder case is when the CLI itself is fine but a dependency it pulls in later gets that runtime authority quietly through a version bump nobody reviewed. does the import allowlist get re-checked on every dependency update in CI, or only when someone touches the CLI's own code?