DEV Community

Amit
Amit

Posted on Originally published at artificialcuriositylabs.ai

Harness vs. Runtime: When to Graduate

#agents #agentcore #bedrock #harness

The short version

I ran the same support-triage task on both AgentCore run modes. Harness stood up a working agent in 20.4 seconds against 398 lines of Runtime Python, and it passed the IAM-gated and no-tool scenarios cleanly. It could not enforce per-user Cedar policies on the refund tool.

Both modes end up on the same compute. CloudTrail records harness operations under AWS::BedrockAgentCore::Runtime. The difference is who owns orchestration — you write the code (Runtime) or AgentCore handles it (Harness).

The documented graduation triggers are hooks, custom loops, and bidirectional streaming. The barrier I hit first was none of those — it was per-user outbound identity. What follows is what worked, what broke, and where the hard boundary is.

What Harness gives you without writing agent code

For nearly every AgentCore capability, Harness delivers it in configuration: Memory (all four strategies), Gateway tool wiring, Browser and Code Interpreter as built-in tools, outbound Identity, automatic Observability, model switching mid-session, versioning, and VPC networking. All available without customer code.

Runtime offers the same capabilities — with "you write the code required" for each. The session manager, the memory client, the tool orchestration, the span instrumentation — all available, all require code.

The three things the documented feature grid says Harness cannot do:

✗  Hooks (AfterToolCallEvent, BeforeToolCallEvent)
✗  Non-agent-loop patterns (graph, workflow, tree-of-thought)
✗  Bidirectional streaming
Enter fullscreen mode Exit fullscreen mode

Those are the published graduation triggers. A fourth one — per-user outbound identity — isn't documented; I found it by running the config.

H1: Two API calls, no entrypoint code

The Harness create-and-invoke path is exactly as small as advertised. The entire orchestration for a triage agent — model, system prompt, one Gateway tool, memory — is about 25 lines passed to create_harness(). The equivalent Runtime agent is 398 lines. Harness is roughly 6% of the Runtime file.

Two measurements stand out:

  • Time to READY: 20.4 seconds. From create_harness() to a harness that answers, a little over 20 seconds. The equivalent Runtime path builds a container — around 180 seconds before the first invoke. Harness is ~9× faster to first response because there's no image to build.
  • The invoke is not SSE. boto3's invoke_harness returns a proper event stream. A JWT-protected harness requires raw HTTPS POST to /harnesses/invoke?harnessArn=... and parsing an application/vnd.amazon.eventstream body with botocore.eventstream.EventStreamBuffer.

Two scenarios passed cleanly:

  • The no-tool password reset: correct steps, clean end of turn, no tool call. Passed.
  • The web-search region question: the harness called the WebSearch gateway tool twice and cited live results. Passed.

Both authorize outbound with the harness's own IAM role (SigV4). There is no per-user identity involved. This is Harness at its best: a governed tool, wired by ARN, no code.

Then I pointed it at the Cedar-gated refund tool. It stopped there.

H2: The wall is identity, not hooks

The support agent has a refund tool behind a CUSTOM_JWT gateway with a Cedar policy engine in enforce mode. The policies are per-user:

permit  alice  process_refund  when amount < 500      // standard tier
forbid  bob    process_refund                          // blocked, any amount
permit  carol  process_refund  when amount < 2000     // enterprise tier
Enter fullscreen mode Exit fullscreen mode

Cedar evaluates on the principal — explicitly the invoking user's identity from the JWT sub claim.

The Runtime path is trivial: the agent forwards the caller's own inbound JWT straight through to the gateway. alice's refund is permitted, bob's is forbidden, on the first call, for any user, with no extra machinery.

A Harness has no code, so it can't forward anything. It authorizes outbound calls through its configured outboundAuth, which offers exactly three modes: awsIam, none, and oauth. I tried the two that could carry identity.

awsIam outbound. The harness signs the gateway call with its own execution role. Cedar sees the role, not a user — there is no OAuthUser principal, so no per-user policy matches. The harness simply is not alice. Correct behavior, wrong outcome.

oauth outbound. This is the intended path: the harness mints the invoking user's token from AgentCore's Identity token vault. It's also where the real work was. Making it go took five distinct fixes, each a genuine requirement:

  1. The response is a binary event stream, not SSE — parse with EventStreamBuffer, and headers are plain strings.
  2. The execution role needs vault access — bedrock-agentcore:GetResourceOauth2Token plus secretsmanager:GetSecretValue. Without it the tool fails with AccessDeniedException.
  3. The grant type defaults to the wrong flow. oauth outbound defaults to CLIENT_CREDENTIALS (machine-to-machine). The provider is user-federation (3LO) — it's either M2M or 3LO, never both. The request fails with Error parsing ClientCredentials response. Fix: set grantType=AUTHORIZATION_CODE.
  4. Authorization-code outbound needs a return URL — defaultReturnUrl, registered on the workload identity.
  5. The token is vaulted under the wrong identity. This is the blocker.

The vault is keyed on (workload_identity, user_id). I had pre-consented users and stored their tokens under a standalone workload identity. But a Harness calls the vault under its own workload identity — minted when the harness is created. It looks up alice under that identity, finds nothing, and falls back to starting a fresh browser-consent flow — which a headless server invocation cannot complete. Every case failed with You must provide a ResourceOauth2ReturnUrl to proceed with this flow.

Why re-vaulting doesn't fix it

You could fix issue 5 mechanically: create the harness, read back its generated workload identity, register return URLs on it, run the 3LO consent loop keyed to that identity, then invoke. It would work. It also carries three operational costs:

  • The harness's workload identity doesn't exist until the harness does, so you can't pre-consent users ahead of deploying.
  • It changes on every rebuild — a new suffix each time — so re-consenting all users becomes part of your deploy.
  • 3LO requires per-user browser consent the first time regardless, which is inherent to the flow.

The one clean escape would be token passthrough — let the harness forward the inbound user JWT unchanged, the way Runtime code does. The gateway target model has a passthrough concept, but it is not exposed on the harness outboundAuth surface. Token exchange (OBO, RFC 8693) is the other theoretical route, and it's dead because Cognito doesn't implement it.

The honest finding: per-user Cedar through a Harness is architecturally possible and operationally impractical. Not because policy enforcement fails — Cedar at the gateway is identical for both paths — but because the code-less path binds outbound identity to an ephemeral, per-deploy workload identity and a consent step it can't drive headless.

The decision rule

Use Harness until you hit one of these. The first three are documented; the fourth is the one this run added.

Trigger 1 — Hooks. You need to intercept what the model sees after a tool call. The AfterToolCallEvent pattern — sanitizing an authorization failure, normalizing output, grounding the model on a deterministic fact — is Runtime-only. If your tool returns something the model must interpret rather than a clean fact, you need the hook.

Trigger 2 — Custom loop logic. Tree-of-thought, graph routing, supervisor/worker, A2A handoffs — any pattern where the orchestration loop itself carries business logic. Harness runs one managed loop; you can't fork it.

Trigger 3 — Model-level controls. Prompt-cache keys, extended-thinking budgets, per-token routing between a cheap model and an expensive one. Harness bedrockModelConfig exposes modelId, temperature, maxTokens, topP. Anything finer belongs in code.

Trigger 4 — Per-user outbound identity. If a tool enforces authorization on the end user's identity — per-user Cedar, an OAuth API scoped to the user, anything that needs the caller's own token downstream — the Harness path forces you through vault tokens keyed to an ephemeral workload identity plus a consent flow it can't run headless. Runtime forwards the caller's JWT in one line. This is the trigger the feature grid doesn't name, and for identity-aware agents it's the one you hit first.

What this means for the support agent

The support agent needs Runtime for two independent reasons, each tied to specific evidence.

The first is Trigger 1. The AfterToolCallEvent hook intercepts the policy engine's raw output and replaces it with a clean REFUND APPROVED / BLOCKED: fact before the model writes the customer reply. Without it, the model infers from raw authorization data — sometimes correct, sometimes fabrication. Cedar at the gateway works in Harness; the deterministic post-processing of Cedar's decision does not.

The second is Trigger 4. The refund tool is per-user Cedar. Runtime forwards each caller's JWT straight to the gateway, so the standard-tier user is permitted and the forbidden user is blocked on the first call with no vault, no pre-consent, no workload-identity juggling. The Harness equivalent is the five-layer, per-deploy, consent-bound path above.

Both are boundaries, not defects. Harness handled the IAM-gated web search and the no-tool tickets with 25 lines of config and a 20-second cold start — less work than the Runtime equivalent. The identity-bound, hook-dependent refund path is what needs code.

That is the graduation the two-model system is designed for: start on Harness, cross to Runtime when a trigger fires, and pay the code cost only for the parts that require it.

Results summary

Metric Harness Runtime
Config lines to working agent ~25 lines 398 lines
Time from create to first invoke 20.4 s ~180 s (container build)
Password reset (no tool) pass pass
EU region web search (IAM gateway) pass pass
Per-user Cedar refund (JWT gateway) blocked — outbound identity pass (JWT forwarded)
Memory wiring one config block ~40 lines of code
Observability automatic ADOT + instrumentation

The pattern is consistent: for governed tools that authorize on the agent's own identity (IAM) or on nothing, Harness is less work and faster. For tools that authorize on the user's identity, or that need deterministic post-processing of the tool result, Runtime is not optional.

So what

Start on Harness. For governed tools that authorize on the agent's identity or on nothing, 25 lines of config and a 20-second cold start beat the Runtime equivalent. Watch for the identity boundary. The first time a tool needs to act as the user rather than as the agent, move to Runtime. That is where the config-only model ends and code begins.

I still haven't fully worked through what happens with token passthrough if it were exposed on the Harness outboundAuth surface — whether the per-user vault keying would become irrelevant if the harness could forward the inbound token verbatim. That's the operational escape hatch nobody's tried yet.

Top comments (0)