Why your local agent keeps dropping brackets (and how Hermes fixes it)

Gabe Dev — Mon, 18 May 2026 05:10:53 +0000

If you’ve tried running a local agent loop using standard frameworks, you know exactly when it breaks: loop 3 or 4, right when the model needs to call a tool.

Most frameworks force you to dump massive JSON schemas of your entire tool repository directly into the system prompt. While this works fine when you're burning OpenAI credits on a massive cloud model, the moment you drop down to local hardware (like a quantized 8B or 32B model running on a consumer GPU), two things happen: your context window gets eaten alive by structural boilerplate, and the model eventually drops a closing brace } mid-generation, completely crashing your regex parser.

While hacking on Hermes Agent for the DEV challenge, I realized its biggest architectural win isn't the slick integration list—it's how it completely bypasses this JSON parsing tax.

**The JSON Schema Tax on Local VRAM
**When an agent framework uses passive prompt coercion, it passes your Python functions through a serializer to generate something like this in your system prompt:

JSON

{
  "name": "query_db",
  "description": "Lookup user records",
  "parameters": { "type": "object", "properties": { "user_id": { "type": "integer" } } }
}

Multiply that by five or ten tools, and you’re wasting thousands of tokens just explaining how to format a response. On local hardware, this context bloat dilutes the model’s actual reasoning attention and spikes your Time-to-First-Token (TTFT) latency.

**How Hermes Shifts to Native Token Steering
**Hermes doesn't try to bully a raw text model into outputting valid JSON via heavy system prompting. Instead, it leverages the fact that the underlying Nous Hermes models are natively fine-tuned to treat tool execution as a structural token sequence using hardcoded XML tags.

Instead of parsing a massive prompt blocks, the framework expects and guides the model into a deterministic streaming state:

XML

<scratchpad>
Dependencies resolved. Need to check user status before updating the record.
</scratchpad>
<tool_call>
{"name": "query_db", "arguments": {"user_id": 4022}}
</tool_call>

The engineering win here is subtle but massive: State Isolation.

The is a sandbox: The model handles its internal reasoning tokens before it hits the tool execution tokens. This stops the "thinking" process from bleeding into the actual execution syntax.

Zero Prompt Bloat: Because the model's weight distribution naturally favors these tags for tool routing, you don't need a 400-token system prompt lecturing the model on bracket placement.

**The Bottom Line
**When you move tool routing from the prompt layer down to the token-generation layer, you save significant VRAM and eliminate the syntax degradation that plagues small models. It’s the reason you can run highly reliable, multi-step tool pipelines on a local 8B model inside Hermes that would normally require a massive, unquantized cloud API just to keep the JSON valid.

If you’re building an entry for the challenge, skip the massive prompt-engineering wrappers. Lean into the native XML schema, keep your system prompts minimalist, and let the model’s structural tuning do the heavy lifting.

DEV Community: Gabe Dev

Why your local agent keeps dropping brackets (and how Hermes fixes it)