How I clean LLM output with another LLM: Slash Claude costs 20%

This article was originally published on BuildZn.

I spent way too much time debugging funky JSON parsing errors from Claude's responses. My agent pipelines were constantly choking on inconsistent formatting, despite aggressive system prompts. Everyone talks about "perfect prompt engineering," but for real production systems, that's often a pipe dream. Figured out the hard way that throwing more prompt tokens at Claude to force perfect formatting is a losing game, especially when you need to clean LLM output with another LLM.

Why Your Claude Output Is Still a Mess (and costing you tokens)

Look, Claude 3.5 Sonnet, heck, even Opus, are incredible for complex reasoning. They shine at understanding nuanced requests, summarizing huge texts, or generating creative content. But ask them for a rigid JSON structure, especially after a long, open-ended thought process, and you often get... close. Or sometimes, just completely off.

In FarahGPT, my AI gold trading system, agents need perfectly structured data to execute trades or update their internal models. When Claude 3.5 Sonnet (version claude-3-5-sonnet-20240620) gives you back a JSON string that's missing a comma, or wraps a boolean in quotes, or adds a chatty intro/outro despite tool_use mode, it breaks the entire downstream flow. My JSON.parse calls kept throwing SyntaxError: Unexpected token 'O' at JSON.parse (<anonymous>) because Claude decided to start with "Okay, here's the data..."

This isn't just annoying; it's expensive. Every time an agent fails, you're either re-prompting Claude (more tokens), running complex regex (fragile, CPU cycles), or worse, just giving up on structured output entirely. I saw scenarios where a simple re-parsing prompt to Claude would add 10-15% to the token count of a workflow, just to fix its own formatting. That’s why I needed a dedicated Claude output cleanup step.

My Multi-LLM Orchestration Blueprint for Clean Output

Here's the thing — we use large, expensive models like Claude for their reasoning. We shouldn't be wasting their precious context windows and compute cycles on strict formatting. That's a job for a smaller, cheaper, specialized model. This is where multi-LLM orchestration comes in.

My approach:

1. Claude (or OpenAI GPT-4o, etc.): Handles the heavy lifting — complex analysis, decision-making, content generation. It produces raw output, aiming for structure but not obsessing over perfection.
2. Secondary LLM (e.g., Mistral 7B via Ollama, or a cheaper cloud model like GPT-3.5 Turbo): This model's only job is to take Claude's raw output and strictly parse, validate, and reformat it into a guaranteed clean, usable structure (JSON, YAML, markdown tables, whatever).

This isn't just about robustness; it's about cost efficiency. I've measured a concrete 20% reduction in downstream token costs across specific agent workflows by implementing this.

How I Measured the 20% Token Reduction

Let's say Claude's primary task output is meant to be a JSON object like { "action": "buy", "asset": "gold", "volume": 10 }.

• Scenario A (No cleanup): Claude spits out Okay, based on the market, I suggest this: { "action": "buy", "asset": "gold", "volume": "10" } (note "10" as string). To correctly use this, a downstream agent might need another Claude call to "Extract JSON from this text, ensuring 'volume' is an integer." This re-prompt costs X input tokens + Y output tokens for the cleanup.
• Scenario B (With secondary LLM cleanup): Claude gives the same raw, slightly imperfect output. My secondary LLM (Mistral via Ollama) processes it. The prompt to Mistral is tiny, fixed, and highly optimized for parsing. Mistral returns {"action":"buy","asset":"gold","volume":10}. The cost is Z input tokens (Mistral prompt) + W output tokens (Mistral's clean JSON).

My methodology:

1. For 100 typical agent outputs from Claude, I measured the average tokens required for in-context re-parsing by another Claude call to achieve strict JSON. This involved crafting a "cleanup" prompt for Claude. Average: ~150-200 tokens (input + output) per cleanup pass.
2. Then, I measured the average tokens for the same cleanup task using Mistral 7B via Ollama. This involved a fixed, tiny prompt (~50 input tokens) and ~20-30 output tokens for the structured JSON.
3. The calculation revealed that the Mistral-based cleanup was, on average, 70-80% cheaper in token count than a Claude-based re-prompt for parsing. When factoring in the total token budget for the entire workflow, this translated directly to a 20% overall reduction in downstream token waste, specifically from avoiding re-parsing prompts to the expensive primary LLM. This was measured over 100 iterations of a specific FarahGPT trade recommendation pipeline, averaging token usage with a custom Anthropic API wrapper and Ollama's count_tokens equivalent.

Honestly, relying on Claude for perfect JSON every single time is like using a sledgehammer to crack a nut, and then complaining when the nut isn't perfectly cracked.

Implementing the LLM Post-Processing Pipeline in Node.js

Let's get into the code. This is a Node.js setup, because that's what I use for NexusOS and most of my backend services.

First, you'll need @anthropic-ai/sdk for Claude and either ollama or another API client for your secondary model. For Ollama, make sure it's running locally with mistral pulled: ollama run mistral.

# Initialize Node.js project
mkdir llm-cleanup-pipeline && cd llm-cleanup-pipeline
npm init -y

# Install dependencies
npm install @anthropic-ai/sdk ollama dotenv

Next, create a .env file for your API keys:

ANTHROPIC_API_KEY=sk-ant-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx

Now, let's set up the core logic.

Step 1: The Primary LLM (Claude) for Reasoning

This example uses Claude to analyze a simple market trend and suggest an action. Notice how the prompt asks for JSON but doesn't strictly enforce it with a schema because, frankly, Claude sometimes ignores it anyway. This helps reduce LLM token waste on overly verbose schema descriptions.

// src/primaryLLM.js
require('dotenv').config();
const Anthropic = require('@anthropic-ai/sdk');

const anthropic = new Anthropic({
  apiKey: process.env.ANTHROPIC_API_KEY,
});

async function getClaudeAnalysis(marketData) {
  const prompt = `You are a financial analyst for FarahGPT. Analyze the following market trend data and recommend a trading action (BUY, SELL, HOLD) for gold. Provide a brief rationale and the recommended action as JSON.

Market Data:
${marketData}

Expected JSON format:
{
  "recommendation": "BUY" | "SELL" | "HOLD",
  "rationale": "string",
  "target_price": number | null
}
`;

  console.log("--- Calling Claude for initial analysis ---");
  const response = await anthropic.messages.create({
    model: "claude-3-5-sonnet-20240620", // The specific version I mentioned
    max_tokens: 500,
    temperature: 0.7,
    messages: [
      { role: "user", content: prompt }
    ]
  });

  const rawOutput = response.content[0].text;
  console.log("\n--- Claude's Raw Output ---");
  console.log(rawOutput);
  return rawOutput;
}

module.exports = { getClaudeAnalysis };

A typical raw output from Claude might look like this:

Okay, based on the market data, here is my recommendation:

json
{
"recommendation": "BUY",
"rationale": "Gold prices are showing strong upward momentum after breaking resistance at $2350. Technical indicators confirm bullish sentiment.",
"target_price": "$2400"
}

javascript

Notice the "Okay, based on..." intro and target_price being a string instead of a number. This is exactly what we need to fix.

Step 2: The Secondary LLM (Ollama Mistral) for Cleanup

Now, we feed that raw output into a smaller, local LLM. Mistral 7B via Ollama is fantastic for this. Its prompt is designed only to extract and reformat, nothing else. This is where the AI agent output quality gets its guaranteed structure.

// src/secondaryLLM.js
const { Ollama } = require('ollama');

const ollama = new Ollama({ host: 'http://localhost:11434' }); // Ensure Ollama is running

async function cleanClaudeOutput(rawClaudeOutput, expectedSchema) {
  const prompt = `You are a strict JSON parser. Your sole task is to extract valid JSON from the provided text and strictly adhere to the given JSON schema. If a field's type does not match, attempt to convert it or set to null if impossible. DO NOT add any conversational text. Return ONLY the valid JSON.

Text to parse:
\`\`\`
${rawClaudeOutput}
\`\`\`

Expected JSON Schema:
\`\`\`json
${JSON.stringify(expectedSchema, null, 2)}
\`\`\`
`;

  console.log("\n--- Calling Secondary LLM for cleanup ---");
  const response = await ollama.chat({
    model: 'mistral', // Using Mistral 7B
    messages: [{ role: 'user', content: prompt }],
    options: {
        temperature: 0.01, // Keep it deterministic
        num_ctx: 2048 // Adjust context window as needed, Mistral 7B is efficient
    }
  });

  const cleanedOutput = response.message.content;
  console.log("\n--- Secondary LLM's Cleaned Output ---");
  console.log(cleanedOutput);
  return cleanedOutput;
}

module.exports = { cleanClaudeOutput };

The expectedSchema is crucial here. It gives the smaller LLM a concrete target.

Step 3: Orchestrating the Pipeline

Finally, an index.js to run the whole LLM post-processing pipeline:

// index.js
const { getClaudeAnalysis } = require('./src/primaryLLM');
const { cleanClaudeOutput } = require('./src/secondaryLLM');

async function runPipeline() {
  const marketData = `
    Latest gold prices: $2360/oz.
    Previous day close: $2355/oz.
    5-day moving average: $2340/oz.
    Volume: Increased significantly by 15% today.
    Economic news: Inflation data came in slightly lower than expected, leading to a weaker dollar.
  `;

  const expectedSchema = {
    type: "object",
    properties: {
      recommendation: { type: "string", enum: ["BUY", "SELL", "HOLD"] },
      rationale: { type: "string" },
      target_price: { type: ["number", "null"] }
    },
    required: ["recommendation", "rationale"]
  };

  try {
    const rawClaudeOutput = await getClaudeAnalysis(marketData);

    // Attempt to parse directly (will likely fail or need cleanup)
    try {
        const directParse = JSON.parse(rawClaudeOutput.match(/```
{% endraw %}
json\n([\s\S]*?)\n
{% raw %}
```/)[1]);
        console.log("\n--- Direct Parse (might fail or be imperfect) ---");
        console.log(directParse);
    } catch (e) {
        console.error("\n--- Direct JSON parse failed from Claude's raw output ---");
        console.error(e.message); // Will likely show the 'Unexpected token' or similar
    }

    const cleanedJSONString = await cleanClaudeOutput(rawClaudeOutput, expectedSchema);
    const finalStructuredData = JSON.parse(cleanedJSONString);

    console.log("\n--- Final Structured Data for AI Agent ---");
    console.log(finalStructuredData);
    console.log(`Recommendation: ${finalStructuredData.recommendation}`);
    console.log(`Rationale: ${finalStructuredData.rationale}`);
    console.log(`Target Price: ${finalStructuredData.target_price}`);

    // This is where you'd pass `finalStructuredData` to your next agent.
    // It's guaranteed to be clean, so no more parsing headaches.

  } catch (error) {
    console.error("Pipeline failed:", error);
  }
}

runPipeline();

To run this: node index.js

This pipeline consistently gives me:

{
  "recommendation": "BUY",
  "rationale": "Gold prices are showing strong upward momentum after breaking resistance at $2350. Technical indicators confirm bullish sentiment.",
  "target_price": 2400
}

Notice target_price is now a number, as per the schema, and all conversational fluff is gone. This is exactly the kind of structured reliability needed for high-stakes systems like FarahGPT.

What I Got Wrong First

Initially, I tried to make Claude perfectly conform by stacking more instructions into its system prompt. "ONLY return JSON. DO NOT include any preamble or postamble. Strictly adhere to this JSON schema..." You get the idea. I even tried to build my own JSON schema validation into the prompt itself. It worked... sometimes. But often, if the underlying reasoning task was complex, Claude would prioritize the reasoning and let the formatting slip. It's like asking a genius to also be a meticulous typist; they can do it, but it's not their primary strength, and it takes mental overhead.

My biggest mistake was thinking a single, powerful LLM could be a jack of all trades. This approach often led to:

• Higher token counts: Overly verbose system prompts to enforce strict formatting eat into your context window and cost.
• Increased latency: Asking a general-purpose LLM to perform a simple parsing task adds unnecessary compute time.
• Fragile pipelines: Even with tool_use mode, Claude 3.5 Sonnet (and even Opus sometimes) can drift, especially with complex outputs or edge cases. SyntaxError: Unexpected token 'O' at JSON.parse (<anonymous>) was a common sight, indicating Claude started its response with "Okay..." instead of the JSON.

I also spent a week trying to build complex regex patterns to clean up Claude's output. That was a nightmare. Regex is brittle, hard to maintain, and completely fell apart when Claude decided to change its preamble style. Using a secondary LLM for this is miles ahead for flexibility and robustness.

Optimization & Gotchas

1. Model Choice for Secondary LLM: For local setups, Ollama with Mistral, Llama, or Qwen is excellent. For cloud, GPT-3.5 Turbo is a strong contender due to its low cost and high reliability for formatting tasks. The key is a model that's cheap and good at following instructions deterministically.
2. Prompt Engineering for Cleanup: Keep the secondary LLM's prompt extremely concise and direct. Its job is parsing, not reasoning. Give it the schema explicitly. Use a very low temperature (e.g., 0.01) to reduce creativity and increase determinism.
3. Latency: Running a local Ollama model introduces minimal latency. If using a cloud model for cleanup, ensure it's fast (like gpt-3.5-turbo-0125 for speed). The overall latency impact should be negligible compared to the primary LLM's processing time.
4. Error Handling: Always wrap JSON.parse in a try...catch block. Even a secondary LLM can occasionally mess up, though it's far less common when prompted correctly. Consider a fallback: if the secondary LLM fails to produce valid JSON, log it and possibly revert to a more robust parser or a human review step.

This setup significantly improved the AI agent output quality for NexusOS and my YouTube automation pipeline. It's a fundamental pattern for building reliable multi-agent systems.

FAQs

What if my primary LLM is already good at JSON?

Even if your primary LLM is "good," a secondary LLM adds a layer of guarantee. It acts as a dedicated schema validator and formatter. For critical applications, this separation of concerns is invaluable, ensuring downstream agents always receive clean, structured data, regardless of the primary LLM's occasional quirks.

Does this add significant latency to my pipeline?

For a local secondary LLM like Mistral via Ollama, the added latency is minimal, often in the tens to low hundreds of milliseconds, which is negligible compared to the seconds taken by a large model like Claude for its primary task. For cloud-based cleanup models, choose fast ones like GPT-3.5 Turbo to keep the overhead low.

Can I use this for non-JSON outputs, like YAML or Markdown tables?

Absolutely. The principle remains the same. The secondary LLM's prompt would simply instruct it to format the raw output into YAML, a Markdown table, or any other structured text format, based on a provided schema or example. It's about offloading strict formatting to a dedicated, cheaper model.

---

Look, the LLM hype often focuses on raw intelligence. But in production, reliability and cost matter more than anything. Trying to force a complex, reasoning LLM to be a perfect formatter is just bad engineering. Use the right tool for the job. Claude for the brains, a smaller LLM for the strict data hygiene. This clean LLM output with another LLM pattern isn't just a hack; it's a fundamental architectural decision that drastically improves AI agent output quality and keeps your token costs in check. If you're building serious LLM applications, this should be your default.