DEV Community: Kristofer Jussmann

Fine-Tuning Is Often the Wrong First Move: Introducing Harneloop

Kristofer Jussmann — Thu, 23 Jul 2026 16:45:04 +0000

When an AI agent repeatedly fails at a specific task, the default reaction is often to use a better model or fine-tune the current one.

That can be the wrong first move.

The model is only one part of an agent. Its harness controls the context it receives, tools it can use, environment it can inspect, feedback it gets, and whether one failure becomes a durable improvement.

I built Harneloop to make that improvement process structured and verifiable.

The loop

Harneloop asks the operating agent to:

attempt a real task;
inspect the resulting artifacts, state, logs, and traces;
trace likely failures back through the recorded run;
create an isolated candidate change to the harness;
retest the candidate and relevant regressions;
promote the change only when evidence supports an improvement.

The candidate can change more than a system prompt. It can add tools, retrieval, examples, validators, environment automation, skills, or any other task-specific infrastructure.

The working harness remains unchanged while the candidate is tested. Failed experiments do not silently become production state.

Why artifacts matter

A command can succeed while the result is wrong.

An interface can compile but fail visually. A scene can render but violate the requested spatial relationships. A document can be valid but communicate the wrong thing. A browser workflow can finish in the wrong state.

For these tasks, the artifact is part of the evidence.

Harneloop links artifacts and observations to the exact run that produced them. That gives the agent a way to move from “this looks wrong” to a trace-backed hypothesis about what should change in the harness.

Portable harness units

Each task-specific development environment is stored as a harness unit.

A harness unit carries the target definition, operational map, reusable harness material, candidates, evidence references, regression cases, and promoted versions. The agent can add custom scripts, research, retrieval data, infrastructure, or entirely new directories when necessary.

Harneloop protects the lifecycle rather than putting the agent in a rigid template.

Successful units can be paused, restored, moved, packaged, or exported into another agent or application.

Why harness-first?

Research has repeatedly shown that inference-time systems can change agent performance without changing model weights.

SWE-agent demonstrated the impact of a purpose-built agent-computer interface. Reflexion and Self-Refine showed gains from feedback and iterative refinement without additional training. The 2026 Self-Harness paper directly demonstrated agents mining weaknesses from traces, proposing harness changes, and validating them with regression tests.

This does not mean harnesses always beat fine-tuning. Fine-tuning can still be useful for behavior, style, latency, or capabilities that cannot be supplied effectively at inference time.

The practical rule is narrower:

When the bottleneck is context, tools, retrieval, feedback, validation, memory, or environment interaction, test the harness before changing the weights.

Initial evidence

Harneloop was first used to develop ViperMesh, a Blender harness for spatial reasoning and scene generation.

The benchmark held the acting model fixed at OpenAI GPT 5.5 High and compared ViperMesh with the Anthropic x Blender MCP server baseline.

Across seven comparable live tasks, ViperMesh:

was faster on 6 of 7 tasks;
achieved a 2.534x mean speedup;
improved preliminary neutral LLM visual evaluation by 8.19 points.

Harneloop did not generate the scenes. It structured the process used to inspect failures, improve the harness, preserve evidence, and verify whether changes helped.

The full ViperMesh case study includes methodology and limitations. It is evidence from one task family, not a universal harness-versus-fine-tuning benchmark.

Try it on a real task

Harneloop is optimized for agent-led setup. Give the repository to a capable coding agent and describe what you want the harness to improve. The agent can install the framework, map the environment, create the first harness unit, and ask for missing information when needed.

I am looking for independent use cases, including unsuccessful ones. The most useful feedback concerns onboarding, environment mapping, evidence quality, runtime cost, candidate decisions, and tasks where the harness reaches its limit.

Repository: https://github.com/Ker102/Harneloop

Case study: https://www.kristoferjussmann.me/case-studies/vipermesh

Kaelux: Engineering the Future of Intelligent Infrastructure

Kristofer Jussmann — Mon, 06 Apr 2026 21:22:48 +0000

Why Custom LLM Systems Are Replacing Off-the-Shelf AI Tools

Published by Kaelux AI Engineering — a global agency building custom LLM systems, RAG pipelines, and intelligent automation for businesses.

The Problem with One-Size-Fits-All AI

Frontier models are incredible tools. But if you're trying to build a serious product or automate a critical business workflow, you've probably hit the wall:

No access to your proprietary data. Generic models don't know your contracts, your product catalog, or your internal documentation.
Unreliable outputs. Hallucinations in customer-facing applications aren't just annoying — they're a liability.
Zero control over reasoning. You can't audit why the model made a decision, and you can't constrain its behavior in production-critical ways.
Vendor lock-in. Building on top of a single provider's API means your entire product roadmap depends on someone else's pricing and deprecation schedule.

This is why teams are increasingly investing in custom LLM systems — purpose-built AI infrastructure that integrates directly with their own data, reasoning chains, and deployment requirements.

What "Custom LLM" Actually Means

Let's be precise. A custom LLM system isn't about training a model from scratch. It's an architecture that typically includes:

1. Retrieval-Augmented Generation (RAG)

Instead of relying on the model's parametric memory, you pipe real-time data from your own knowledge base into the model's context window at inference time.

At Kaelux, we've built RAG pipelines ranging from naive vector retrieval to Corrective RAG (CRAG) architectures that:

Detect when retrieved documents are irrelevant
Fall back to live web search for grounding
Re-rank results using cross-encoder models before passing them to the LLM

This matters because retrieval quality is the single biggest determinant of AI output quality in enterprise settings.

2. Multi-Model Routing: Density vs. Speed

Stop sending simple tasks to frontier models. We build routers that classify intent and dispatch queries to the most cost-effective compute:

Small Language Models (SLMs) for extraction and classification.
Frontier LLMs for deep reasoning and creative synthesis.

This cuts inference costs by 60-80% while maintaining accuracy where it matters.

3. Structured Generation & Tool Use

Production AI systems need to output valid JSON, call APIs, and interact with databases — not just generate prose. Structured generation using JSON schemas, function calling, and constrained decoding ensures the model's output is machine-readable and actionable.

4. Agentic Workflows

The most advanced systems use AI agents — autonomous processes that:

Plan multi-step workflows
Execute tool calls (database queries, API requests, file operations)
Self-evaluate and retry on failure
Orchestrate across multiple services

At Kaelux, we build these using LangGraph for complex reasoning chains and n8n for event-driven workflow automation.

When Should You Go Custom?

Go custom when:

Your AI interacts with proprietary/sensitive data (legal, medical, financial)
You need deterministic behavior and audit trails
Cost-per-query matters at scale
You're building AI as a product feature, not just an internal tool

Stay with off-the-shelf when:

By deploying on high-performance Enterprise IaaS, we achieved sub-400ms latency. The same system on a generic API would have cost 10x more and gated the user behind a 5-second "Thinking..." spinner.

The Kaelux Engineering Framework

Rather than relying on off-the-shelf boilerplates, we've engineered a unified framework for rapid, high-performance deployment:

Layer	Specialization
Delivery	Edge-Native Serverless & Hybrid-Cloud Orchestration
Orchestration	LangGraph, n8n, and Custom Event-Driven Buses
Retrieval	CRAG pipelines, Cross-Encoder Re-rankers, and ModernBERT embeddings
Intelligence	Frontier LLMs (Gemini/OpenAI), specialized SLMs (Mistral/Qwen), and proprietary fine-tuned model-weights
Infrastructure	Proxmox-managed Private Cloud, Azure ML clusters, and containerized IaaS
Monitoring	Distributed latency tracking and RAG retrieval-quality observability

Wrapping Up

The era of the "all-in-one" frontier model is shifting. We are entering the age of Agentic Orchestration — where the value isn't in the model itself, but in the systems that wrap around it.

If you're exploring this path, reach out to Kaelux or check our AI Engineering Wiki for technical deep dives on RAG, hallucination prevention, and agentic workflows.

About the author: This article is published by Kaelux (kaelux.dev), an AI engineering agency building custom LLM systems, RAG pipelines, and intelligent automation for businesses worldwide. Founded by Kristofer Jussmann.

Tags: #ai #llm #rag #machinelearning #webdev #kaelux #engineering #automation

What We Learned From Analyzing 28,000 Production AI System Prompts

Kristofer Jussmann — Tue, 24 Mar 2026 15:13:26 +0000

Over the last few months developing PromptTriage, we've collected and analyzed over 28,000 production system prompts. Most are bloated, contradictory, and actively hurt reasoning quality.

📉 Anti-Pattern 1: The "Emotional Blackmail" Scaffold (14%)

Caption: Anti-Pattern Prevalence in 28,000 Production System Prompts. (Full-width hero chart)

Over 14% still contain emotional appeals:

"Take a deep breath. If you miss a bug, the company will lose millions."

Why it fails: Modern RLHF has trained out the "anxiety" response. Emotional context distracts self-attention from the actual task.

🏗️ Anti-Pattern 2: The "Just in Case" Clause (62%)

62% of prompts over 300 words contained contradictory constraints. Our Study E data proved short prompts (<50 words, scoring 80.1/100) consistently outperform long ones (>300 words, 66.9/100).

🎭 Anti-Pattern 3: The "World Class Expert" Trap (80%)

Nearly 80% started with "Act as a world-class expert." Our Study C proved this provides zero lift on modern models (~78/100 with or without it).

🚀 The Fix: The 50-Word Rule

State the role (10 words): "Extract data from SEC filings."
State negatives (20 words): "Do not include pleasantries. Do not output markdown."
Halt.

PromptTriage compresses 500-word prompts to the optimal 50-word framework.

AI Format Wars: Does the Shape of Your Prompt Matter? (1,080 Evals Later)

Kristofer Jussmann — Sun, 22 Mar 2026 22:40:35 +0000

AI Format Wars: Does the Shape of Your Prompt Matter? (1,080 Evals Later)

We spend hours tweaking the words in our prompts, but how much thought do we give to the structure? If you ask an AI to return data in JSON vs. Markdown, or if you write a concise 50-word prompt vs. a detailed 500-word prompt, does the quality of the reasoning actually change?

To find out, I ran Study E v2: The Format Wars.

I subjected 5 frontier models to 1,080 rigorous evaluations across 12 distinct task domains (coding, math, data extraction, analysis, creative writing, and more). Every single evaluation was scored blindly by a 3-judge LLM jury on a 100-point scale.

The results completely changed how I build AI applications.

🔬 The Setup: 1,080 Evaluations

We tested five heavyweight models:

GPT-5.4 (OpenAI)
Nemotron 3 Super 120B (Nvidia)
Claude Sonnet 4.6 (Anthropic)
Gemini 3.1 Pro (Google)
Qwen 3.5 397B (Alibaba)

For each model, we ran 216 evaluations testing 18 unique prompt configurations:

6 Formats: Plain Text, Markdown, XML, JSON, YAML, Hybrid (Text + Code Blocks)
3 Lengths: Short (<50 words), Medium (~150 words), Long (>300 words)

The scoring was handled by a ruthless 3-judge panel (Llama 4 Maverick, Claude Opus 4.6, and Atla Selene Mini) grading on instruction following, reasoning quality, formatting adherence, and edge-case handling.

🏆 Finding 1: The Model Rankings

Before looking at formats, how did the models perform overall across all 18 permutations?

🥇 GPT-5.4: 88.1 / 100 — Won 10 out of 12 task domains
🥈 Nemotron 120B: 85.1 / 100 — Won 1 domain (Data Extraction), extremely close to GPT-5.4
🥉 Claude Sonnet 4.6: 69.5 / 100
Gemini 3.1 Pro: 62.6 / 100 — Won 1 domain (Question Answering)
Qwen 397B: 61.0 / 100

Takeaway: GPT-5.4 is the undeniable reasoning king right now. But Nvidia's Nemotron 120B is a shocking powerhouse—it scored incredibly close and actually beat GPT-5.4 outright in Data Extraction tasks. If you aren't testing Nemotron in your pipelines, you are missing out.

🧱 Finding 2: The Best Format is... JSON?

If you want the highest quality reasoning and instruction following from an LLM, what format should you ask it to return?

YAML: 74.6 / 100
JSON: 74.4 / 100 (Statistical tie with YAML)
Hybrid: 73.5 / 100
XML: 73.3 / 100
Markdown: 72.9 / 100
Plain Text: 70.8 / 100

Takeaway: Asking the model to structure its output in JSON or YAML doesn't just make it easier for your code to parse—it actually improves the model's reasoning.

Why? Forcing the model into a strict structural schema (like JSON keys) acts as a cognitive scaffold. It forces the model to categorize its thoughts before generating output, leading to fewer hallucinations and better instruction adherence. Plain unstructured text performed the worst across the board.

But here's the nuance: different models prefer different formats:

Note: While JSON was the best overall, Nemotron and Qwen actually performed slightly better when outputting YAML.

📏 Finding 3: The Prompt Length Paradox

We've been trained to write massive, highly detailed "megaprompts" with endless context. But the data reveals a startling paradox:

Short Prompts (<50 words): 80.1 / 100
Medium Prompts (~150 words): 72.8 / 100
Long Prompts (>300 words): 66.9 / 100

Takeaway: Across all 5 models and all 6 formats, short prompts absolutely demolished long prompts.

When you flood the context window with too many instructions, constraints, and examples, the model suffers from attention dilution. It forgets the primary objective and gets bogged down trying to satisfy secondary constraints.

The worst combination in the entire study? Qwen 397B given a Long prompt asking for Plain Text (38.8/100).

🏅 Finding 4: The Best and Worst Combinations

What are the absolute best and worst model + format + length trios?

The Golden Combo scored 92.2 / 100: GPT-5.4 + Hybrid Output + Short Prompt.

🚀 The Ultimate Prompting Formula

If you want to maximize the performance of a modern LLM, the data points to a clear formula:

Keep it brief: State your objective clearly in under 50 words. Drop the fluff.
Demand structure: Always ask the model to return its answer in JSON or YAML. Avoid asking for unstructured text.
Use the right model: GPT-5.4 for general reasoning/coding, Nemotron 120B for extraction.

I built PromptTriage specifically to help developers automatically refactor those bloated 500-word metaprompts down into the high-scoring "Short + Structured" format this data proves works best.

Data lovers: The full 1,080-row dataset and analysis script are open-sourced in the PromptTriage repo.