How DSH Builds an RLHF Data Flywheel for AI Agents

Abstract

Modern Agent Harness systems are widely discussed for task execution, sandbox isolation and tool calling capabilities. Yet one under‑explored value of the Harness runtime lies in its capacity for offline reinforcement‑learning from human feedback (RLHF). DSH (DeepSeek‑Agent‑Harness) implements a data flywheel mechanism: human‑reviewed guardrail approval/rejection events inside runtime get structured into preference datasets, which can feed post‑training and improve agent behaviour iteratively.

This paper differentiates itself from conventional agent framework articles. Instead of focusing purely on how agents run tasks, it analyses how runtime design shapes the quality of training data collected from live agent workflows. For multi‑model production deployments that connect diverse model backends to Harness infrastructure, an API gateway simplifies endpoint routing and credential orchestration. 4sapi offers unified access control for heterogeneous LLM workloads.

This article first clarifies three conceptual layers: pre‑training, post‑training and live production runtime. It then breaks down DSH’s three core design pillars: event‑driven state logging, event‑derived state, and pluggable capability system. It explains how guardrails act not only as safety interceptors but also as a training environment to generate high‑quality preference signals. Real‑world POC experimental results are cited to validate end‑to‑end flywheel functionality. Finally, it outlines three major operational transformations brought by this architecture for enterprise agent development.

1. Three Layers of Complete Agent Workflow: Pre‑training, Post‑training, Live Runtime

To understand DSH’s data flywheel, we need to separate three distinct operational layers for enterprise‑grade agents.

Layer 1: Pre‑training. This stage builds general‑purpose capabilities. Base models acquire general reasoning, coding and language competence from large‑scale public corpora. Pre‑training delivers universal skills independent of enterprise‑specific requirements.

Layer 2: Post‑training (SFT, RLHF). Supervised fine‑tuning and reinforcement learning from human feedback adapt general‑purpose base models toward target behaviours. Post‑training has three key practical characteristics for industrial implementation.

1. Offline pipeline: Training and inference run in separate pipelines. Model weight snapshots are static for online inference. All end‑user requests hit identical model weights. New preference data must go through offline data curation, training jobs, model evaluation and version release cycles before improvements reach production. This workflow creates latency measured in weeks.
2. Version‑based behaviour shift. Model behaviour changes occur through version releases. There is no gradual real‑time update. End‑users observe distinct behavioural differences once they upgrade to a new model snapshot.
3. Immutable weight artefacts. After release, model weights cannot be modified in‑place for individual sessions. A new training run is required to adjust any behavioural tendency.

Post‑training relies on curated static datasets. It cannot fix every edge case instantly. This is exactly why a third runtime layer becomes indispensable.

Layer 3: Live production runtime. The Harness runtime sits between user inputs and model inference. It governs tool access, approval workflows, sandbox constraints and session state. Even with identical base‑model weights, runtime configuration can swing practical agent performance from total failure to acceptable quality.

DSH’s core innovation builds on the interaction between layer‑2 post‑training and layer‑3 runtime. Its core thesis can be summarised:

The Harness runtime serves as a safe experimental playground for post‑training. New behaviours can be tried in runtime with reversible guardrail constraints. Human reviewer approval or rejection events get logged as structured preference signals, feeding back into post‑training datasets.

This design creates a closed‑loop pipeline: risky candidate behaviours are first tested inside runtime guardrails; human reviewers filter outcomes; structured event logs get converted into preference samples for fine‑tuning; improved model weights are shipped in subsequent releases.

2. Three Core Design Pillars of DSH

All flywheel capabilities are built upon three foundational architectural pillars: event‑log‑driven state, state derived purely from events, and pluggable capability modules.

2.1 Pillar One: State derived strictly from event logs

Conventional application logging captures descriptive traces for debugging. DSH’s event log works differently. It is the single source‑of‑truth for system state. The full current state of every agent session can be reconstructed completely by replaying the ordered sequence of logged events. No separate mutable database table stores snapshot state independently.

Key properties enforced for event logs:

• Append‑only writes. Events can only be appended. Historical log entries cannot be modified or deleted retroactively. Any state change must create a brand‑new event record.
• Strict sequence ordering. Event ordering defines identity. Sequence position determines causality.
• Re‑playable reconstruction. The complete session state can be rebuilt by replaying logs from start to finish. There exists no separate “state table” that can diverge from event history. This eliminates the difficult “state‑log inconsistency” bug class common in agent systems.

Every meaningful action, tool invocation, guardrail review decision, approval or rejection becomes an immutable structured event. Session state is a computed artefact reconstructed from logs, never manually edited.

2.2 Pillar Two: All state is derived from event records

Since event logs are the sole source of truth, no secondary copies of state get persisted. UI renderings, snapshot displays and derived metadata are computed on demand by replaying historical events.

This architectural principle delivers critical practical benefits:

• Single source‑of‑truth removes synchronisation overhead. Developers never maintain duplicate copies of session data.
• Reproducibility is guaranteed. Identical event sequences always reconstruct identical agent states.
• All human decisions such as approvals and rejections are permanently embedded within event sequences. They can be extracted later for preference dataset assembly without extra manual labelling work.

2.3 Pillar Three: Plug‑in‑oriented capability architecture

DSH implements a fully plugin‑based component model. Core loop logic is isolated inside one minimal package. All other capabilities including sandboxes, permission rules, reviewer workflows, tool definitions are implemented as interchangeable plugins following the unified cordis plugin contract.

Plugins can be dynamically loaded or unloaded at runtime:

• Activating a sandbox plugin enables isolated code execution capability.
• Unloading a plugin immediately revokes corresponding agent capabilities. Permission fields for tool access update dynamically alongside plugin lifecycle.

Plugins can even be installed dynamically by the agent itself during runtime. Nevertheless, DSH enforces a critical safety rule: any capability change that persists beyond the current session must pass human reviewer approval. This separates ephemeral in‑session experimentation from durable capability updates.

3. Guardrails: Not Just Safety Barriers, but a Training Environment

Most agent frameworks treat guardrails as blocking interceptors. When unsafe actions appear, guardrails terminate execution. DSH re‑imagines guardrails as a dual‑purpose component: safety enforcement plus preference‑signal generation.

DSH guardrail implementation carries two distinctive structural features.

First, guardrails and human reviewer approval pipelines run within the same execution pipeline with a default closed‑fail policy. Once a tool call is submitted by the model, it flows sequentially through rule‑based checks, then human reviewer queues. Final outcomes are logged as immutable events: allowed‑once, rejected, cancelled. Even unattended headless deployments apply implicit rejection as default behaviour. Review decisions are logged before action execution, not appended after‑the‑fact. This avoids data leakage where post‑hoc labelling misrepresents what actually happened inside runtime.

Second, every guardrail trigger generates structured events decoupled from model conversation history. Approval and rejection records exist as independent event entries. They are not embedded inside chat message payloads.

From a RLHF data perspective, this design delivers immense practical value:

• Each human review event forms ready‑made preference samples. Developers obtain high‑quality chosen/rejected pairs without separate manual annotation projects. Real failure cases naturally observed in production become fine‑tuning material.
• Events separate conversation context from reviewer judgement. You can slice and filter datasets by reviewer outcomes, tool categories or risk levels for targeted fine‑tuning.
• Real‑world operational failure modes get captured automatically. Rather than manually constructing synthetic failure examples, you harvest authentic runtime events.

The boundary sandbox further complements guardrail workflows. It provides ephemeral, reversible runtime environments. Agents can test new tooling and behaviours inside these sandboxes. If human reviewers validate the behaviour, it can graduate into persistent capabilities for future sessions. If rejected, the experimental sandbox instance gets discarded, with zero lasting side‑effects.

4. Self‑Evolution Workflow: How Agent Capabilities Grow

DSH divides capability evolution into three distinct phases, governed by human review gates.

1. Experiment phase: Capability is loaded only for the current active session. Changes vanish once the session terminates. No human approval required for purely local trial‑and‑error.
2. Solidification phase: To preserve a capability beyond one session, the corresponding plugin or skill definition must pass human reviewer approval. After approval, the capability definition is persisted.
3. Dissemination phase: Once validated skills are merged, they can be distributed into model post‑training datasets. Corresponding behaviour patterns get learned by the model during fine‑tuning. Gradually the model starts to correctly invoke these capabilities without runtime plugin installation steps.

This three‑phase workflow enforces an important invariant: risky capability modifications cannot become globally effective without human audit. Agents are permitted to invent and test tools within their sandbox session, but cannot roll out changes broadly without reviewer sign‑off.

5. POC Experimental Validation & Empirical Evidence

Local POC tests ran on DSH version 0.1.0‑rc5, using Qwen‑3‑max and related model variants. Three groups of controlled experiments validate the complete RLHF data flywheel.

5.1 Four criteria for verifying genuine capability evolution

Researchers established four objective criteria to confirm that agents truly acquired new capabilities, rather than producing superficial imitation outputs:

1. Capability existence: Agent can invoke the newly‑available tool.
2. Correct invocation: Tool calling parameters match specification.
3. Robustness: The capability works under held‑out test scenarios.
4. Recall & forgetting tracking: The agent can use the new capability when appropriate and fall back to prior behaviours when the tool is removed.

Only when all four criteria pass can we confirm that capability acquisition has taken place. POC metrics demonstrated measurable improvement in agent success rates as the flywheel cycles ran.

5.2 Three groups of controlled comparison experiments

Three sets of contrast experiments reveal critical constraints of the DSH flywheel mechanism.

1. Interface definition ablation test: When tool interface definitions were incomplete or ambiguous, agents failed to adopt new skills reliably, even given repeated trial opportunities. Clear, rigorous interface specifications proved mandatory for capability transfer.
2. State representation ablation test: Agent evolution stalled when runtime state could not be faithfully expressed within message‑history context. Structured event logging and state reconstruction are required for reliable self‑improvement. Pure natural‑language contract descriptions cannot substitute structured runtime state tracking.
3. Feedback signal ablation test: Improvement did not automatically happen simply given large volumes of feedback. Progress depended on whether the feedback signal carried usable structural information for learning. Unstructured loss signals without structured event metadata failed to drive agent capability advancement.

These experimental results deliver a clear conclusion: self‑evolution is not an emergent miracle. It depends on well‑defined interfaces, faithful state representation and properly structured preference signals harvested from runtime events.

5.3 Interpreting DSH flywheel from reinforcement‑learning perspective

Viewed through RL frameworks, DSH builds a complete reinforcement‑learning infrastructure within runtime:

• Action space: Tool calls, plugin loading and capability modification operations.
• Observation space: Reconstructed session state built by replaying event logs.
• Reward signal: Implicit human preference encoded in guardrail approval or rejection events.
• Trajectory storage: Immutable event logs preserve full interaction trajectories, ready to be converted into RLHF fine‑tuning datasets.

All trajectories stay persistently recorded. Data curators can filter logs to build chosen/rejected pairs for supervised fine‑tuning and preference optimisation. Real operational failure scenarios from production become training samples, closing the end‑to‑end flywheel.

6. Three Transformations for Enterprise Agent Workflows

The DSH event‑log‑driven flywheel changes how organisations build and maintain agent systems, bringing three major shifts.

6.1 Shift 1: From prompt‑driven iteration toward data‑flywheel‑driven iteration

Traditional agent tuning cycles centre on prompt engineering. Developers iterate over system prompts manually to adjust agent behaviour. In DSH‑style runtime‑first architecture, human decisions made during live operation get logged automatically. These structured events become training‑set materials. Behaviour improvement flows from real runtime incidents rather than manual prompt tweaking alone. Agent behaviour evolves through real production events, not developer guesswork about hypothetical inputs.

6.2 Shift 2: From manual snapshot export to event‑native dataset assets

In older workflows, engineers manually extract conversation snapshots to assemble fine‑tuning datasets. In DSH, every human judgement inside runtime is already persisted as structured event logs. Dataset construction becomes filtering and transformation over event streams. Enterprises accumulate reusable preference data assets as agents run business tasks. Every human review decision simultaneously serves operational safety and model improvement.

6.3 Shift 3: From static configuration maintenance toward event‑derived system consistency

Conventional agent platforms require developers to manually synchronise configuration across model prompts, permission settings and training datasets. Misalignment frequently occurs between runtime permission and what the model expects to do. DSH derives state entirely from event logs. Configuration changes manifest as logged events. Training datasets are built from those same event sequences. Runtime enforcement and training data originate from identical event sources, drastically reducing consistency bugs.

7. Discussion & Conclusion

DSH demonstrates a practical path for connecting runtime agent Harness infrastructure with post‑training RLHF pipelines. Guardrails are repurposed beyond pure safety enforcement. Every human approval and rejection in production runtime turns into high‑quality preference signals. Immutable event logs are the foundation of this architecture: session state is reconstructed purely from event replay, eliminating state‑log inconsistency bugs and enabling automatic dataset harvesting.

POC controlled experiments highlight critical prerequisites for agent self‑evolution: precise interface definitions, faithful state representation and structured feedback signals. Self‑improvement does not emerge spontaneously; it must be enabled by deliberate system design. Enterprises adopting this pattern experience three workflow transformations: moving away from prompt‑centric tuning, building event‑native training‑data assets, and achieving natural consistency between runtime and training through shared event sources.

Agent capability iteration no longer depends purely on offline labelling cycles. Runtime itself becomes part of your ML pipeline. When running multi‑model agent stacks across multiple LLM vendors, 4sapi simplifies unified endpoint management for mixed model deployments.

International access: https://4sapi.com
Domestic access: https://4sapi.cn