DEV Community

Achyut Srivastava
Achyut Srivastava

Posted on Edited on

# Training a 57M Edge Model: SFT Tool-Calling Done, Now Branching into 3 RLVS Loops

TL;DR: We just completed Supervised Fine-Tuning (SFT) on Nova, our home-grown 57M parameter edge model, hitting 100% deterministic tool-calling precision. Instead of continuing single-track training, we are cloning the base weights into three distinct forks, each trained under a specialized Reinforcement Learning with Verifiable Supervision (RLVS) verifier: Cybersecurity, ARC-AGI, and Balanced Desktop Execution.


🎯 Milestone 1: SFT & 100% Deterministic Tool Calling

When training a sub-100M parameter model (Nova is exactly 57.2M parameters), the biggest danger is format drift during structured JSON tool calls. Traditional LLMs often hallucinate brackets or invent parameter keys when parameters drop below 1B.

By curating a clean, syntactically-verified tool-calling dataset with Program-of-Thought (PoT) execution traces:

  • Tool-Call Accuracy: 100% strict JSON schema compliance.
  • Latency: Sub-50ms token generation on consumer hardware.
  • Execution: Zero hallucinations on Win32 API calls, CLI terminal commands, and file diff generation.

With the base grammar and tool execution locked in, we reached a crossroads: How do you teach a 57M model high-level reasoning without bloating its parameter count?


🧬 The 3-Way RLVS Forking Strategy

Instead of forcing a single 57M model to master everything and diluting its weights, we are taking the identical base SFT checkpoint and cloning it into three parallel training tracks:

                      β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
                      β”‚   Nova 57M Base SFT Checkpoint   β”‚
                      β”‚    (100% Deterministic Tools)    β”‚
                      β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”¬β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
                                        β”‚ Clone Weights
            β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”Όβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
            β–Ό                           β–Ό                           β–Ό
β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”  β”Œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”
β”‚ Track 1: Nova-Sec        β”‚  β”‚ Track 2: Nova-ARC        β”‚  β”‚ Track 3: Nova-Balanced   β”‚
β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€  β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€  β”œβ”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€
β”‚ RLVS Focus:              β”‚  β”‚ RLVS Focus:              β”‚  β”‚ RLVS Focus:              β”‚
β”‚ β€’ Prompt injection       β”‚  β”‚ β€’ ARC-AGI visual grid    β”‚  β”‚ β€’ Low-latency Cowork     β”‚
β”‚ β€’ Secure code audit      β”‚  β”‚ β€’ Spatial transform      β”‚  β”‚ β€’ General debugging      β”‚
β”‚ β€’ Exploit verification   β”‚  β”‚ β€’ Step-wise logic        β”‚  β”‚ β€’ Real-time OS agent     β”‚
β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜  β””β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”€β”˜
Enter fullscreen mode Exit fullscreen mode

1. Track 1: Nova-Sec (Cybersecurity & Defenses)

  • Reward Function: Deterministic AST security analyzers and sandbox test runners.
  • Goal: Detecting hardcoded secrets, injection sinks, broken access control, and sanitizing untrusted inputs.
  • Why RLVS? In cybersecurity, a fix either prevents an exploit or it doesn't. Binary unit-test verification rewards the model only when the security patch passes all attack vectors.

2. Track 2: Nova-ARC (ARC-AGI Reasoning)

  • Reward Function: FranΓ§ois Chollet's ARC (Abstraction and Reasoning Corpus) transformation rules.
  • Goal: Inductive spatial reasoning, rotational invariants, and pattern extrapolation in small matrix grids.
  • Why RLVS? Grid outputs have an absolute mathematical ground truth. A test either matches the grid or fails, providing zero-noise reward gradients.

3. Track 3: Nova-Balanced (Desktop Coworker)

  • Reward Function: Multi-turn task completion across terminal commands, file edits, and browser actions.
  • Goal: Powering the local subagent engine for LuxurAI Cowork with ultra-low RAM and sub-second reaction speeds.

πŸ’‘ Why RLVS Beats Standard Multi-Task Fine-Tuning

In typical supervised fine-tuning, training on diverse datasets causes gradient interference (learning security rules degrades spatial reasoning, and vice-versa).

By freezing the SFT tool-calling base and training three separate RLVS specialists:

  1. Each model focuses its 57M parameters on a single verifiable objective.
  2. We can hot-swap models dynamically inside our client runtime depending on whether the task is a code audit, reasoning puzzle, or desktop workflow.
  3. Memory stays locked at under ~50MB per running instance.

πŸ’¬ Community Question

For those training small-language models (SLMs) under 100M parameters:

  • Have you found greater success with specialized RL verifiers vs multi-task SFT?
  • What reward penalty works best for preventing regression during reinforcement loops?

Drop your thoughts below!

Top comments (0)