DEV Community: dang phan

A 4B Model Just Beat a 397B Baseline - By Changing How Training Data Was Made

dang phan — Sun, 28 Jun 2026 03:02:33 +0000

Meta FAIR just published a result that's hard to ignore: a 4B parameter model, after being trained on data generated by Autodata, outperformed their own 397B model on PRBench-Legal - without any architectural changes.

The only variable: how the training data was created.

The Problem With Current Synthetic Data Pipelines

Most synthetic data workflows follow the same pattern: prompt a model, collect outputs, filter, done. The problem is that data quality is essentially uncontrolled.

Two failure modes keep showing up:

Too easy - the model you're trying to train already solves it. No learning signal.
Too hard - every rollout scores near zero. GRPO has no gradient to work with.

Autodata reframes the question: instead of generating data and then evaluating it, why not let model behavior itself define what good data looks like?

How the Pipeline Works

Autodata runs an orchestrator agent that coordinates four LLM subagents:

Challenger - generates questions and rubrics from source material (papers, legal docs, math problems)
Weak solver - a small model that should struggle if the data is good
Strong solver - a large model that should succeed, validating that the question is actually answerable
Judge - scores both solvers against the rubric and sends structured feedback back to the orchestrator

An example is only accepted when all three conditions hold simultaneously: weak solver scores low, strong solver scores high, and the gap between them is large enough. Otherwise, the orchestrator sends specific feedback to the Challenger to generate a new question from an entirely different angle - not a rephrasing, but a new angle.

It takes an average of 6.59 iterations to produce a single accepted question.

The Results

On both PRBench-Legal and the harder PRBench-Legal-Hard subset, graded independently by both GPT-5 and Kimi-K2.6, the 4B model trained on Autodata came out on top across every column - outperforming the CoT-trained 4B and the 397B baseline without RL.

The same pattern holds across CS research tasks and scientific reasoning: Agentic data leads from the start, and the gap widens through training.

What's Actually Going On Here

The 4B > 397B result isn't the most interesting part. The more important question is why it happened.

On legal tasks, standard CoT Self-Instruct produced questions that were too hard - weak solver scores clustered near zero across almost every rollout. When every rollout fails the same way, GRPO has nothing to learn from. Autodata didn't make the questions easier. It reshaped the reward distribution, pushing the weak solver into a range with enough variance for gradient descent to do its job.

That's the difference between hard data and useful data.

The Real Takeaway

If you're building an SFT or RL training pipeline, it's worth asking: what are you actually using to measure data quality?

If the answer is a static rubric or a generic LLM-as-judge score, Autodata suggests the more important metric is target model behavior: good data is data that sits in the right difficulty zone for the model you're training - not data that scores well on a judge prompt.

Full paper:

[2606.25996] Autodata: An agentic data scientist to create high quality synthetic data

We introduce Autodata, a general method that enables AI agents to act as data scientists who build high quality training and evaluation data. We show how to train (meta-optimize) such a data scientist agent, so that it learns to create even stronger data. We describe the overall formulation, and a specific practical implementation, Agentic Self-Instruct. We conduct experiments on computer science research tasks, legal reasoning tasks and reasoning with mathematical objects, where we obtain improved results compared to classical synthetic dataset creation methods. Further, meta-optimizing the data scientist agent itself delivers an even larger performance uplift. Agentic data creation provides a way to convert increased inference compute into higher quality model training. Overall, we believe this direction has the potential to change the way we build AI data.

arxiv.org

The 'Shipping Container' for AI: What is the Model Context Protocol (MCP) and Why It Matters?

dang phan — Fri, 26 Jun 2026 13:48:43 +0000

Imagine you are building a custom AI agent to manage your company's logistics. On day one, it needs to read from a PostgreSQL database. You write the integration. On day two, it needs to check a third-party weather API. You write another integration.

By day ten, your agent is a tangled mess of custom code for every single data source. Every time you switch Large Language Models (LLMs), you have to rebuild those connections from scratch. This "integration tax" is the silent killer of AI productivity.

We need a way to stop reinventing the wheel every time an agent needs context. We need a universal standard that allows any AI agent to talk to any data source without custom glue code.

The Intuition: The Shipping Container for AI Data

Imagine global logistics before the 1950s. Every company had custom-sized crates, and dock workers had to learn a different way to handle every single package. It was a logistical nightmare. The world only became efficient when we standardized the "shipping container." A crane operator doesn"t need to know if a container holds grain or high-end electronics—they just need to know how to move a container.

The Model Context Protocol (MCP) is exactly that: a standardized shipping container for your AI.

What Is MCP?

Introduced by Anthropic in late 2024, MCP is an open standard protocol designed to provide context to LLMs. It replaces the need for custom-built Application Programming Interfaces (APIs) for every single tool or data source an agent uses.

Architecturally, MCP operates on a client-server relationship using JSON RPC 2.0 sessions. There are three core components you need to know:

MCP Host: The main application, such as a chat app or a code assistant in your IDE.
MCP Client: Resides within the host and maintains the connection to the servers.
MCP Server: A small service that exposes specific capabilities like database access, file systems, or web search.

These components communicate through the standardized MCP protocol, which defines how clients and servers exchange requests and capabilities.

The Three Primitives of MCP

A server doesn't just "send data." It advertises its capabilities through three specific primitives:

Tools: Discrete actions the AI can call, like get_weather or create_calendar_event. The server lists the tool name, description, and input/output schemas.
Resources: Read-only data items such as text files, database schemas, or specific documents that the client can fetch on demand.
Prompt Templates: Predefined templates that suggest how an agent should frame its requests to the server.

Because every MCP server publishes a machine-readable catalog (e.g., tools/list), an agent can discover and use new functionality at runtime without you having to redeploy your code.

Why MCP Exists: Solving the Standardization Gap

Before MCP, developers had to write copious amounts of code to specify tool metadata and enable third-party integrations for every project.

Standardization matters because it brings five key benefits to the development lifecycle:

Interoperability: You can build applications that work across different platforms, models, and vendors (e.g., switching from OpenAI to Anthropic without changing your tool code).
Reusability: Build a tool once and reuse it across multiple projects.
Consistency: Your tools will behave the same way regardless of the model being used.
Extendability: It is much easier to add new functionality in the future without breaking the existing environment.
Rapid Development: No more reinventing the wheel for every custom integration.

Aspect	Before MCP	After MCP
Integration Method	Custom API glue code	Standardized protocol (JSON RPC 2.0)
Data Handling	Siloed, manual ingestion	Normalized via "shipping containers"
Maintenance	High; code-heavy updates	Low; discoverable tools at runtime
Architecture	Rigid, direct connections	Scalable, plug-and-play
Flexibility	Vendor/Model lock-in	Interoperable across models

Beyond developer productivity, MCP improves the AI's performance. By connecting to live, factual databases, it significantly reduces AI hallucinations.

It also supports enhanced security through industry standards like OAuth 2.0 for authorization and TLS/SSL for encrypted communication traffic.

MCP vs API: What's the Difference?

You might be wondering: "Aren't we already doing this with APIs?" Not exactly. While APIs (especially RESTful ones) are the "web default," they function as abstraction layers.

With a REST API, you must follow specific rules (GET, POST, PUT, DELETE) and formats for each individual service. A library system API might require a specific URL format just to fetch one book.

MCP, on the other hand, is a protocol that sits on top of these sources. An MCP server can actually connect to a REST API or a database and then present those capabilities to the AI in a uniform way.

Aspect	MCP (Model Context Protocol)	REST API
Nature	Standardized communication protocol	Web default architectural style
Connection	JSON-RPC 2.0 over standardized transport	HTTP methods (GET, POST, PUT, DELETE)
Discovery	Dynamic runtime discovery of tools/resources	Manual integration/endpoint mapping
Abstraction	Standardized containers (High-level)	Endpoint-specific (Low-level)
Primary Goal	Context & Tool standardization for AI	Universal web data & service access

A Real-World Scenario: The 100,000 Document Bank

To see MCP in action, imagine you are an AI engineer building a Retrieval Augmented Generation (RAG) system for a bank with 100,000 documents.

Normally, you would have to manage a complex vector database and scale an infrastructure to handle embeddings and retrieval logic yourself.

With MCP, you can use a dedicated MCP server. When the LLM receives a query, it sends a prompt to the MCP server.

The server performs the retrieval step, finds the most relevant document chunks, and returns only those snippets to the LLM. The server handles the "how," while the agent focuses on the "what."

Key Takeaways

Standardization is the Core: MCP acts as a universal layer, allowing developers to build tools that interact with any resource uniformly.
Architecture is Simple: It uses a client-server, plug-and-play model based on JSON RPC 2.0.
Discovery at Runtime: Agents can query a server’s catalog to discover tools and resources without manual coding for each update.
Better AI Results: By providing real-time data access, MCP reduces hallucinations and ensures information relevance.
Broad Utility: It’s already being used in enterprise tasks (CRM, ticketing), agentic workflows, and specialized fields like DevOps and SecOps.

hat Makes a Good SFT Sample (And Why Most Synthetic Datasets Get It Wrong)

dang phan — Wed, 03 Jun 2026 02:45:48 +0000

You've decided to fine-tune a language model. You generate a few hundred samples, load them into Axolotl or LLaMA-Factory, kick off training, and wait.

The model comes out... worse. Or weirdly repetitive. Or it answers every question the same way regardless of context.

The problem usually isn't your training setup. It's the data.

The Uncomfortable Truth About Synthetic Datasets

Generating synthetic SFT data is easy. Generating good synthetic SFT data is not.

Most pipelines do something like this:

Write a prompt: "Generate 500 customer support examples"
Call an LLM
Save the output
Fine-tune

The problem? There's no mechanism to catch what goes wrong. And a lot goes wrong.

What a Bad Sample Looks Like

Here's a real example of what gets generated without any quality filter:

Instruction: Respond to a customer asking about their refund status.

Output: "Thank you for reaching out. Your refund is being processed. Please wait and we will get back to you soon."

Looks fine at first glance. But think about what your model learns from this:

When someone asks about their refund, say something vague
Don't give a timeline
Don't offer any next steps
Just acknowledge and deflect

Fine-tune on enough samples like this and your model becomes an expert at saying nothing helpfully.

Now here's what a good sample looks like:

Instruction: Respond to a customer asking about their refund status.

Output: "Your refund was approved on the 14th and has been submitted to your bank. Most banks process these within 3–5 business days, so you should see it by the 19th. If it hasn't appeared by then, reply here and I'll escalate immediately."

Same instruction. Completely different training signal.

The 4 Things That Actually Matter

When I evaluate an SFT sample, I look at four dimensions:

1. Relevance
Does the response directly address the instruction? An off-topic or partially relevant answer teaches your model to drift. Even a grammatically perfect response scores zero if it doesn't answer what was asked.

2. Factual Consistency
Are the claims in the response plausible and internally consistent? Hallucinated order numbers, impossible timelines, contradictory policies - these all slip through if you're not checking. Your model will learn to hallucinate the same way.

3. Format Quality
Is the response correctly structured for the schema you're using? A broken JSON field or a response that ignores the output format contaminates your training data at the structural level.

4. Response Usefulness
Would this response actually help someone? This is the hardest one to catch automatically - a response can be relevant, factually consistent, and correctly formatted while still being completely useless. Vague acknowledgements without concrete next steps fail here.

Why You Can't Just Prompt Your Way Out of This

A common fix people try: write a better generation prompt. Add instructions like "be specific", "include timelines", "don't be vague".

It helps. But it doesn't solve the problem.

LLMs drift across long generation runs. The first 50 samples might follow your instructions carefully. By sample 300, the model is taking shortcuts, repeating patterns, and producing outputs that technically match the format but miss the intent.

You need a separate evaluation pass - not more instructions in the generation prompt.

How a Judge Stage Works

The idea is simple: use a second, stronger model to score every sample your generation model produces.

The judge model doesn't generate. It evaluates. It reads each instruction-output pair and scores it on the four dimensions above, using a calibrated rubric with fixed anchor examples so the scoring stays consistent across the entire dataset.

Samples below a quality threshold get cut. You deliberately generate more than you need so the filtering doesn't leave you short.

The result: every sample that reaches your training loop has been independently evaluated, not just generated.

The Full Pipeline

In practice, the judge stage is one part of a larger quality process. Here's what a complete pipeline looks like:

Stage 1 - Generation
Domain-aware generation with a domain-specific system prompt. Rolling context injection prevents semantic drift across batches.

Stage 2 - Validation & Deduplication
Schema validation rejects malformed rows. Token length filtering removes samples outside the training-safe range. Deduplication (MinHash or semantic-based) removes similar samples that inflate dataset size without adding diversity.

Stage 3 - LLM-as-Judge Scoring
Every sample scored on the four dimensions above. Only samples above the threshold proceed.

Stage 4 - Human Review
Outputs reviewed for quality patterns before the final split. If a recurring issue is found, the threshold is adjusted and the affected stage re-runs.

Stage 5 - Split & Export
Shuffled 90/10 train/validation split. Output as production-ready JSONL in Alpaca or ShareGPT format.

What This Looks Like in Practice

On a recent 500-sample run, here's what the filter funnel looked like:

Generated: 600 samples
After schema validation: 584 (-16 malformed rows)
After token length filter: 572 (-12 too short / too long)
After deduplication: 569 (-3 near-duplicates)
After LLM judge: 500 (-69 below quality threshold)

The judge stage removed ~11% of samples. Deduplication removed less than 1%. Schema errors were caught early.

The 69 samples removed by the judge are the ones that would have quietly degraded your model.

The Takeaway

A good SFT sample isn't just one that looks correct - it's one that teaches your model the right behavior.

Vague responses teach vague behavior. Hallucinated details teach hallucination. Near-duplicate samples teach repetition.

If you're building a domain-specific LLM, the quality of your training data matters more than almost any other variable. A better base model won't save you from bad data.

What's Next

I'm currently building a public Vietnamese legal Q&A dataset using this pipeline - one of the few domains with almost no public SFT data available. I'll share it on HuggingFace when it's ready.

If you're working on a fine-tuning project and need a validated dataset for your domain, I build these as a service on Fiverr. Link in the comments.

If this was useful, follow for more posts on LLM fine-tuning, dataset preparation, and practical ML engineering.

Building Idea2Socdia: A Multimodal AI Agent with Gemini and Vertex AI

dang phan — Sun, 15 Mar 2026 15:26:01 +0000

I created this piece of content for the purposes of entering the Gemini Live Agent Challenge.

As a recent Computer Science graduate, transitioning from academic machine learning models to deploying a production-ready, Cloud-Native AI system is a thrilling leap. For this hackathon, I wanted to tackle a real-world problem: the "context-switching fatigue" that content creators face when juggling scriptwriting, image generation, and video rendering tools.

The solution is Idea2Socdia, a Human-In-The-Loop (HITL) multimodal AI agent. Here is a deep dive into how I built it using Google's ecosystem.

The Architecture: 100% Cloud Native

To ensure scalability and maintain a stateless architecture, the system is fully deployed on Google Cloud:

Frontend: A responsive Next.js application hosted on Google Cloud Run.
Backend: A high-performance FastAPI server, also containerized via Docker and running on Cloud Run.
Media Storage: Google Cloud Storage (GCS) securely holds all generated assets and serves public URLs back to the client.

The Brain: Gemini 3 Flash & Interleaved Generation

The core orchestration relies on Gemini 3 Flash via the new google-genai SDK. Instead of traditional multi-step prompting, Idea2Socdia leverages interleaved generation.

The LLM acts as a "Content Director." As it streams the strategic outline and script back to the Next.js frontend via NDJSON, it autonomously evaluates the narrative. When it determines a visual is needed, it pauses the text stream, constructs a highly contextual prompt, and triggers a media generation tool call.

The Media Engine: Vertex AI

For the visual components, the backend securely authenticates with Google Cloud via OAuth2 to access Vertex AI endpoints.
Depending on the target platform (e.g., a Facebook post vs. a YouTube Short), the agent dynamically decides whether to call state-of-the-art text-to-image models (like Nano Banana) or text-to-video models (like Veo). Once Vertex AI returns the media bytes, the backend streams them directly to GCS, bypassing the need for local persistent storage on Cloud Run.

Conclusion

Building Idea2Socdia provided deep, practical experience in orchestrating complex LLM workflows and managing enterprise-grade cloud resources. By strictly grounding the model and keeping the human in the loop, the system transforms raw ideas into ready-to-publish social campaigns seamlessly.

You can check out the public repository here: Idea2Socdia