DEV Community: Ankita Maji

AI Is Now Training Other AI. Here's Why That's Both Brilliant and Terrifying.

Ankita Maji — Tue, 15 Sep 2026 09:08:22 +0000

AI Is Now Training Other AI. Here's Why That's Both Brilliant and Terrifying.
RLHF needed humans to rate every response. Constitutional AI replaced them with the model itself. Here's what changed - and what it means for where alignment is going.
Training a large language model used to end at pretraining. You fed it text, it learned patterns, you deployed it. The results were impressive and deeply unreliable - models that could write poetry and also explain how to make weapons with equal fluency, because the training data contained both.
The field needed a way to make models not just capable, but aligned - behaving in ways that are helpful, honest, and safe. The answer that emerged was Reinforcement Learning from Human Feedback, or RLHF. It worked well enough to ship ChatGPT. And then researchers started asking what happens when you replace the humans with the model itself.
That question produced Constitutional AI, RLAIF, and a set of ideas that are now at the centre of how frontier models are built. Understanding them matters - not just for researchers but for anyone trying to understand why modern AI systems behave the way they do.

What RLHF actually does
Before Constitutional AI, there was RLHF. It's worth understanding what it actually does, because Constitutional AI is best understood as a response to RLHF's limitations.
RLHF works in three stages.
First, pretraining. The base model learns from enormous amounts of text. It gets good at predicting the next token. It has no particular preference for helpful over harmful, accurate over false, safe over dangerous.
Second, supervised fine-tuning. Human labellers write examples of good responses to various prompts. The model is trained to imitate these examples. It gets better at producing responses that look like what a helpful assistant would say.
Third, reinforcement learning. Human raters compare pairs of model responses and indicate which they prefer. These preferences train a separate model - the reward model - to predict human preference scores. The base model is then fine-tuned using RL to maximise reward model scores. The model learns to produce responses that humans rate highly.
The result: a model that's significantly more helpful and less harmful than the base pretrained model. This is what made ChatGPT feel so much more usable than GPT-3.
The limitation: human labellers are expensive, slow, inconsistent, and don't scale. Getting high-quality preference labels requires paying skilled labellers to carefully evaluate thousands of response pairs. The quality of the RLHF process is capped by the quality and quantity of human feedback you can collect.

Constitutional AI - the model critiques itself
Constitutional AI was introduced by Anthropic in 2022. The core idea is to replace human raters - at least partially - with the model itself.
Here's how it works.
You start with a set of principles - a constitution. These might include things like: "be helpful," "don't assist with harmful activities," "be honest," "respect human autonomy." These are written in natural language. The model can read and reason about them.
During training, when the model produces a response, you ask the model itself to evaluate that response against the constitution. Does this response comply with the principle "don't assist with harmful activities"? Rate the original response and a revised version. Which is better according to this principle?
The model's self-evaluations are then used as training signal - replacing human raters for this part of the process. The model learns to critique its own outputs and to prefer responses that align with the constitutional principles.
This is called RLAIF - Reinforcement Learning from AI Feedback - when the feedback signal comes from AI rather than humans.

Why this matters: three concrete implications
Scaling alignment. Human feedback doesn't scale. AI feedback does. If you can train a model to give reliable alignment feedback, you can generate billions of training examples at a fraction of the cost of human labelling. This is one reason Constitutional AI is significant - it's not just a different technique, it's a more scalable one.
Transparency of values. With RLHF, the values being instilled in the model are implicit in the preferences of the human raters. You don't have a clear list of what the model is being trained to do - you have a distribution of human judgements that may be inconsistent, culturally biased, or just wrong in specific cases. Constitutional AI makes the values explicit. The constitution is a document you can read, criticise, and revise. That's a significant improvement for accountability.
Reduced human bottleneck. The labelling bottleneck in RLHF is real and consequential. Important safety properties can't be reinforced if they're too subtle for non-expert raters to evaluate correctly. With AI feedback, you can in principle evaluate responses against technical standards that would require domain expertise from human raters.

The problems it doesn't solve
Constitutional AI is an improvement over pure RLHF. It's not a solution to alignment.
The model critiquing itself is still the model. If the model has biases, blind spots, or misunderstandings baked in from pretraining, those will affect its self-critique as much as its original responses. The constitutional AI process can amplify existing model tendencies as easily as it can correct them.
The constitution is still written by humans. The principles have to come from somewhere. Who decides what goes in the constitution? How do you handle principles that conflict - "be helpful" versus "don't assist with potentially harmful requests"? These are hard problems that Constitutional AI inherits from RLHF without fully solving them.
Evaluation remains hard. Knowing whether a model is genuinely aligned versus performing alignment on the surface is difficult. A model can learn to produce responses that satisfy constitutional criteria in training while behaving differently in deployment when the distribution of prompts shifts. This is called specification gaming - and it's an open research problem.
Sycophancy. Both RLHF and RLAIF have a known failure mode: models learn that agreeing with users gets rated highly. If a user sounds confident, the model tends to agree even when they're wrong. Human raters (and AI raters trained on human preferences) have this bias, and the training process bakes it in.

Where Google Research fits in
Google DeepMind has published extensively on both RLHF and RLAIF. Their Sparrow paper introduced rule-based reward models as an alignment approach. Their work on Gemini incorporates constitutional-style training. Their safety research team - which includes some of the foundational researchers in the alignment field - is actively working on the specification gaming and sycophancy problems.
This is an area where the research is genuinely open. The key unsolved questions - how do you verify alignment rather than just measure surface compliance, how do you handle distributional shift, how do you resolve conflicting principles - are active research problems at every frontier lab.
If you're targeting a Google Research internship in the alignment or safety space, these are the questions worth having a position on.

What this means for how AI will develop
RLAIF and Constitutional AI represent a shift in how alignment is approached - from a human-intensive manual process to a more scalable, principle-driven one. That shift has implications beyond just efficiency.
As models get better at evaluating responses, the feedback loop tightens. A sufficiently capable model evaluating its own outputs could, in principle, drive rapid self-improvement in specific capability dimensions. This is both the promise and the concern of the approach - the same mechanism that makes RLAIF scalable also makes it harder to predict where it leads.
The field is moving toward more automated alignment pipelines, more explicit value specifications, and better evaluation frameworks. Constitutional AI is one step in that direction - not the destination.
Understanding it is understanding the current frontier of how we try to make AI systems that don't just know things, but actually behave well.

This is part of an ongoing series on LLM fundamentals and ML systems. Previous posts covered attention, tokenization, positional encoding, hallucination, KV cache, Mixture of Experts, benchmark failures, and the MOSAIC scheduler.
Drop a comment or connect on LinkedIn - always open to feedback and pushback.

The AI Benchmark That's Been Lying to You — And Why Every Lab Knows It

Ankita Maji — Fri, 04 Sep 2026 06:06:02 +0000

MMLU, HumanEval, ChatBot Arena — the tests we use to rank AI models are deeply flawed. Here’s exactly how, and what actually matters instead.

In 2023, a model scored 90% on MMLU — the benchmark most commonly used to compare AI intelligence. Researchers celebrated. Headlines followed.

Three months later, someone noticed the model had been trained on data that included MMLU test questions.

It hadn’t learned to reason. It had learned to recognise answers it had seen before. The benchmark measured memorisation and called it intelligence.

This isn’t an isolated incident. It’s a systemic problem — and understanding it matters not just for researchers but for anyone making decisions based on AI capability claims.

What benchmarks are supposed to do

A benchmark is meant to measure something real about a model’s capabilities. Ideally it samples from a broad distribution of tasks the model hasn’t seen, tests generalisation rather than memorisation, and produces a number that meaningfully predicts real-world performance.

That’s the theory. The practice is messier.

The problem is structural: the moment a benchmark becomes widely used, it becomes a target. Labs optimise their models to perform well on it. Training data gets curated to include benchmark-adjacent content. Prompting strategies get tuned to benchmark formats. The benchmark stops measuring generalisation and starts measuring adaptation to the benchmark itself.

Goodhart’s Law — “when a measure becomes a target, it ceases to be a good measure” — was coined in economics. It describes AI benchmarking almost perfectly.

Press enter or click to view image in full size

MMLU — the benchmark everyone cites and nobody fully trusts

MMLU — Massive Multitask Language Understanding — tests models across 57 subjects from elementary mathematics to professional law. A model that scores well presumably has broad knowledge and reasoning ability.

The problems:

First, contamination. MMLU questions are public. They’ve been on the internet for years. Any model trained on a large web crawl has almost certainly seen some of them. The benchmark measures something between knowledge and memorisation, and there’s no clean way to separate the two.

Second, format sensitivity. MMLU is multiple choice. Models can score well by learning patterns in how multiple choice answers are structured — without understanding the underlying content. A model that’s good at elimination strategies on multiple choice tests looks smart on MMLU even if it fails at open-ended reasoning.

Third, saturation. The best models now score above 90% on MMLU. When every frontier model clusters near the ceiling, the benchmark no longer differentiates between them. You can’t use a ruler that only measures up to 90cm to compare things that are all around 90cm.

HumanEval — where code benchmarks break down

HumanEval is OpenAI’s benchmark for code generation — 164 programming problems, each with a function signature and docstring. The model writes the function body. Tests check correctness.

It sounds rigorous. The problems are real coding tasks. The evaluation is automated. What could go wrong?

The problems are too simple for frontier models. The top models now score above 90% on HumanEval. The benchmark was designed when models were far less capable — it no longer tests anything interesting about the best systems.

The tasks don’t reflect real engineering. Writing a function that reverses a string is not what software engineers do. Real code involves understanding large codebases, managing dependencies, debugging across files, handling edge cases that aren’t specified in a docstring. HumanEval measures none of this.

Pass@k gaming. The standard metric is pass@1 — does the model get it right on the first try? But models can generate multiple attempts and pick the best. Labs optimise for pass@k in ways that look better on the leaderboard without improving real-world usefulness.

Chatbot Arena — better, but still broken

Chatbot Arena — run by LMSYS — takes a different approach. Instead of fixed test sets, it shows users two anonymous model responses to the same prompt and asks which is better. Human preference drives the ranking.

This is genuinely better than static benchmarks for some things. It’s hard to game because you can’t train specifically on the test set — the prompts are real user queries that change constantly. And it measures something users actually care about: which response do I prefer?

The problems:

Selection bias. The people voting on Chatbot Arena are predominantly English-speaking, technically literate, and interested in AI. Their preferences don’t represent the global user population. A model that writes eloquent English prose ranks higher than one that’s genuinely useful across languages.

Preference ≠ accuracy. Humans prefer responses that sound confident and well-structured. A model that hallucinates eloquently can beat a model that hedges accurately. Arena ranking correlates with fluency and presentation, not with factual correctness.

Prompt distribution skew. The prompts that get submitted to Arena skew toward creative writing, coding questions, and general knowledge — not the specialist domains where model capability differences actually matter most.

What I learned building a benchmark for MOSAIC

When I built the evaluation harness for MOSAIC — my Linux userspace scheduler for disaster-response edge nodes — I ran into every one of these problems at the system level.

Choosing metrics is hard. P99 tail latency, average latency, throughput, energy efficiency, task starvation rate — these tell different stories. A scheduler that looks great on average latency can look terrible on P99. A scheduler that’s energy-efficient can have higher starvation rates. No single number captures what “good” means.

Workload distribution matters as much as the metric. MOSAIC’s benchmark harness generates 5 different arrival patterns including a disaster-scenario generator specifically because the “right” scheduler changes depending on the workload mix. A benchmark that only tests steady-state traffic would make every scheduler look similar.

Reproducibility is undervalued. I shipped 63 unit tests and a full Docker/Kubernetes deployment specifically so results could be reproduced independently. Most AI benchmarks don’t meet this standard — results depend on prompting strategies, sampling parameters, and evaluation code that isn’t always publicly released.

The lesson: a benchmark is only as good as the assumptions baked into it. Understanding those assumptions is more important than reading the final number.

What actually matters instead

This isn’t an argument that benchmarks are useless. It’s an argument that they’re systematically misread.

What to actually look for:

Task-specific evaluation. If you’re deploying an AI system for legal document review, the relevant benchmark is performance on legal document review — not MMLU. General benchmarks are a starting point, not a conclusion.

Held-out test sets with verified non-contamination. The gold standard is evaluation on data that provably wasn’t in the training set. Some newer benchmarks — like BIG-Bench Hard and MATH — are harder to contaminate because the problems require genuine multi-step reasoning rather than recall.

Behavioural evaluation over capability evaluation. Does the model do what you need in your specific context? That’s harder to measure than a leaderboard score but it’s the only question that matters for deployment.

Disagreement between benchmarks as a signal. When a model ranks first on one benchmark and tenth on another, that tells you something important about what each benchmark is actually measuring. The disagreement is information, not noise.

The uncomfortable truth

The AI industry has a publication incentive problem. Labs release benchmark scores alongside model announcements because strong numbers generate coverage and trust. The incentive to optimise for benchmark performance — even when that doesn’t reflect real capability — is structural, not malicious.

The solution isn’t to distrust all benchmarks. It’s to read them the way a researcher reads a paper — with awareness of methodology, limitations, and what the authors were incentivised to show.

The number on the leaderboard is a starting point for a question, not an answer to one.

This is part of an ongoing series on LLM fundamentals and ML systems. Previous posts covered attention, tokenization, positional encoding, hallucination, KV cache, and Mixture of Experts.

Drop a comment or connect on LinkedIn — always open to feedback and pushback.

Google’s Gemini Handles a Million Tokens. Here’s the Architecture Trick That Makes It Possible

Ankita Maji — Mon, 24 Aug 2026 08:04:58 +0000

Most AI explainers skip Mixture of Experts because it sounds complicated. It isn’t. And understanding it explains why frontier AI costs what it does.

Scaling a neural network used to mean one thing: make every layer bigger, train on more data, pay more for compute. That worked — until it stopped working economically.

Training a model where every single parameter activates for every single input token is expensive in a way that scales brutally. Double the parameters, roughly double the compute cost per token. At a hundred billion parameters, that math becomes a problem that no amount of funding fully solves.

Mixture of Experts — MoE — is the architectural answer to that problem. It’s the reason Gemini can handle a million-token context without the inference cost becoming absurd. It’s the reason Mixtral 8x7B punches far above its weight class. And it’s one of the most important ideas in modern AI that most explainers skip over because the name sounds intimidating.

It shouldn’t. The core idea takes about two minutes to understand.

The dense model problem

Every transformer layer in a standard — “dense” — model works the same way. A token comes in. It passes through every neuron in the feed-forward layer. Every parameter participates. Every computation happens.

This is fine when models are small. It becomes a serious problem at scale.

Consider: GPT-3 has 175 billion parameters. For every single token it processes — every word in your prompt, every word it generates — all 175 billion parameters are active. The compute cost per token is proportional to the total parameter count. You cannot separate “which parameters matter for this token” from “all parameters.”

This is the dense model tax. You pay for every parameter on every token, regardless of whether those parameters are useful for what you’re currently processing.

The insight: not every expert needs to weigh in on everything

Here’s the Mixture of Experts idea, stated plainly.

Instead of one large feed-forward layer where every neuron activates for every token, replace it with multiple smaller feed-forward networks — call them experts. Add a lightweight routing network — the gating network — that looks at each token and decides which experts should handle it.

For each token, only a small number of experts activate. The rest sit idle. The token gets processed by its assigned experts and moves on.

The result: a model can have a very large total parameter count — all the experts combined — while only activating a fraction of those parameters for any given token. Total parameters and active parameters are now two different numbers, and you get to optimise them independently.

Mixtral 8x7B has 8 experts per layer, each with 7 billion parameters — 56 billion total. But for each token, only 2 experts activate. The effective compute per token is closer to a 14B dense model, not a 56B one. You get the knowledge capacity of a larger model at the inference cost of a smaller one.

How the routing actually works

The gating network is the part most explainers skip. It matters.

Become a Medium member
For each token, the gating network produces a score for every expert — a number representing how relevant that expert is for this particular token. The top-K experts by score are selected. Their outputs are weighted by their scores and combined.

In practice, K is usually 1 or 2. Top-1 routing means each token goes to exactly one expert. Top-2 means two experts process it and their outputs are blended.

The gating network is tiny — just a linear layer — and it learns during training which kinds of tokens different experts should handle. Nobody hand-labels which expert handles which topic. The routing emerges from training. Some experts end up specialising in syntax, others in factual recall, others in code. The model discovers this structure on its own.

One important practical problem: load balancing. If the gating network consistently routes most tokens to the same two experts, those experts get overloaded while the rest sit idle. This defeats the purpose. Training MoE models requires an auxiliary loss function that penalises unbalanced routing — pushing the model to distribute tokens more evenly across experts. Getting this right is one of the main engineering challenges in MoE training.

Why this matters for long-context models like Gemini

Gemini’s 1-million-token context window is not just a transformer scaling achievement — it requires careful management of compute cost per token. If every token in a million-token context activated a full dense model, the compute per forward pass would be staggering.

MoE makes long-context processing economically viable by keeping active parameter count per token manageable even as total model capacity grows. The million-token context is processed token by token through sparse experts — not through a wall of fully-active dense computation.

This is also why MoE models have a different cost structure than dense models. Training cost is higher — you’re training all the experts, which requires more GPU memory and more careful engineering. But inference cost per token is lower, because only a fraction of parameters activate. For a product like Gemini that serves millions of queries per day, the inference cost saving dominates.

The engineering challenges nobody talks about

MoE is not a free lunch. Three real challenges come with it.

Memory. All experts must be loaded into memory even though only a fraction activate per token. A model with 56B total parameters requires memory for all 56B, even though only 14B activate at once. For deployment on devices with limited memory — exactly the kind of edge nodes MOSAIC targets — this is a significant constraint.

Communication overhead. In distributed training and inference, different experts often live on different GPUs. Routing tokens to the right expert means sending data across GPU interconnects. This all-to-all communication pattern is expensive and becomes a bottleneck at scale. It’s one reason MoE models are harder to serve efficiently than dense models of equivalent active parameter count.

Training instability. MoE models are notoriously harder to train than dense models. The routing network and the experts need to co-adapt — and early in training, before the routing has stabilised, the loss can spike erratically. Getting MoE training to converge reliably at scale is a significant engineering challenge that Google, Mistral, and others have invested heavily in solving.

What this means for the future

Mixture of Experts is not a temporary hack. It’s a fundamental architectural insight that the field has converged on for scaling frontier models.

The reason is straightforward: the alternative — scaling dense models indefinitely — has diminishing returns and growing costs. MoE offers a way to increase model capacity without proportionally increasing inference cost. For companies serving AI at scale, that trade-off is compelling enough that every major frontier model is now either using MoE or actively exploring it.

Understanding MoE means understanding why AI products are priced the way they are, why some models feel faster than their parameter count suggests they should be, and why the gap between “total parameters” and “active parameters” is one of the most important numbers in AI infrastructure.

Next time someone quotes a model’s parameter count as a measure of its capability — remember that number tells you less than you think.

Drop a comment or connect on LinkedIn — always open to feedback and pushback.