DEV Community: Rijul Rajesh

Continuous Batching: How Do LLM Servers Handle Many Users at Once?

Rijul Rajesh — Fri, 23 Oct 2026 08:18:21 +0000

Hello, I'm Rijul, and I'm building LiveReview — a blast-radius aware AI code review built for your business-critical systems. Star us to help devs discover the project, give it a try, and share your feedback to help improve the product.

When you use an AI chatbot, have you ever wondered how it can respond to thousands of users with a limited number of GPUs?

One technique that makes this possible is continuous batching.

To understand how it works, let's first look at why processing one request at a time is inefficient, and how continuous batching helps LLM servers serve more users with the same hardware.

1. Why Generating Tokens Can Be Wasteful

A language model generates its response one token at a time. For each generation step, the GPU must perform computations using the model's weights.

For many models during token generation, moving these weights from memory can take more time than performing the calculations themselves. This means the GPU's computing power may not be fully utilized.

Think of a chef who walks to a distant pantry to collect ingredients but prepares only one dish per trip. Much of the time is spent walking instead of cooking.

The obvious solution is to prepare several dishes during each trip.

2. The First Fix: Batching

Instead of processing one user's request at a time, an LLM server can process multiple requests together. This is called batching.

The server loads and uses the model's weights to process several requests during the same generation step. Although processing a larger batch requires additional computation, the cost does not necessarily increase in proportion to the number of requests.

The result is that the server can generate tokens for multiple users during the same step.

This improves throughput, which means the total number of tokens generated per second across all users.

However, batching introduces another problem: different users' requests finish at different times.

3. The Problem With Static Batching

A simple batching strategy, called static batching, works like this:

Collect a group of requests, such as four.
Process them together.
Wait until all four requests finish before starting the next group.

Consider these four requests:

Request	Tokens to generate
A	10
B	500
C	40
D	60

Request A finishes after generating 10 tokens. Requests C and D finish at steps 40 and 60, respectively. But request B continues generating tokens until step 500.

In a conventional static batch, the completed requests cannot simply be replaced with new ones while the batch is running. Their slots may remain unused until the longest request finishes.

For most of the generation process, the server could be working on request B while the other three slots sit idle.

Meanwhile, new users may be waiting in a queue, even though some of the batch's capacity is no longer being used.

4. The Solution: Continuous Batching

Continuous batching solves this problem by reconsidering the batch after every generation iteration instead of waiting for the entire group to finish.

The process works like this:

The server generates the next token for the active requests.
When a request finishes, the server releases its slot.
If another request is waiting, the server can schedule it into the available capacity.
The process repeats while requests continue arriving and finishing.

Let's return to our example.

Request A finishes at step 10. The server can schedule a new request, E, to use the available capacity in a subsequent iteration. When request C finishes at step 40, another waiting request, F, can take its place.

Instead of keeping the original group fixed, the server continuously adapts to the requests that are still running and the requests waiting to start.

The goal is to keep the GPU productively occupied and improve overall throughput.

Think of a restaurant. With static batching, the restaurant waits for every table in a group to become empty before seating another group of customers. With continuous batching, a new party can be seated as soon as a table becomes available.

You may also encounter the terms in-flight batching and iteration-level scheduling. These are commonly used to describe this general approach.

5. What Happens When a New Request Arrives?

A new request does not immediately start generating its answer. It first needs to pass through two distinct stages.

Prefill: Reading the prompt

During prefill, the model processes the user's input prompt. Unlike generation, where the model produces one new token per step, prefill can process the prompt's tokens in parallel within the model's sequence computation.

Long prompts can require substantial computation and memory, so prefill can temporarily compete with the decoding work of existing requests.

Decode: Generating the answer

During decode, the model generates the response one token at a time. Each new token depends on the preceding context.

Continuous batching is especially useful for keeping multiple requests' decode steps moving together.

The challenge of long prompts

Suppose a new user submits a very long document while several other users are waiting for their next tokens.

Processing the entire prompt in one large prefill operation could delay the decoding of existing requests.

One technique used to manage this is chunked prefill. The server splits a long prompt into smaller chunks and schedules those chunks alongside ongoing decode work.

This can help balance prompt processing and token generation, improving scheduling efficiency. The exact behavior depends on the serving system and its configuration.

6. Where Can You Try Continuous Batching?

You generally do not need to implement continuous batching from scratch. Modern LLM serving systems provide batching and request-scheduling mechanisms.

Examples include:

vLLM
NVIDIA TensorRT-LLM
Hugging Face Text Generation Inference (TGI)

The specific implementation and available configuration options differ between systems and versions. If you deploy a model using one of these servers, check its documentation to understand how continuous batching is enabled and configured.

Conclusion

Serving many users with a limited number of GPUs is not just about using a bigger model or adding more hardware. It is also about making better use of the hardware already available.

Static batching groups requests together but can waste capacity when shorter requests finish before longer ones. Continuous batching addresses this by allowing new requests to join as existing requests complete.

By adapting the batch at each generation iteration, LLM servers can use their available capacity more efficiently and improve throughput.

The key idea is simple: don't wait for every request to finish before making room for the next one.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Risk-Aware AI Code Review for Business-Critical Systems

LiveReview: Risk-Aware AI Code Review for Business-Critical Systems

LiveReview is built for engineering teams that ship often, but also care about keeping production stable for customers. It approves some 40-50% of low-risk changes automatically, and it gives your team the tools to handle the rest:

Understand risky changes faster.

Fix problems in those changes, with human attention directed to the work that needs it.

Enforce your engineering standards.

Learn from production outcomes, such as incidents, rollbacks, and queue wait times.

Improve the review process over time and make it more efficient.

Connect to the major engineering tools that your team already uses.

Auto-Approve: Ship the Safe 50% Without a Human

Auto-Approve Upto 50% of Your Pull Requests, Safely. Risk scoring decides which MRs/PRs are safe enough to ship without a human, so your team only spends review time on risky and critical changes.
demo.mp4
LiveReview auto-approving a low-risk change while…

View on GitHub

Click below to try LiveReview with your codebase:

You Speak to AI, and It Speaks Back. What Happens in Between?

Rijul Rajesh — Thu, 22 Oct 2026 04:04:27 +0000

You might have heard of audio-to-audio AI models. But have you wondered how they work behind the scenes?

An audio-to-audio model listens to your voice and responds with a voice. Unlike a traditional speech pipeline, it does not necessarily need to convert your speech into text before generating a response.

Instead, it can represent sound as a sequence of small codes. The model processes these codes to understand the input and generate a response, then converts the output codes back into audio.

Let's break down how this works.

1. Sound Is Just Numbers

Consider a microphone.

When you speak, sound waves travel through the air. The microphone captures these vibrations and converts them into an electrical signal. An audio system then samples that signal many times per second, storing each measurement as a number.

For example, audio sampled at 24,000 samples per second contains 24,000 numerical samples for every second of audio.

That means a 10-second voice message contains 240,000 samples.

That's a lot of numbers for a model to process directly. More importantly, the raw waveform contains far more detail than the model needs to represent speech efficiently.

We need a way to compress the sound while preserving the information that matters.

2. The Compression Tool: A Neural Audio Codec

A codec is a system that encodes data into a representation and decodes it back. Think of it as a tool for compressing and reconstructing information.

A neural audio codec does this for sound using neural networks. Examples include SoundStream, EnCodec, and Mimi.

A neural audio codec has two main components:

Encoder: Converts the audio waveform into a compact sequence of codes.
Decoder: Uses those codes to reconstruct an audio waveform.

Think of a song and its sheet music. The recording contains a huge amount of information, while the sheet music provides a more compact representation of what needs to be played. A musician can use it to recreate the song, although the performance may not be identical to the original recording.

Audio codes serve a somewhat similar purpose. They represent the sound in a form that a model can process more efficiently.

3. Turning Sound Into Frames

The encoder does not necessarily represent the entire recording as one giant code. Instead, it divides the audio into small time steps called frames.

Consider a codec that produces 12.5 frames per second. Each frame corresponds to approximately 80 milliseconds of audio.

The encoder converts each frame into a compact representation. The model can then process these representations in sequence, much like a language model processes tokens.

But there is another detail: a frame can contain more than one code.

4. Why Each Frame Can Have a Stack of Codes

Some neural audio codecs use a technique called Residual Vector Quantization (RVQ).

The name sounds complicated, but the basic idea is straightforward: represent the audio using several layers of codes, with each layer refining the representation provided by the previous layers.

Imagine approximating the number 7.34:

Code 1: Represents approximately 7.
Code 2: Adds a correction of approximately 0.3.
Code 3: Adds another correction of approximately 0.04.

Together, these values represent 7.34 more accurately than the first approximation alone.

RVQ works with learned codebooks and vector representations rather than literally adding decimal numbers like this. The example illustrates the idea of progressively refining a representation.

Each frame can therefore contain a stack of codes, with each code contributing additional information about the audio.

5. How Much Does the Representation Shrink?

Let's use a simplified example for one second of audio.

Measurement	Count
Raw audio at 24 kHz	24,000 samples
Encoded frames at 12.5 frames per second	12.5 frames
Codes per frame	8
Total audio codes	100

With these assumptions, the audio is represented by approximately 100 discrete codes instead of 24,000 waveform samples.

That is a reduction of roughly 240 to 1 in the number of represented values.

The decoder can use the resulting codes to reconstruct speech that preserves much of the original audio's important characteristics.

6. Speech Contains Meaning and Sound Details

When someone speaks, the audio carries more than the words themselves.

There are two useful ways to think about this information:

Semantic information: What is being communicated. For example, "Please close the door."
Acoustic information: How the message sounds, including voice characteristics, intonation, speaking rate, pauses, and emotional expression.

A voice assistant needs to handle both. Understanding the words is not enough if the goal is to produce natural, expressive speech. Sound details matter too.

Different audio models organize this information in different ways. For example:

Separate representations: A system may use distinct representations for semantic information and acoustic details.
Layered representations: Some codec designs arrange multiple code sequences so that different layers capture different levels of detail.

However, these are not universal rules. In an RVQ codec, the first code is not automatically a semantic representation, and later codes are not necessarily dedicated to emotion or voice characteristics. That behavior depends on the model's architecture and training.

The key idea is that a model must represent enough information to understand the input and generate the kind of audio response it is designed to produce.

7. How an Audio-to-Audio Model Responds

Once the input speech has been encoded into codes, the main model can process them and generate codes for its response.

The overall process has three main stages.

Step 1: Listen

The audio encoder converts the incoming waveform into a sequence of audio codes.

Step 2: Predict

The main model processes the input representation and generates a sequence of output codes. Depending on the architecture, it may also use text tokens, separate audio representations, or other intermediate information.

Like an autoregressive language model, an audio model may generate tokens one step at a time. Other architectures can organize generation differently.

Step 3: Speak

The audio decoder converts the generated codes back into a waveform, producing the model's spoken response.

The simplified pipeline looks like this:

Input waveform → Encoder → Audio codes → Model generates response codes → Decoder → Output waveform

The model does not have to treat audio as raw numbers throughout the entire process. The codec gives it a compact representation that is more suitable for modeling.

8. Why Not Convert Speech to Text First?

A traditional speech-based assistant might use a three-stage pipeline:

Speech → Speech recognition → Text model → Text-to-speech → Spoken response

This approach works well for many applications. However, converting speech into text can discard information that is not represented in the transcript.

For example:

Emotion: A transcript may preserve the words but lose much of the excitement, frustration, or uncertainty in the speaker's voice.
Timing: Pauses, speaking rhythm, and interruptions may not be fully represented in the text.
Non-speech sounds: Laughter, sighs, and other vocal sounds may be omitted or represented only with additional labels.

An audio-to-audio model can preserve and process more of this information when its architecture and training support it. That can help it produce responses with more natural timing and vocal expression.

It may also reduce the need for separate speech-recognition and speech-synthesis stages, potentially lowering latency. The actual speed advantage depends on the model and the system in which it runs.

Conclusion

Audio-to-audio AI models do not have to process an entire sound waveform directly. A neural audio codec converts the waveform into compact codes, a model processes those codes to generate a response, and a decoder reconstructs the output as sound.

The important idea is that audio codes give AI models a compact representation of sound that they can learn to understand and generate.

This is one of the building blocks behind more natural voice interactions, where an AI system can respond to spoken input without relying entirely on a speech-to-text-to-speech pipeline.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

What If a Small AI Model Could Speed Up a Big One Without Changing Its Answers?

Rijul Rajesh — Tue, 20 Oct 2026 05:28:54 +0000

Language models generate text one token at a time. Each new token depends on the tokens generated before it, so the model usually needs another forward pass before it can produce the next token.

For a small model, this may be fast enough. For a large model, however, generating text can feel slow. The reason is not always what you might expect.

1. Why Generating Tokens Can Be Slow

You might assume the GPU spends all its time performing heavy computations. In many inference workloads, that isn't the case.

To generate a token, a large language model must access its weights, which can occupy many gigabytes of memory. For models limited by memory bandwidth, moving these weights from memory to the GPU's compute units can take longer than performing the calculations themselves.

As a result, the GPU's compute units may sit underutilized while waiting for data.

Think of a chef who spends 10 minutes fetching ingredients from a distant pantry and only 10 seconds cooking. Faster cooking wouldn't solve the main problem. Fewer trips to the pantry would.

This is similar to memory-bound inference: the bottleneck is moving data, not necessarily performing calculations.

2. The Opportunity: Process Multiple Tokens at Once

During autoregressive generation, the model typically processes one new token at a time. But when processing multiple known tokens together, the model can reuse the weights it has loaded instead of repeatedly loading them for each token.

Processing five tokens together can therefore cost much less than running five separate single-token steps, especially when inference is memory-bound.

This does not mean five tokens always cost the same as one. The exact cost depends on the model, hardware, sequence length, and workload. But processing several tokens in parallel can make better use of the available computing resources.

Now imagine you already had a good guess for the next five tokens. The large model could check those guesses together rather than generating every token sequentially.

But where would those guesses come from?

3. The Idea: A Small Draft Model

Speculative decoding uses a second, smaller model called a draft model to propose upcoming tokens.

The larger model, called the target model, then verifies those proposals.

Think of it this way:

Draft model: A quick intern who prepares a rough draft.
Target model: An expert editor who checks the draft.

The editor can review an existing draft more efficiently than writing every word from scratch. If most of the draft is correct, the editor can approve several words at once and correct the first mistake.

That is the basic idea behind speculative decoding.

4. How Speculative Decoding Works

Let's use this prompt as an example:

“The capital of France is”

Suppose the draft model proposes four tokens:

“ Paris”, “,”, “ a”, “ city”

Here's what happens next.

Guess: The small model quickly guesses the next four words: “ Paris”, “,”, “ a”, “ city”.
Check: The big model looks at all four guesses at the same time. For each spot, it figures out which word it would have picked itself.
Keep the good ones: Going from left to right, we keep each guess that matches the big model’s choice. Here, the big model agrees with “ Paris”, “,”, and “ a”. But for the fourth spot, it would have picked “ major” instead of “ city”.
Fix the first mistake: We swap “ city” for the big model’s word, “ major”. If there were any guesses after the wrong one, we throw them away, because they were built on a mistake.
Start again: The small model takes the corrected sentence and guesses the next few words.

In this example, the target model produces three accepted tokens plus its correction in one verification round. That gives four new tokens instead of the single token it would ordinarily produce in a standard autoregressive step.

If all four draft tokens are accepted, the target can also contribute a bonus token, potentially producing five new tokens in that round.

The important point is that the target model verifies several proposed tokens together rather than generating each one separately.

5. What Happens When a Draft Is Wrong?

Sometimes the very first guess is wrong. When that happens, the big model rejects it and picks the word itself.

So you still get one correct word, just like you would without this trick. The only cost is a little wasted time: the small model made a guess, and the big model checked it, but the guess was thrown away.

If the small model is wrong more often, each check produces fewer words, so you save less time.

But here’s the important part: if it’s set up correctly, the final text is exactly what the big model would have written on its own.

In other words, this trick makes writing faster without changing the quality or the style of the answer. Only the speed changes.

Greedy decoding

When a model writes, it gives a score to every possible next word. There are two common ways to pick one:

Greedy: always pick the word with the highest score. No randomness.
Sampling: pick randomly, with higher-scoring words more likely to be chosen.

This section is about the first way, where the big model always picks its top choice.

In that case, checking the small model’s guess is simple:

If the guess is the word the big model would have picked, we keep it.
If the guess is a different word, we throw it away and use the big model’s word instead.

For example, say the sentence so far is “The capital of France is”. The small model guesses “ Paris”, and the big model’s top choice is also “ Paris”. Match, so we keep it. If the small model had guessed “ Lyon”, we would replace it with “ Paris”.

Result: the final text is exactly the same as what the big model would have written alone, just faster. (This holds as long as you use the same big model and the same settings.)

Sampling

Sometimes the big model adds a bit of randomness when it picks a word. If we kept every guess from the small model just because it sounds reasonable, the final text would start to follow the small model’s habits instead of the big model’s.

So we don’t keep guesses automatically. Each guess gets a chance of being kept, based on how much the big model likes it. If a guess is rejected, the big model picks a replacement.

This way, the final text still matches what the big model would have written on its own.

In short: the small model suggests, but the big model decides.

6. How Much Faster Can It Be?

The speedup depends heavily on how often the target model accepts the draft model's proposals.

Let:

k be the number of drafted tokens per round.
alpha be the probability that an individual draft token is accepted, assuming a simplified constant acceptance rate.

Under this simplified model, the expected number of tokens produced per target-model verification pass is:

(1 − α^(k+1)) / (1 − α)

For k=4, the expected number of tokens per verification pass is approximately:

Acceptance rate	Expected tokens per pass
50%	1.9
70%	2.8
80%	3.4
90%	4.1

These numbers show how many words we get each time the big model checks the small model’s guesses. They do not show how much faster the whole thing runs.

The real speed also depends on other things: how fast the small model is, how long the checking takes, and the extra work of running two models together.

So getting more words per check doesn’t mean you get the same amount of speedup overall. To know the real gain, you have to test it on your own model, computer, and task.

7. When Does Speculative Decoding Work Best?

Speculative decoding works best when the draft model can predict the target model's next tokens accurately.

It can be especially useful for:

Code and boilerplate: Common patterns and repeated structures can be predictable.
Summaries and rewrites: Some continuations closely follow the input text.
Structured or formulaic text: Repetitive formats can make the next tokens easier to predict.
Predictable factual answers: Familiar phrases and common sequences can lead to higher acceptance rates.

It may offer less benefit for open-ended creative writing, where many continuations are plausible and the draft model may disagree with the target more often.

The task alone doesn't determine the outcome. The acceptance rate between the particular draft and target models is what matters.

8. Trying Speculative Decoding

Several inference frameworks support speculative decoding. In Hugging Face Transformers, one approach is called assisted generation, which allows a smaller assistant model to help a target model generate tokens.

A simplified example looks like this:

output = target_model.generate(
    **inputs,
    assistant_model=draft_model,
    max_new_tokens=200,
)

Here, target_model is the model whose output you want, and draft_model proposes candidate tokens for it to verify.

This is an illustrative example. The models and inputs must be loaded and configured appropriately, and the selected model combination must be supported by the Transformers version you use.

Conclusion

Speculative decoding addresses a key bottleneck in language model inference: generating tokens sequentially can repeatedly incur the cost of accessing a large model's weights.

A smaller model proposes several tokens, and the larger model verifies them together. When the proposals are accepted frequently enough, the target can produce multiple tokens per verification pass while preserving its original output distribution.

The central trade-off is simple: the more accurately the draft model predicts the target, the more opportunity there is to accelerate generation.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

What If You Could Boost or Ban Specific Tokens in an LLM?

Rijul Rajesh — Mon, 19 Oct 2026 05:23:13 +0000

Sometimes, you may want a specific word to appear more often in an LLM's output. In other cases, you may want to prevent the model from generating certain words altogether.

Prompting is one way to influence the model's choices. But there is another approach: logit bias.

Logit bias lets you adjust the likelihood of specific tokens being selected during text generation. Let's understand how it works and how to use it.

1. How a Language Model Generates Text

When you type on your phone, it suggests the next word. Language models work similarly, but on a much larger scale.

A language model generates text one token at a time. A token can be a complete word, part of a word, punctuation, or even a space combined with a word.

Suppose you ask a model to complete this sentence:

The cat sat on the ...

The model might consider several possible next tokens:

floor
mat
sofa

It calculates a score for each possible next token. These scores help determine which token the model selects.

2. How the Model Scores Its Options

Before choosing the next token, the model assigns a score called a logit to each candidate token.

For example, imagine the model produces these logits:

Token	Logit
cat	4.1
dog	3.4
fish	2.3

A higher logit means the token is more favored relative to the other candidates.

However, logits are not probabilities. A logit of 4.1 does not mean the token has a 410% probability or a 4.1% probability.

The model converts these scores into probabilities using a function called softmax.

3. From Logits to Probabilities

Softmax converts logits into probabilities that add up to 100%.

Using the example above, the model's probabilities are approximately:

Token	Probability
cat	60.9%
dog	30.2%
fish	9.0%

The model now has a probability distribution over these three candidates. Depending on its decoding settings, it can select a token from that distribution.

Two settings can influence this process: temperature and logit bias.

How temperature changes the probabilities

Temperature controls how concentrated or spread out the probability distribution becomes.

Using the same logits from the previous example, here's how different temperatures affect the probabilities:

Token	T = 0.5	T = 1	T = 2
cat	78.5%	60.9%	46.7%
dog	19.4%	30.2%	32.9%
fish	2.1%	9.0%	20.4%

At a lower temperature, the model favors the highest-scoring token more strongly. At a higher temperature, the probabilities become more evenly distributed, giving lower-scoring tokens a better chance of being selected.

Notice that the logits stay the same. Temperature changes how the model converts those logits into probabilities.

Logit bias works differently. Instead of changing the temperature, it modifies the logits of selected tokens before the probabilities are calculated.

4. What Is Logit Bias?

Logit bias lets you add a positive or negative adjustment to the logit of a specific token.

Positive bias: Increases the token's logit, making it more likely to be selected.
Negative bias: Decreases the token's logit, making it less likely to be selected.
A sufficiently strong negative bias: Can effectively prevent a token from being selected.

Let's return to our example.

Initially, the logits are:

Token	Original logit
cat	4.1
dog	3.4
fish	2.3

Suppose we apply a logit bias of +2 to dog.

Its new logit becomes:

(3.4 + 2 = 5.4)

The adjusted logits are now:

Token	Adjusted logit
cat	4.1
dog	5.4
fish	2.3

After softmax, the approximate probabilities become:

Token	New probability
cat	21.0%
dog	76.3%
fish	2.7%

The model now strongly favors dog, which previously had a lower probability than cat.

This illustrates the core idea: logit bias changes the odds of selecting particular tokens without changing the model's underlying parameters.

5. Think of Logit Bias as a Dial

Imagine each token has a dial controlling how strongly the model favors it.

Turning the dial up makes the token more likely to appear.
Turning the dial down makes it less likely to appear.
Turning it all the way down can prevent the token from being selected.

The adjustment applies to the token's logit, not directly to its final probability. Therefore, the resulting probability change depends on the logits of the other candidates too.

A positive bias does not guarantee that a token will appear. It only changes its relative likelihood.

6. Your First Logit Bias Request

The OpenAI Chat Completions API supports a logit_bias parameter that maps token IDs to bias values.

For example, in Python:

from openai import OpenAI

client = OpenAI()

response = client.chat.completions.create(
    model="YOUR_MODEL",
    messages=[
        {
            "role": "user",
            "content": "Write a short sentence about animals."
        }
    ],
    logit_bias={
        "3797": -100
    }
)

print(response.choices[0].message.content)

In this example, "3797" is an illustrative token ID, not a guaranteed ID for a particular word. You must find the correct token ID for your chosen model and tokenizer.

A bias of -100 strongly suppresses the specified token. The API accepts bias values from -100 to 100.

Before using this parameter, check the documentation for your chosen model to confirm that it supports logit_bias. Support can vary by model.

What does the token ID mean?

The API does not accept a word directly as the key in the logit_bias dictionary. It expects a token ID represented as a string.

To find token IDs, you need to tokenize the text using a tokenizer compatible with your model.

7. How to Find Token IDs

For models supported by the tiktoken library, you can inspect token IDs with Python:

import tiktoken

enc = tiktoken.encoding_for_model("YOUR_SUPPORTED_MODEL")

print(enc.encode(" cat"))

The output contains the token IDs corresponding to the input text.

Replace the example model name with a model supported by tiktoken. Do not assume that every model uses the same tokenizer or that a token ID from one model will work for another.

Also, pay attention to the exact text you encode. For example, these strings may tokenize differently:

"cat"
" cat"
"Cat"
"cats"

A word may consist of multiple tokens, and different capitalization or spacing may produce different token IDs.

8. Where Can You Use Logit Bias?

Logit bias is useful when you want more control over the vocabulary a model uses.

Constraining possible answers

Suppose you're building a classifier that should respond with positive, negative, or neutral.

You can use prompting to request those labels and potentially use logit bias to discourage unwanted alternatives.

However, logit bias alone does not guarantee valid output. For strict structured responses, use an appropriate constrained-output or structured-output feature when available.

Banning specific tokens

Suppose you don't want a model to use a particular token. A strong negative bias can suppress it.

But there is an important limitation: suppressing one token does not necessarily prevent the model from generating the same word through a different tokenization.

Reducing common clichés

Suppose a model repeatedly uses words such as "delve" or "tapestry" in its answers.

You could apply a negative bias to the relevant token IDs to make those choices less likely.

This is a form of vocabulary steering. It is not a perfect word filter, because a word may have several tokenization variants or appear as part of a larger token.

Steering vocabulary

You can also use positive bias to encourage certain vocabulary. For example, a writing assistant could make selected technical terms more likely to appear.

Still, positive bias does not guarantee that a word will appear. The model must also consider the surrounding context and its other candidate tokens.

For important requirements, combine logit bias with clear prompting and appropriate validation.

9. The Tricky Part: Words Aren't Always Single Tokens

One of the most important details about logit bias is that it operates on tokens, not complete words.

Consider the word cat. Depending on the tokenizer and context, it might be represented differently from cat or Cat. Other words may be split into multiple tokens.

This creates a limitation when you want to ban a word completely.

If you suppress one token ID, you may only suppress one tokenization variant. The model might still generate the word using another sequence of tokens.

Similarly, applying a positive bias to one token does not guarantee that the complete word will appear.

Logit bias provides token-level control, not reliable word-level enforcement.

When you need to enforce a strict rule, use a mechanism designed for that purpose rather than relying on logit bias alone.

10. Choosing the Right Bias Value

The API accepts bias values between -100 and 100, but the right value depends on your use case and the model's behavior.

A practical approach is to start with a small adjustment and test the results:

Small positive or negative values: Gently influence token selection.
Larger adjustments: Apply stronger pressure toward or away from a token.
Extreme negative values, such as -100: Strongly suppress the selected token.

These are general guidelines, not universal thresholds. A bias value does not correspond to a fixed percentage-point change in probability.

Test your settings with representative prompts. Check whether the model follows the intended behavior and whether the adjustment introduces unwanted side effects.

Conclusion

Prompting lets you tell a language model what you want. Logit bias gives you another way to influence its choices by adjusting the scores of specific tokens.

It can help steer vocabulary, discourage certain tokens, or make selected outputs more likely. But because it operates at the token level, it cannot reliably enforce every word-level or content-level rule.

Understanding that distinction helps you use logit bias where it is useful, while choosing more suitable controls when you need stronger guarantees.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

Temperature Controls Randomness. But Why Does That Matter More for Agents?

Rijul Rajesh — Sat, 17 Oct 2026 18:48:04 +0000

When you type a message to an AI, it doesn't write the entire reply at once. It generates one small piece of text (a "token") at a time, choosing the next token based on the probabilities it assigns to possible options.

Suppose the model needs to choose its next word. It might assign probabilities like these:

Next word	Probability
"search"	70%
"calculate"	26%
"email"	4%

The model then samples from this distribution, like rolling a weighted die. "Search" is the most likely choice, but "calculate" and "email" are also possible.

This sampling process introduces randomness, which is one reason the same question can produce different answers.

What Does Temperature Do?

Temperature controls how sharply or evenly the model distributes probabilities across its possible next tokens.

Technically, it rescales the model's raw scores, called logits, before converting them into probabilities. This changes how likely each token is to be selected.

Let's look at how different temperature settings can affect the same choices.

Next word	T = 0.5	T = 1	T = 2
"search"	88%	70%	55%
"calculate"	12%	26%	33%
"email"	0.2%	4%	12%

These values are illustrative. Actual probabilities depend on the model's raw scores and the temperature setting.

Low temperature: The most likely options become more dominant. The model tends to produce more predictable output.
High temperature: The probabilities become more evenly distributed, giving less likely options a better chance of being selected. The output can become more varied and surprising.
Temperature near zero: The model increasingly favors the highest-probability token. At zero, many implementations use greedy decoding, which selects the top token directly.

Think of temperature as a dial that controls how much randomness influences the model's choices.

How Temperature Affects a Chatbot

In a chatbot, you ask a question, the model generates an answer, and you read the result. Temperature influences how that answer is phrased and which ideas it explores.

Low temperature: Answers tend to be more consistent, focused, and predictable.
High temperature: Answers can be more varied and creative, but may also become less focused or more prone to errors.

If an answer is slightly off, you can notice it and ask again. A human is usually checking the result. The cost of a poor choice might be an awkward sentence or an irrelevant suggestion.

Now consider what happens when the same model operates as an agent.

How an AI Agent Works

An AI agent typically follows a loop:

Observe the current situation.
Decide what to do next, such as searching the web, running code, calling an API, or editing a file.
Execute the action.
Examine the result.
Repeat until the task is complete.

A single task might involve 10, 20, or even 100 steps. Some steps may run without a human checking each decision.

The model still chooses tokens, but those tokens can now determine actions that influence everything that follows. This makes temperature an important setting to consider.

Why Temperature Matters More in Agents

1. Small Errors Can Compound

Suppose an agent completes a 20-step task, and each step has a 98% chance of being correct. If we assume the steps are independent and every step must be correct, the probability that all 20 steps succeed is:

0.98 ** 20  # approximately 67%

At 95% reliability per step, the probability falls to about 36%. At 99%, it is about 82%.

This illustrates how reliability can decrease across a long sequence of steps.

Higher temperature does not automatically make every step less reliable. However, when it increases the chance of selecting unsuitable tokens or actions, those mistakes can affect later steps. An incorrect early decision might send the agent down the wrong path, making the rest of the task harder to complete.

In a chatbot, a poor choice might produce an awkward sentence. In an agent, it could derail an entire workflow.

2. Agents Often Produce Structured Output

Chatbots primarily produce text for humans to read. Agents often generate structured outputs that software must interpret, such as tool calls, JSON objects, and API parameters.

For example:

{
  "tool": "search_files",
  "query": "invoice_2026.pdf",
  "max_results": 5
}

A computer processes this output according to strict rules. A misspelled tool name, an invalid parameter, or malformed JSON can cause a call to fail. Depending on the system, an incorrect but valid call could also perform an unintended action.

Structured output often leaves less room for variation than ordinary prose. This makes reliable generation especially important when an agent interacts with tools.

3. Agent Actions Have Real Consequences

A chatbot's response is usually text on a screen. An agent can act on that response by sending an email, deleting a file, running a command, or spending money.

An unsuitable choice is no longer just an odd sentence. It might trigger an action that is difficult to reverse.

Lower temperature can help make token selection more predictable, but it is not a safety mechanism by itself. Agents still need safeguards such as permission limits, input validation, human approval for sensitive actions, and checks before executing destructive operations.

4. Debugging Requires Repeatability

Suppose an agent fails on step 14 of a 30-step task. You want to run it again, reproduce the failure, and identify what went wrong.

At higher temperatures, sampling may send the agent down a different path on each run. The failure might disappear, making it harder to investigate. Lower temperatures can make the agent's behavior more repeatable, which helps when testing changes and comparing results.

However, repeatability also depends on factors such as the model, the tools it uses, the environment, and the state of external systems. Lower temperature alone cannot guarantee identical runs.

Should You Always Set an Agent's Temperature to Zero?

Not necessarily. Although predictable behavior is useful for agents, setting temperature to zero is not a universal solution.

There are situations where variation is valuable:

Getting stuck in a loop: An agent might repeatedly choose the same failed action. Exploring an alternative approach may help it recover.
Exploring different solutions: Tasks such as finding information, brainstorming search queries, or generating candidate solutions can benefit from variation.
Creative tasks within a workflow: An agent writing a draft or proposing ideas may benefit from a moderate temperature, even if other steps require more predictable behavior.

The right setting depends on the task. A workflow can also use different temperatures for different stages, favoring predictable outputs for tool calls and allowing more variation during brainstorming.

Conclusion

Temperature controls how randomness influences a language model's token choices. In a chatbot, this mainly affects the variety and predictability of its responses. In an agent, those choices can influence a chain of tool calls and real-world actions.

That is why temperature deserves more attention in agent workflows. Small differences in individual decisions can affect the outcome of a longer task, while unpredictable behavior can make failures harder to reproduce and debug.

The goal isn't to eliminate randomness everywhere. It's to control where variation is useful and where predictable behavior matters more.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

AI Models Can Learn from Data. But What Can They Learn from Other Models?

Rijul Rajesh — Fri, 16 Oct 2026 05:23:40 +0000

Imagine you've trained a huge AI model that is very good at its job, such as recognizing objects in photos or answering questions.

But there's a problem. The model is enormous. It needs powerful servers, costs a lot to run, and may be too slow or too large for a phone, smartwatch, or car.

You want a smaller model that performs almost as well but runs faster and costs less.

One option is to train a small model from scratch using your original data. However, it may not reach the same level of performance as the larger model.

Knowledge distillation offers another approach. Instead of learning only from the original data, the smaller model also learns from the larger model, much like a student learning from a teacher rather than studying a textbook alone.

Let's see how it works.

1. The Teacher and the Student

Think about two ways of learning mathematics.

In the first, you get a textbook containing problems and their final answers.

In the second, you also have a tutor who explains how they approach each problem. The tutor might show you that one problem resembles another, even though the answers are different.

The second approach gives you additional information that can help you learn.

Knowledge distillation applies a similar idea to AI models:

Teacher model: The larger model that already performs well.
Student model: The smaller model that learns from the teacher.

The teacher provides additional learning signals that help the student reproduce some of its capabilities.

2. Learning from More Than the Correct Answer

Let's say we have an AI model that classifies animal photos into three categories:

When the model receives a photo of a cat, it produces a score for each category. These scores are called logits. A function called softmax can convert them into probabilities that add up to 100%.

For example, the teacher might produce these probabilities:

Class	Probability
Cat	90%
Dog	9%
Fox	1%

The teacher predicts cat because it assigns the highest probability to that class.

But notice something else: it considers dog more likely than fox.

Those smaller probabilities provide additional information about how the teacher distinguishes between the categories.

Hard labels vs. soft labels

In ordinary supervised classification, the training target might be a hard label:

Cat = 100%, Dog = 0%, Fox = 0%

This tells the model which category is correct, but it doesn't directly communicate how the other categories compare.

The teacher's output, on the other hand, gives us a distribution:

Cat = 90%, Dog = 9%, Fox = 1%

These are called soft labels. They provide information about the teacher's relative confidence across all three categories.

The teacher is effectively saying, "This is a cat. Dog is less likely, but fox is even less likely."

This information can help the student learn distinctions that a simple correct-or-incorrect label doesn't express.

Geoffrey Hinton and his colleagues popularized the term dark knowledge for the useful information contained in a teacher's output probabilities, particularly the information in the less likely alternatives.

The key idea is:

Train the student to learn from the teacher's full probability distribution, not just the correct answer.

3. Making the Hidden Information Easier to See: Temperature

There's a problem with the example above.

Imagine the teacher is extremely confident and produces these probabilities:

Class	Probability
Cat	99.9%
Dog	0.09%
Fox	0.01%

The probabilities for dog and fox are tiny. The difference between them may provide a useful learning signal, but the teacher's output is dominated by the cat prediction.

This is where temperature comes in.

Temperature, usually represented by (T), controls how soft or sharp the probability distribution becomes when softmax is applied to the teacher's logits.

T = 1: The usual softmax distribution.
Higher temperature: The probabilities become more evenly distributed, making the differences between less likely classes easier to observe.
Lower temperature: The distribution becomes sharper, giving more weight to the highest-scoring class.

For example, with a higher temperature, the teacher might produce a distribution like this:

Class	Illustrative probability
Cat	70%
Dog	22%
Fox	8%

These numbers are illustrative, not a guaranteed result of a particular temperature.

The important point is that the ranking remains the same, while the probability distribution becomes softer.

The student can now receive a more informative signal about how the teacher relates the different classes.

4. How the Student Actually Learns

The student needs to learn two things:

How to imitate the teacher.
How to predict the correct answers from the original training data.

During training, these goals can be combined into a single loss function.

A. Match the teacher

First, the student produces its own probability distribution using the same temperature.

The training process compares the student's softened probabilities with the teacher's softened probabilities.

One way to measure this difference is KL divergence. It measures how different two probability distributions are.

The student is trained to reduce this difference, making its outputs more similar to the teacher's.

B. Match the correct labels

The student is also trained using the actual labels in the dataset.

For example, if the image contains a cat, the student should still learn that cat is the correct category.

This is commonly done using cross-entropy loss, a standard method for training classification models.

C. Combine both learning signals

The two losses can be combined using a weighted sum:

Distillation loss: Encourages the student to imitate the teacher.
Label loss: Encourages the student to predict the correct answers.

The weights determine how much each objective contributes to training.

This combination helps the student learn from both the teacher's behavior and the original training labels.

5. The Knowledge Distillation Process

Here's the overall workflow:

Prepare the teacher: Train a large model or use an existing one.
Choose the student: Select a smaller model architecture suitable for your use case.
Prepare the training data: Run the examples through the teacher to obtain its outputs.
Train the student: Use the teacher's outputs and the original labels to calculate the training loss.
Deploy the student: Once training and evaluation are complete, use the smaller model for inference. The teacher usually isn't needed during deployment.

The result is a student model that aims to retain as much of the teacher's useful behavior as possible while requiring fewer resources.

6. Does Knowledge Distillation Really Work?

Yes. One well-known example is DistilBERT, a smaller version of BERT developed by Hugging Face.

According to its creators, DistilBERT is approximately 40% smaller and 60% faster than BERT while retaining around 97% of its language-understanding capabilities on the evaluated benchmarks.

These figures describe that particular model and evaluation setup. Results vary depending on the teacher, student architecture, training data, and task.

Knowledge distillation is also relevant to large language models.

A smaller language model can be trained using outputs generated by a larger model. This is one form of learning from a teacher, although the exact methods differ from probability-based distillation used in classification.

For example, a larger model might generate answers to a collection of questions. Those answers can then be used as training examples for a smaller model.

This can help transfer some of the larger model's capabilities into a model that is cheaper to run.

However, distillation doesn't guarantee that the student will match the teacher. The quality of the training data, the student architecture, and the distillation method all matter.

Conclusion

Knowledge distillation is a way to transfer useful behavior from a larger AI model to a smaller one.

Instead of learning only from the correct answers, the student can also learn from the teacher's output probabilities or other signals. This gives the student additional information about how the teacher approaches a task.

The goal is to preserve as much useful performance as possible while reducing the cost and resources needed to run the model.

In simple terms, knowledge distillation helps a smaller model learn to do more with less by learning from a larger model.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

OpenCode Can Write Code. What If It Could Use Your Browser Too?

Rijul Rajesh — Wed, 14 Oct 2026 05:12:41 +0000

We use AI agents for all sorts of tasks. Some agents can even interact with your browser and act as companions while you work.

As someone who uses OpenCode, I wanted to bring that capability into my workflow. I wanted OpenCode to interact with my browser so it could access the relevant browser context while helping me with tasks.

In this article, let's see how to connect your browser to OpenCode using Browser MCP.

1. What You Need

To set this up, you need two components:

Browser extension: Connects your browser to the MCP server.
Local MCP server: Allows your AI client to interact with the browser through the Model Context Protocol (MCP).

Let's set them up.

2. Install the Browser Extension

First, visit the following page to install the Browser MCP extension:

https://browsermcp.io/install

After installing the extension, you'll be directed to a page with more information.

This page links to the official Browser MCP documentation, where you can find additional setup instructions and details about how it works:

https://docs.browsermcp.io/

3. Add the MCP Server to OpenCode

Next, configure OpenCode to use the Browser MCP server.

Open your OpenCode configuration file and add the following configuration:

    {
      "$schema": "https://opencode.ai/config.json",
      "mcp": {
        "browsermcp": {
          "type": "local",
          "command": ["npx", "-y", "@browsermcp/mcp@latest"]
        }
      }
    }

This configuration tells OpenCode to start the Browser MCP server locally using npx.

Save the configuration after adding it.

4. Connect and Test the Browser

Once you've configured the MCP server, make sure Browser MCP is connected.

Now, let's test whether it works.

Go to OpenCode and give it a task that requires browser interaction. For example:

    Go to the Wikipedia page for "Open-source software" and list the main section headings.

The Browser MCP server allows OpenCode to interact with the browser and navigate to the requested page.

Here's the result returned by the agent.

As you can see, OpenCode can use the browser to complete the task instead of relying only on the information already available in the conversation.

Conclusion

That's how you can connect Browser MCP to OpenCode.

This setup lets you bring browser interaction into your AI-assisted workflow. Instead of working with your AI agent and browser separately, you can give OpenCode tasks that involve navigating web pages and gathering information through the connected browser.

It can be useful when you want your AI coding assistant to work with relevant information from the web alongside your existing workflow.

That's it for this article. See you in the next one!

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Risk-Aware AI Code Review for Business-Critical Systems

LiveReview: Risk-Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by tests…

View on GitHub

Click below to try LiveReview with your codebase:

AI Generates Text One Token at a Time. But How Does It Know When to Stop?

Rijul Rajesh — Tue, 13 Oct 2026 03:18:29 +0000

A language model generates text one token at a time. But how does it know when to stop?

It doesn't understand completion in the same way a human does. Instead, it learns patterns that help it predict when a response should end. Special tokens, instructions, and generation settings all play a role in this process.

Let's explore how it works.

1. How a Model Generates Text

Each time a language model generates text, it predicts the next token based on the tokens that came before it.

A token can represent a word, part of a word, punctuation, or another piece of text. The model assigns probabilities to possible next tokens and selects one according to its generation settings.

One possible token is a special marker that signals the end of generation. This is commonly called an end-of-sequence (EOS) token. Depending on the model, it might look like </s> or <|endoftext|>.

When the model generates an EOS token, generation normally ends. In chat models, a separate end-of-turn token may serve this purpose by indicating that the assistant has finished its response.

2. How a Model Learns to Stop

During training, models learn to predict the next token from examples of text. Special tokens are included in the training data to mark boundaries, such as the end of a sequence or a conversational turn.

For example, a training sequence might look like this:

User: What is the capital of France?
Assistant: Paris. [end-of-turn token]

By learning from many examples, the model learns when these ending tokens tend to appear.

After generating a complete answer, the model may assign a high probability to an ending token. However, the model does not explicitly verify that it has answered the question correctly or included every necessary detail. It predicts the next token based on patterns learned during training.

The exact behavior depends on how the model was trained and which ending tokens it uses.

3. What Influences When a Model Stops?

Several factors influence whether a model generates another token or ends its response.

Context

The model uses the preceding text to predict what should come next. If the response resembles patterns associated with a completed answer, an ending token may become more likely.

However, a response can appear complete to a reader without the model actually selecting an ending token.

Instructions

Instructions such as "Answer in one sentence" or "Keep your response under 100 words" can influence the length of the generated response.

Fine-tuning methods, including reinforcement learning from human feedback (RLHF), can also shape how a model responds and when it tends to finish.

These factors influence the model's behavior, but they do not guarantee a particular response length.

Sampling settings

Generation settings affect how the model selects its next token.

Temperature: Adjusts how strongly the model favors higher-probability tokens over lower-probability ones.
Top-p: Restricts sampling to a set of tokens whose cumulative probability reaches a specified threshold.

These settings can affect whether an ending token is selected, particularly when it competes with other plausible next tokens.

4. External Conditions That Stop Generation

Not every response ends because the model generates an ending token. The application running the model can also stop generation.

Common external stopping conditions include:

Maximum token limit: The application sets a maximum number of tokens the model can generate. Once the limit is reached, generation stops even if the response is incomplete.
Stop sequences: The application can specify text sequences that trigger a stop when detected. For example, an application might stop when it encounters "\n\nUser:".
Context window limits: The model can only process a limited amount of text at once. If the available context or generation capacity is exhausted, the application may be unable to continue generating.

These mechanisms are different from the model selecting an EOS token. They are stopping conditions imposed by the model's runtime or application.

5. Common Stopping Problems

Understanding the difference between model-generated endings and external stopping conditions helps explain several common problems.

Rambling or repetitive responses

The model continues generating because it keeps predicting additional tokens instead of selecting an ending token. It may repeat earlier points, introduce unnecessary details, or continue beyond what the user needs.

Premature stopping

The model generates an ending token before fully answering the question. This can happen even when the response is incomplete because the model predicts that the sequence should end.

Truncation

The application reaches its maximum generation limit and stops the response. The output may end mid-sentence, making it appear as though the model decided to stop on its own.

The distinction matters: premature stopping comes from the model's generated ending, while truncation occurs because an external limit was reached.

Conclusion

A language model does not have a reliable, human-like understanding of when an answer is complete. It learns patterns that help it predict when to generate an ending token, while instructions and sampling settings influence its behavior. External limits can also stop generation independently of the model's predictions.

This distinction matters when building AI applications. If a model stops too early, users may receive incomplete answers. If it continues for too long, it can increase token usage, latency, and cost.

Understanding these mechanisms helps developers configure generation settings, choose appropriate token limits, and diagnose whether a response ended because the model generated an ending token or the application stopped it.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Risk-Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by tests…

View on GitHub

Click below to try LiveReview with your codebase:

Jev Can Make Decisions. But How Do You Work With It?

Rijul Rajesh — Sat, 10 Oct 2026 21:04:19 +0000

In the previous article, we explored the basic concepts behind Jev and its three decision types: Choice, Score, and Noul.

Now, let's look at how to work with Jev more effectively. We'll cover the kind of input it accepts, how to structure a request, how multiple questions are evaluated, and how to design Choice questions that avoid forced or misleading answers.

1. Understanding State

In Jev, state is the information the model uses to evaluate your questions.

A state can be provided in three formats:

String: Plain text, such as a customer message or incident report.
JSON object: Structured information containing fields such as an order ID, customer status, or error message.
Array: A collection of context items that provide information for a decision.

For example, a customer support application might provide this state:

{
  "customer_message": "My payment failed, but I was charged twice.",
  "account_status": "active",
  "previous_contact": "No previous complaints"
}

Jev can use this information to answer questions about the customer's issue, account, or next steps.

However, Jev is a text-based model. It does not directly accept images, audio, video, or other binary inputs as state. If your application receives these formats, you need to extract relevant information into text or structured data before sending it to Jev.

2. Understanding Request Anatomy

A Jev request consists of three main components:

model: The Jev model you want to use.
state: The information the model should evaluate.
questions: A map of named questions you want answered.

Each question has its own identifier and definition. The identifier helps your application locate the corresponding answer in the response.

Here's an example:

{
  "model": "jev-latest",
  "state": {
    "customer_message": "My payment failed, but I was charged twice."
  },
  "questions": {
    "issue_type": {
      "type": "choice",
      "instructions": "What is the primary issue?",
      "criteria": {
        "billing": "Payment failures, duplicate charges, or refunds",
        "technical": "Application errors or technical failures",
        "account": "Account access or account management issues",
        "other": "The issue does not fit any of the other categories"
      }
    },
    "needs_follow_up": {
      "type": "noul",
      "instructions": "Does this customer need a follow-up from support?"
    }
  }
}

In this example, the request contains two questions about the same customer message.

The first uses Choice to classify the issue. The second uses Noul to evaluate whether follow-up is needed.

Notice that each question has its own type and instructions. Choice uses a criteria field to define the available options. Score uses criteria to define an ordered scale, while Noul can evaluate a yes-or-no question without requiring criteria.

This structure lets you define multiple decisions in one request rather than making a separate request for every question.

3. Parallel, Independent Evaluation

Jev can evaluate multiple questions in the same request. However, each question is evaluated independently against the shared state.

This means one question does not receive another question's answer as additional context.

Consider an AI agent that needs to process a customer support ticket. You might want it to:

Identify the issue type.
Determine its severity.
Decide whether it needs human intervention.

You can ask all three questions in one request. But the severity question cannot use the answer from the issue classification question, and the escalation question cannot inspect the severity answer.

Each question evaluates the original state independently.

Why does this matter?

Suppose your application should escalate a ticket when its severity reaches the high-severity threshold.

You cannot assume that asking Jev to classify the issue and score its severity will automatically connect those answers. Instead, your application needs to combine the returned values and apply the escalation rule.

For example, if your Score scale defines 2 as the severe level:

# Example scale: 0 = minor, 1 = moderate, 2 = severe
if severity >= 2:
    escalate_to_human()

Here, the application makes the final decision using the returned score. If the scale allows values between defined levels, establish and test the threshold against representative examples before using it in production.

If a later question genuinely depends on an earlier answer, use a second request with the relevant result included in its state. Alternatively, combine independent answers in your application code when a second model evaluation is unnecessary.

The key principle is to separate evaluating individual questions from combining their answers into a workflow.

4. Designing Better Choice Questions

Choice is useful when you need to select one answer from a predefined set of options.

However, the quality of the result depends partly on how you define those options.

Consider this example:

{
  "type": "choice",
  "instructions": "What is the primary issue?",
  "criteria": {
    "billing": "Payment failures or duplicate charges",
    "technical": "Software bugs or application errors",
    "account": "Login or account access problems"
  }
}

What happens if a customer asks about a shipping delay?

None of these options fits the issue. Without a suitable fallback, the model may still select one of the available options, producing a misleading classification.

Include an "other" or "none" option

When your predefined choices might not cover every possible input, include a fallback option.

For example:

{
  "type": "choice",
  "instructions": "What is the primary issue?",
  "criteria": {
    "billing": "Payment failures or duplicate charges",
    "technical": "Software bugs or application errors",
    "account": "Login or account access problems",
    "other": "The issue does not fit any of the categories above"
  }
}

Now, the application has an option for issues that fall outside the expected categories.

This is especially useful when processing real-world data, where customer messages and other inputs may not always fit neatly into predefined classifications.

Make the options distinct

A fallback option is only one part of good question design. The other options should also be clearly distinguishable.

For example, if both billing and account include subscription problems, Jev may have difficulty distinguishing between them.

Define each option in terms of what it covers, and avoid unnecessary overlap. If two options represent different outcomes, their descriptions should make that difference clear.

Remember that Choice selects from the options you provide. It does not automatically expand the list when an unexpected case appears.

Wrapping Up

Working with Jev involves more than choosing between Choice, Score, and Noul. You also need to structure the state, define each question clearly, and understand how the answers relate to one another.

Keep these principles in mind:

Provide the relevant context through a string, JSON object, or array of context items.
Give every question clear instructions and the appropriate criteria.
Treat questions in the same request as independent evaluations.
Combine their answers in your application when a workflow requires multiple decisions.
Include an other or none option when your Choice categories may not cover every case.

With these basics in place, you can build more controlled decision workflows around Jev while keeping the application logic under your control. Clear instructions and well-defined options can help, but they do not guarantee correct decisions. Evaluate your workflow against labelled examples before deployment, and continue monitoring its performance as real-world inputs change.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Risk-Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by tests…

View on GitHub

Click below to try LiveReview with your codebase:

AI Can Generate Paragraphs. But What If You Just Need a Decision? Meet Jev

Rijul Rajesh — Fri, 09 Oct 2026 17:56:47 +0000

There’s a new AI model that’s been getting people talking, and you might have heard of it. It’s called Jev.

But what exactly is Jev, and what makes it different from the general-purpose LLMs we use every day?

Let’s take a look at what Jev is all about.

Jev is an AI model from TypeSafe AI designed to make structured decisions rather than generate long-form text.

To understand why it exists, let's start with a simple problem.

Imagine you're building an application that receives 100,000 customer messages. For every message, your application needs to decide:

Is this a billing problem?
Is this a technical problem?
Is this an account problem?

You could send every message to a general-purpose LLM like GPT and ask it to classify each message. But you're using a model capable of writing essays and generating code just to select one of three options.

Jev is designed for this narrower kind of task: making decisions based on the information provided.

Three Things Jev Can Answer

Jev provides three types of decisions called Choice, Score, and Noul.

1. Choice

Which option is correct?

Input: "My payment failed, and I was charged twice."
Options: billing, technical, account
Example output: billing

This is useful when you need to classify information into a predefined set of options.

2. Score

How well does this fit a defined scale?

Input: "The production service is down for every customer."
Scale: 0 = minor, 1 = moderate, 2 = severe
Example output: 2

This is useful when you need to evaluate something against a predefined scale.

3. Noul

How likely is a particular statement to be true?

Input: "The production service is down."
Question: "Does this incident require immediate escalation?"
Example output: 0.95, meaning a 95% estimated probability of a positive answer, assuming the output represents a calibrated probability.

This is useful when you need a structured answer to a specific question rather than a long explanation.

Take Jev for a Spin

The fastest way to try it is through the Jev Playground.

Go to https://jevplayground.com/.

It's an independent playground where you can experiment with Jev.

Choice

Score

Noul (Boolean)

Wrapping Up

So, that's Jev.

For decision-based tasks in an AI agent, Jev could be an option to consider instead of relying entirely on general-purpose LLMs. Using a model designed for structured decisions can help diversify your setup and may make specific decision-making tasks more efficient.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by…

View on GitHub

Click below to try LiveReview with your codebase:

What Makes a 3D Model Move? Understanding Animation Basics in Blender

Rijul Rajesh — Wed, 07 Oct 2026 06:56:53 +0000

In this article, we will explore how you can animate anything in Blender.

Before that, we need to understand an important concept called keyframes.

What Is a Keyframe?

Suppose you want to animate a cube by moving it from point A, to point B, and then to point C.

There are three points where we want the cube to move.

In software like Blender, we can set these positions as keyframes.

When the animation is played, Blender calculates the movement between these keyframes, creating the in-between animation as the cube moves from point A to point B and then to point C.

Let's start keyframing.

How to Add a Keyframe in Blender

First, select the object you want to animate. For this simple example, I will select a cube.

Now press I to insert a keyframe.

Once you have inserted it, you can see the keyframe in the timeline below.

Creating a Simple Moving Animation

Let's start with a simple moving animation.

I want to animate the cube moving along the X axis.

In the timeline, you can see that the first keyframe is placed at the 0th frame.

My goal is to make the cube move towards the right by the end of the 10th frame.

I have moved the playhead to the 10th frame.

Now go to the cube, press G, and then X to move it along the X axis.

If you notice, on the right, you can see Location X highlighted in orange.

This means the change has not been added as a keyframe yet.

So let's apply it by pressing I.

Now the keyframe is added, and the orange turns yellow, indicating that a keyframe has been added.

Let's play our newly created animation.

Move the playhead back to the beginning and press play.

The block has come to life.

Animating a Rigged Model

Now let's try animating a rigged model.

I rigged this model in my previous article.

I want to make him wave.

So I select his lower arm and add a keyframe.

Now, after 10 frames, I want to move his arm a bit.

So I select the bone and rotate it after 10 frames.

After that, I have to mark the keyframe by pressing I.

Again, I need to create a back-and-forth effect. In the 20th frame, I rotate his arm to the rightmost position.

One back-and-forth movement is done.

If I want to move his hand back to the left, I don't need to rotate it manually. I can simply copy the keyframe from frame 10 and paste it into frame 30.

Let's play the animation now.

Wrapping Up

So this is the basic idea behind animation in Blender.

We alter the position, rotation, scale, or other properties of an object, mark the changes as keyframes in the timeline, and Blender creates the animation between them.

See you in the next article.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by…

View on GitHub

Click below to try LiveReview with your codebase:

How Do You Bring a 3D Character to Life? 6 Steps to Rigging in Blender

Rijul Rajesh — Tue, 06 Oct 2026 05:55:32 +0000

In this article, I'll show you how to rig a character model in Blender.

You can imagine rigging like adding a skeleton inside a character. The bones give you something to control so you can move different parts of the model and eventually animate it.

So, rigging is essential when you want to move parts of a character around or animate it.

I'm starting with a simple character model here.

Add the First Armature

First, we need to add an armature.

Press Shift + A to add a new object.

Then click Armature.

We have added a bone now.

I want the bone to always be visible, so click the bone, go to Data, open Viewport Display, and check In Front.

Understanding a Bone

A bone has three main parts:

Head
Body
Tail

Now let's move this bone into the torso.

Press Tab to go into Edit Mode.

Select the bone and press S to scale it down.

Then press G to move it up.

The bone now becomes the first bone in the torso.

Building the Spine

Let's add the second bone.

Select the tail of the bone and press E to extrude.

Now let's add the neck.

Press E to extrude again.

Repeat the same process for the head.

Adding the Arms and Hips

Now let's add the arms.

Extrude the bones like before, and you can get something like this.

Let's add the hips now.

We now have the bones on one side, but we need the same structure on the other side.

Click this button to apply the changes to the matching bones on the opposite side of the axis.

Naming the Bones

We also need to name our bones properly.

A common naming convention is:

bone_name.l

Here, .l stands for left.

Select the bone, press F2 to rename it, and follow the naming convention.

After renaming the bones, select the entire armature by pressing A.

Then right-click and press Symmetrize.

We now have the same changes on the other side.

Adding the Legs

Now let's create the legs just like before.

Switch between the views to get better positioning.

Parenting the Model

We now need to attach the bones to the body.

Switch to Object Mode.

First, select the body, and then select the bones.

Press Ctrl + P.

Then click With Automatic Weights.

This tells Blender to automatically determine how the character's mesh should move when we move the bones.

Time to Pose

Now let's see if our rig actually works.

Click the armature and switch to Pose Mode.

You can use Ctrl + Tab to switch modes.

Now select any bone and change the pose.

For example, select the upper arm, press R, and rotate it.

You can now try out different poses with the rig.

And with that, our basic rigging is complete.

You can pair this up with animations as well.

See you in the next article.

Your team's attention is limited, and the deluge of AI-generated code is making it harder to keep production reliable and secure without slowing you down.

I'm building LiveReview, a blast-radius aware AI code review built for your business-critical systems.

Instead of presenting every diff with equal emphasis, LiveReview scores each change by blast radius — how far its impact reaches through your call graph — so you can focus attention where it actually matters.

Spend code review effort where business risk is highest — not spread evenly across every diff.

⭐ Star it on GitHub:

HexmosTech / LiveReview

Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview: Blast-Radius Aware AI Code Review for Business-Critical Systems

LiveReview is an AI code reviewer that scores every hunk of a diff by blast radius: how far a change reaches through your call graph, how much persistent state it touches, and how well-tested it is. A 3-line change to a shared auth check can outrank a 300-line UI tweak. Your team's attention goes to the highest-risk code first, not spread evenly across every diff.
blast-radius-demo.mp4
LiveReview's Blast Radius & Review Priority scoring, live in the diff viewer.

The exact math, not a black box Visualize blast radius at a glance Every factor that feeds the score

How does Blast Radius scoring work? (a more technical explanation)
Here's the goal:

A 3-line fix in a function used by 40 other files, that also writes to a database, should score high.

A 300-line UI change in one file, fully covered by…

View on GitHub

Click below to try LiveReview with your codebase: