DEV Community: NaveenKumar Namachivayam ⚡

Stop Your AI Coding CLI From Wasting Tokens on "Hi" and "Thanks"

NaveenKumar Namachivayam ⚡ — Wed, 05 Aug 2026 17:23:35 +0000

In this blog post, we will see how a small Python script called Pleasantries can stop your AI coding CLI from burning a full model call every time you type "hi", "ok", or "thank you". I built this after noticing how often my own prompts to Claude Code and Qwen Code started with a greeting out of pure habit, and how each one quietly cost tokens and time for zero task value.

Pleasantries is a lightweight Python pre-hook script that intercepts greeting-only prompts before they reach an AI coding CLI model. It uses regex fullmatch logic to block inputs like "hi," "thank you," or "ok" while allowing prompts that contain pleasantry words but carry a real task.

The tool supports 15 AI coding CLIs, including Claude Code, Gemini CLI, and Cursor, and installs via a single Python script with no external dependencies. Users can customize the blocklist and add new CLI adapters with minimal effort.

1. Why I built Pleasantries

Pleasantries is a pre-hook script for AI coding CLIs. It blocks pleasantry-only prompts like "hi", "hello", "ok", and "thank you" before they ever reach the model.

Here is the observation that started this. AI coding assistants are task tools, not chat buddies. Every "hello" still triggers a full round trip: the CLI reads it, sends it to the model, and the model replies. That is a wasted call, wasted tokens, and a small break in your flow, all for a prompt that carries no actual task.

I spend a lot of my time thinking about efficiency, whether that is a load test or a model call, so this felt like an easy win to automate away.

2. How it works

The idea is simple. A hook intercepts your prompt before it reaches the model:

You type "hi" --> Hook reads prompt --> Regex fullmatch --> Blocked
You type "fix auth bug" --> Hook reads prompt --> No match --> Prompt proceeds

Under the hood, the matcher normalizes your input (lowercase, strip punctuation, collapse whitespace) and checks if the entire prompt is a pleasantry using a fullmatch, not a partial match. That distinction matters a lot. Prompts that contain a pleasantry word but also carry a real task pass through untouched.

Each CLI has its own way of blocking a prompt once the hook decides to reject it, as shown below:

Block method	CLIs
stderr + exit code 2	Claude, Kiro, Copilot Chat, Copilot CLI, Cursor, Factory Droid, Kimi Code, Devin
JSON `{"decision": "block"}` + exit 0	Codex
JSON `{"decision": "deny"}` + exit 0	Gemini

3. Which AI coding CLIs are supported

Pleasantries currently hooks into 15 CLIs, including Claude Code, Codex CLI, Gemini CLI, Cursor, Copilot Chat, Copilot CLI, Qwen Code, Junie CLI, Factory Droid, Kimi Code, grok-cli, Kun, Open Interpreter, Kiro, and Devin CLI.

A few tools do not have a hook system to plug into yet, so they are not supported: CodeBuddy, OpenCode, Aider, Kilo Code, Trae, Hermes, Pi, OpenClaw, Amp, Google Antigravity, Cline, oh-my-pi, Freebuff, and Command Code.

4. A quick example

Say you fire up Claude Code and type "hi" out of habit before getting to your real ask. With the hook installed, that prompt never reaches the model:

Prompt	Result
`hi`	Blocked
`thank you so much`	Blocked
`fix the login bug`	Allowed
`hello world program in python`	Allowed

Notice the last two rows. "hello world program in python" contains the word hello, but it is clearly a real task, so it sails through. That fullmatch logic is what keeps this from being an annoying false-positive machine.

5. Installing it in under a minute

Head to github.com/QAInsights/pleasantries and clone the repo. Then run the installer:

python install.py

The installer will:

Copy block_pleasantries.py to ~/.pleasantries/
Detect which AI coding CLIs are installed on your machine
Let you pick which ones to hook
Merge the hook into each CLI's config, skipping anything already installed

If you would rather wire it up by hand, here is the Claude Code config as an example. Add this to ~/.claude/settings.json:

{
  "hooks": {
    "UserPromptSubmit": [
      { "hooks": [{ "type": "command",
          "command": "python3 /path/to/block_pleasantries.py claude",
          "timeout": 5 }] }
    ]
  }
}

Swap in the real path to block_pleasantries.py, and you are set. Every other supported CLI follows the same pattern with its own config file, all documented in the README.

To remove every hook later:

python install.py --uninstall

Requirements are light: Python 3.10+, no external dependencies beyond the standard library (json, re, sys), and it runs on macOS, Linux, and Windows.

6. Customizing the blocklist

The blocklist lives in the PLEASANTRY_PATTERNS regex inside block_pleasantries.py, grouped by category:

Greetings: hi, hello, hey, howdy, good morning, and similar
Acknowledgments: ok, sure, yeah, got it
Thanks: thank you, thanks, thx, ty
Please: please, pls, plz
Farewells: bye, goodbye, see ya, later

If your team has its own shorthand, like a Slack-style "yo" or "np", just add it to the matching category and the hook picks it up on the next run.

Adding support for a brand new CLI is just as simple, since the core matcher is CLI-agnostic and each CLI only needs a small adapter function to plug into the shared ADAPTERS dict.

7. Wrap up

Pleasantries is a small tool solving a small but real problem. If you run multiple AI coding CLIs day to day and catch yourself typing "hi" before your actual ask, this hook quietly saves you a wasted model call every single time.

Happy Testing!

Do you type greetings to your AI coding assistant out of habit too, or am I the only one guilty of that?

Mixture of Experts (MoE) Explained

NaveenKumar Namachivayam ⚡ — Thu, 23 Jul 2026 14:54:49 +0000

If you've used Mixtral, DeepSeek, or heard that GPT-4o uses a "Mixture of Experts" architecture, you've encountered one of the biggest efficiency breakthroughs in modern AI. MoE lets models scale to trillions of parameters while keeping inference costs manageable, because it activates only a small fraction of the network for any given input.

Key benefits include scalability, lower per-request compute, and potential specialization across input types. Known challenges include load balancing across experts and added infrastructure complexity. Production implementations include Google's Switch Transformer, Mistral's Mixtral, and DeepSeek-V3, with practical implications for inference cost, latency, and fine-tuning strategies.

Mixture of Experts (MoE) is a machine learning architecture that divides a model into specialized subnetworks called experts, with a gating network routing each input token to only the most relevant subset. This sparse activation approach allows models to scale to trillions of parameters while keeping inference compute costs proportional to active parameters rather than total model size.

What Is Mixture of Experts

Mixture of Experts is a machine learning architecture that divides a model into multiple specialized subnetworks called "experts," with a gating network (or router) deciding which experts should handle each input. Instead of running every parameter for every token as dense models do MoE selectively activates only the most relevant experts, making it a sparse activation technique.

A simple analogy: imagine a classroom where some students excel at math, others at writing, and others at science. Rather than asking every student to answer every question, a "gate" routes each question to the student best suited to answer it. That's the core intuition behind MoE specialization plus selective routing.

Core Components of MoE Architecture

MoE layers typically replace the feedforward network (FFN) portion of a transformer block, and consist of three key pieces:

Experts: Independent feedforward neural networks, each capable of specializing in different patterns in the data
Gating network (router): A smaller network that looks at the input and computes scores to decide which experts are best suited to handle it
Sparse activation: Only the top-k experts (commonly the top 1 or 2) are activated per token, rather than running the entire set of experts

Mathematically, for an input x, the output is a weighted sum of selected experts' outputs, where the gate G(x) assigns a weight to each expert Ei(x), and only a few experts are actually chosen.

How MoE Works Step by Step

The MoE process follows a consistent flow inside a transformer layer:

An input token enters the transformer layer
The gating network computes scores for every available expert based on the token
The top-k experts (highest-scoring ones) are selected
The token is routed only to those selected experts the rest stay idle for that token
Each activated expert processes the token independently
Their outputs are combined into a weighted sum based on the gate's confidence scores
This combined output continues through the rest of the network

Over training, the gating network learns to make better routing decisions if it picks a suboptimal expert, it adjusts itself to improve future choices.

A Simple Worked Example

Say you feed the sentence "Scale this API horizontally without causing db bottleneck" into an MoE-based model with three experts: one that leans toward cloud/deployment concepts, one that leans toward scaling concepts, and one that leans toward db patterns.

The router might compute something like:

Expert 1 (deployment-leaning): high relevance score
Expert 2 (scaling-leaning): moderate relevance score
Expert 3 (db): low relevance score

If the model uses top-2 routing, only Experts 1 and 2 get activated for this token, while Expert 3 stays dormant. Their outputs are combined into a weighted sum, and that becomes the token's representation moving forward. Instead of activating all available experts, only 2 out of 3 fired — and in real large-scale models, that ratio can be far more dramatic, such as 8 experts activated out of 64.

MoE vs Dense Models

Aspect	Dense Model	MoE Model
Parameter activation	All parameters run for every input	Only top-k experts activate per token
Compute cost per inference	Scales with total parameter count	Scales with active parameters only, not total
Scalability	Harder to scale due to compute cost	Can scale to trillions of parameters efficiently
Specialization	Single network learns everything	Experts can specialize in different patterns

Benefits of MoE

MoE architectures deliver several practical advantages that explain their adoption in frontier models:

Massive scalability, enabling models with far larger total parameter counts than dense architectures could practically support
Lower inference compute per request, since only a subset of the network activates
Specialization, which can improve quality on certain types of inputs
Efficient training relative to a dense model of similar total capacity, due to sparse activation

Challenges and Trade-offs

MoE isn't without complexity. A major known issue is load balancing without safeguards, the router can disproportionately favor a few experts, leaving others underused, a problem researchers address with auxiliary load-balancing losses and techniques like router z-loss for training stability. There's also added infrastructure complexity: distributing experts across devices introduces communication overhead, and interestingly, research has found that what individual experts actually learn to specialize in isn't always intuitive or human-interpretable.

Real-World Implementations

Several production and research systems have popularized MoE:

Switch Transformer (Google Research), one of the foundational sparsely-gated MoE architectures
Mixtral (Mistral AI), a widely used open MoE model
DeepSeek MoE and DeepSeek-V3, which introduced refinements like fine-grained experts and auxiliary-loss-free load balancing

Minimal MoE Layer (PyTorch-style)

# AI Generated

import torch
import torch.nn as nn
import torch.nn.functional as F

class Expert(nn.Module):
    """A single expert: just a small feedforward network"""
    def __init__(self, dim, hidden_dim):
        super().__init__()
        self.net = nn.Sequential(
            nn.Linear(dim, hidden_dim),
            nn.ReLU(),
            nn.Linear(hidden_dim, dim)
        )

    def forward(self, x):
        return self.net(x)


class MoELayer(nn.Module):
    """
    A simple top-k Mixture of Experts layer
    """
    def __init__(self, dim, hidden_dim, num_experts, top_k=2):
        super().__init__()
        self.num_experts = num_experts
        self.top_k = top_k

        # The "gating network" / router
        self.gate = nn.Linear(dim, num_experts)

        # The pool of experts
        self.experts = nn.ModuleList([
            Expert(dim, hidden_dim) for _ in range(num_experts)
        ])

    def forward(self, x):
        # x shape: (batch_size, dim)

        # Step 1: Router computes scores for each expert
        gate_logits = self.gate(x)                      # (batch, num_experts)
        gate_scores = F.softmax(gate_logits, dim=-1)

        # Step 2: Pick top-k experts per token
        topk_scores, topk_idx = torch.topk(gate_scores, self.top_k, dim=-1)

        # Normalize the selected scores so they sum to 1
        topk_scores = topk_scores / topk_scores.sum(dim=-1, keepdim=True)

        output = torch.zeros_like(x)

        # Step 3: Route each token only to its selected experts
        for i in range(self.top_k):
            expert_idx = topk_idx[:, i]        # which expert for this slot
            expert_weight = topk_scores[:, i]  # confidence weight

            for e in range(self.num_experts):
                mask = (expert_idx == e)
                if mask.any():
                    expert_output = self.experts[e](x[mask])
                    output[mask] += expert_weight[mask].unsqueeze(-1) * expert_output

        # Step 4: Combine outputs (already summed above as weighted sum)
        return output

Why This Matters for Developers

If you're building LLM-powered applications, MoE directly affects cost and latency, since inference cost tracks active parameters rather than total model size. If you're working on inference infrastructure, expert routing and load balancing become real engineering concerns, not just theoretical details. And if you're exploring fine-tuning or model architecture design, MoE introduces distinct considerations around expert parallelism versus traditional data parallelism.

How Does an LLM Request and Response Cycle Work? A Full Walkthrough

NaveenKumar Namachivayam ⚡ — Mon, 20 Jul 2026 23:36:59 +0000

In this blog post, we will see how an LLM request and response cycle works, from the second you hit send to the moment the last word lands on your screen. I will not throw a wall of transformer math at you. Instead, we will follow one single prompt through every stage of the journey, so nothing feels abstract.

A single prompt sent to a large language model passes through multiple distinct stages before a response appears on screen. The process begins with the client app building a structured JSON request, which then clears an API gateway before the input text is broken into numeric tokens and assembled into the model's context window.

The model then runs a forward pass through transformer layers to predict one token at a time, repeating this loop until an end-of-sequence signal is reached. Tokens stream back to the client as they are produced and are converted back into readable text incrementally, which explains the word-by-word appearance of responses.

I got curious about mapping this out properly while building iamspeed.dev, my browser based LLM benchmarking tool. Before I could measure things like tokens per second or time to first token, I had to understand exactly what happens between "you press Enter" and "words start appearing." Turns out that gap has a lot more steps than most people assume.

The example we will follow

Let's keep one example running through the whole post. Say you type this into a chat app:

You: What's the capital of India?

And a few seconds later, the model replies:

Assistant: The capital of India is New Delhi.

Simple enough on the surface. But that one exchange touches a client app, an API layer, a tokenizer, a neural network with billions of parameters, a sampling algorithm, and a streaming pipeline, all before those six words show up in your chat window. Let's go step by step.

Step 1: You hit send

The moment you press Enter, your chat app does not just fire off the raw sentence. It builds a structured request, usually JSON, that looks something like this:

{
  "model": "some-llm-model",
  "messages": [
    { "role": "system", "content": "You are a helpful assistant." },
    { "role": "user", "content": "What's the capital of India?" }
  ],
  "temperature": 0.7,
  "max_tokens": 100,
  "stream": true
}

Notice a few things here. The system prompt is bundled in even though you never typed it. Any earlier messages in the conversation get bundled in too, since the model has no memory of its own between calls. And stream: true is already set, which matters a lot later. This payload gets sent over HTTPS to an API endpoint.

Step 2: The API gateway

Your request does not go straight to a model. It first hits an API gateway that handles the boring but essential stuff:

Validating your API key or session token
Checking rate limits, so nobody floods the system
Routing the request to the right model version and the right compute cluster, often based on region for lower latency
Logging the request for billing and abuse monitoring

Think of this layer as the bouncer and the traffic cop combined. Only after it clears the request does anything resembling "AI" actually happen.

Step 3: Tokenization

Here is where things get interesting. The model does not read English. It reads tokens, which are chunks of text mapped to numbers. A tokenizer breaks your sentence apart, usually using a scheme like byte pair encoding.

Our example sentence, "What's the capital of India?", might get tokenized into something like:

["What", "'s", " the", " capital", " of", " India", "?"]

Each of these chunks maps to a specific integer ID from the model's vocabulary, something like [1780, 434, 262, 3139, 286, 4881, 30]. The exact split depends on the tokenizer the model uses, but the idea is always the same: text becomes numbers, because that is the only thing a neural network can actually compute on.

This step matters more than people realize. It is also why providers bill you per token instead of per word or per character. A short word can be one token, a rare word can be split into three or four.

Step 4: Building the context window

Your new tokens do not go in alone. The system stitches together:

The system prompt tokens
Any prior conversation history tokens
Your new message tokens

All of this gets packed into the model's context window, which is just a fancy term for "how many tokens the model can look at in one shot." Each token also gets a positional encoding attached, so the model knows word order, since without it, "India of capital the" and "capital of India" would look identical to the network.

Step 5: The forward pass

Now the actual model runs. Your token sequence flows through dozens of transformer layers stacked on top of each other. Inside each layer, two things happen repeatedly:

Self-attention, where every token looks at every other token in the sequence and decides how much to "pay attention" to it. This is how the model knows that "capital" relates to "India" and not to some other country mentioned three messages ago.
Feed-forward processing, where each token's representation gets transformed further, layer after layer, refining what the model "understands" about the sequence so far.

After all layers, the model produces a probability distribution over its entire vocabulary, essentially a ranked guess of what the very next token should be. For our example, after processing "What's the capital of India?", the model's internal state is now primed to predict something like "The" as a strong first candidate.

Step 6: Sampling the next token

Here is the part that surprises people the first time they learn it: the model does not generate the whole sentence at once. It generates one token, feeds that token back into itself as part of the input, and generates the next one. This repeats until it decides to stop.

Parameters like temperature, top_p, and top_k control how the next token gets picked from that probability distribution. Lower temperature means the model plays it safe and picks the highest probability token almost every time. Higher temperature adds randomness, useful for creative writing, less useful for factual answers.

For our example, the autoregressive loop looks roughly like this:

Input so far: "What's the capital of India?"
→ predict: "The"

Input so far: "...India?" + "The"
→ predict: " capital"

Input so far: "...The" + " capital"
→ predict: " of"

... and so on, until:
"The capital of India is New Delhi." + <end-of-sequence token>

That end-of-sequence token is the model's way of saying "I am done," which is also how it knows when to stop rather than rambling forever.

Step 7: Streaming the response back

Remember stream: true from Step 1? This is where it pays off. Instead of waiting for the entire response to finish generating and sending it as one big blob, the server pushes each token to your client the moment it is produced, usually over Server-Sent Events or a chunked HTTP connection.

That is exactly why you see words appear on screen one at a time instead of the whole answer popping in at once. It is not a visual effect, it is literally showing you tokens as the model produces them.

Step 8: Detokenization and rendering

On the client side, each incoming token ID gets converted back into readable text, the reverse of Step 3. The chat app appends each piece to the message bubble as it arrives, re-renders the UI, and by the time the end-of-sequence signal shows up, you are looking at the complete sentence:

The capital of India is New Delhi.

Six words. Eight steps. All of it usually finishing in under two seconds.

Where latency actually lives

Given my performance engineering background, I cannot write this post without pointing at the clock. Two numbers matter a lot more than "the response took 3 seconds":

Time to first token (TTFT): how long you wait between hitting send and seeing the very first word appear. This is dominated by Steps 1 through 5, especially the forward pass on a long context window.
Inter-token latency: how quickly tokens keep arriving after the first one, which shapes how "smooth" the typing effect feels.

A model can have a slow TTFT but fast inter-token speed, or the reverse, and the two create very different user experiences even if the total time is identical. This is the exact gap I was trying to measure when building iamspeed.dev, and honestly, understanding this full life cycle is what made the benchmarking numbers actually make sense instead of just being numbers on a dashboard.

Check iamspeed.dev to measure the LLM performance.

Wrap up

So the next time you type a question into a chat app and watch the answer type itself out, you now know the full trip that sentence takes: request formation, gateway checks, tokenization, context assembly, a forward pass through a massive network, token-by-token sampling, streaming, and finally detokenization back into words you can read.

It looks like magic from the outside. From the inside, it is a very well engineered pipeline.

Happy Testing!

Have you ever watched an LLM response stream in and wondered what was happening under the hood? Let me know in the comments.

Leitner Loop - AWS Cert Pilot

NaveenKumar Namachivayam ⚡ — Mon, 20 Jul 2026 14:36:37 +0000

Vision & What the Agent Does

Studying for an AWS certification usually means remembering to sit down, open a study app, and grind through flashcards and that habit dies the moment life gets busy. Leitner Loop removes the "open the app" step entirely. It's an always-on study partner that runs on its own schedule, decides what you need to review next, writes brand-new practice questions for it, and emails them to you no button click required.

Leitner Loop is an automated AWS certification study tool that runs on a scheduled basis, requiring no manual interaction. Every two hours, EventBridge Scheduler triggers a Lambda function that identifies due review topics using a five-box spaced-repetition system, generates fresh multiple-choice questions via Amazon Bedrock, and delivers them by email through SES with clickable answer links.

Clicking an answer routes through API Gateway to a second Lambda that grades the response, provides an explanation, and updates the topic's review schedule in DynamoDB. Key implementation challenges included defensively parsing Bedrock's JSON output and adapting to AWS's deprecation of direct model ID invocation in favor of cross-region inference profiles.

Every two hours, an EventBridge Scheduler rule triggers the agent. It checks a DynamoDB table of exam topics for whichever ones are "due" for review (using a Leitner spaced-repetition schedule), asks Amazon Bedrock (Claude Haiku 4.5) to generate up to 10 fresh, scenario-based multiple-choice questions across those due topics, and emails them to me via SES each answer option is a clickable link.

When I click an answer from my inbox, API Gateway routes it to a second Lambda that grades it, shows the correct answer and explanation, and reschedules that topic further out (if correct) or back to square one (if wrong). It reports back the only way that matters when you're away from the keyboard: an email waiting for you, and a running record of what you actually know.

The Leitner System

It's a classic spaced-repetition study technique (invented by Sebastian Leitner in the 1970s using physical flashcard boxes). The core idea: things you know well get reviewed less often, things you get wrong get reviewed again soon.

How it works, generically:

Cards live in numbered "boxes" (1 = review most often, 5 = review least often).
Answer correctly ? card moves to the next box up (longer interval before it's due again).
Answer incorrectly ? card drops back to box 1 (reviewed again soon).

How agent implements it:

BOX_INTERVALS = {1: 1, 2: 3, 3: 7, 4: 14, 5: 30}

Each exam topic has a box (1–5) and a next_due date stored in DynamoDB. When you answer a question:

Correct ? box = min(box + 1, 5) — moves up a box, next review pushed further out (1 ? 3 ? 7 ? 14 ? 30 days).
Incorrect ? box = 1 — resets to daily review.

So "Leitner Loop" is the agent's name because the whole thing is an automated, always-running loop of that Leitner box system: lambda_quiz_generator.py picks topics that are due (next_due <= today) every 2 hours and generates questions for them; answering them via lambda_grading_handler.py feeds the box/interval update back into DynamoDB, which determines what the next run picks.

That closed feedback loop — pick due topics ? quiz ? grade ? reschedule ? repeat is the "loop" in the name.

How I Built It

The core design decision was splitting the agent into two single-purpose Lambdas quiz-generator and grading-handler each with its own IAM role and least-privilege inline policy, rather than one monolithic function. This kept the "generate" path (Bedrock + SES + writes) cleanly separated from the "grade" path (reads + updates triggered by an untrusted public URL click), which also made reasoning about security boundaries much simpler.

The trickiest part was making Bedrock's output reliable enough to parse automatically every run. Claude occasionally wraps JSON in markdown code fences or adds stray text, so generate_question() strips backticks and a leading json tag before parsing a small defensive step that turned an intermittent failure into a non-issue. I also had to work around AWS deprecating direct invocation of newer Claude model IDs; the fix was switching to a us.-prefixed cross-region inference profile ID instead of a bare model ID.

For the spaced repetition itself, I used a classic 5-box Leitner scheme (`BOX_INTERVALS = {1:1, 2:3, 3:7, 4:14, 5:30}` days) simple, well-understood, and easy to tune. Switching which certification the agent studies is just a one-line change to the EventBridge target's input payload (`{"cert_code": "SAA-C03"}`), no redeploy needed, since all cert-specific data (exam domains, topic names) lives in DynamoDB via seed_topics.py rather than in code.

AWS Services Used / Architecture Overview

Services: Amazon EventBridge Scheduler, AWS Lambda (2 functions), Amazon Bedrock (Claude Haiku 4.5 via cross-region inference profile), Amazon SES, Amazon API Gateway (HTTP API), Amazon DynamoDB (2 tables), AWS IAM.

Both DynamoDB tables use on-demand billing and stay within the AWS Free Tier. TTL on `cert-quiz-pending` auto-expires unanswered questions after 48 hours so nothing lingers. The trigger is entirely schedule-driven I never open an app; the agent decides on its own whether there's anything worth sending.

Cost

Based on current AWS pricing (all us-east-1), assuming the default rate(2 hours) schedule (max 12 runs/day) and up to 10 questions/email:

Service	Pricing	Usage	Est. Monthly Cost
EventBridge Scheduler	$1.00/million invocations after 14M free	~360 invocations/mo	$0.00 (free tier)
Lambda (2 functions)	$0.20/million requests + $0.0000166667/GB-s; free tier: 1M req + 400K GB-s/mo	~4K invocations/mo, short duration	$0.00 (free tier)
Amazon Bedrock (Claude Haiku 4.5)	$1.00/1M input tokens, $5.00/1M output tokens	~120 calls/day worst case (~800 in / 300 out tokens each) ? ~2.9M input + 1.1M output tokens/mo	$3–$9/month
Amazon SES	$0.10/1,000 emails; 3,000 free/mo for first 12 months	?360 emails/mo	$0.00–$0.04
API Gateway (HTTP API)	$1.00/million requests; 1M free/mo for 12 months	Few hundred requests/mo	$0.00
DynamoDB (on-demand, 2 tables)	~$1.25/million WRUs, $0.25/million RRUs	Low hundreds of ops/mo	~$0.01
CloudWatch Logs	Free tier: 5GB ingestion/mo	Minimal log volume	$0.00

Total: ? $3–$10/month

Cost driver: Bedrock dominates almost entirely. The range depends on how often topics are actually "due" the Leitner reschedule logic means most 2-hour runs are silent (no due topics), so real-world cost is likely closer to the $1–$3/month range unless you're actively answering and cycling through many topics daily.

Notes:

All other services stay within their AWS Free Tier for this workload, so they're effectively $0.
First-year SES free tier (3,000 messages/mo) and Lambda's permanent free tier further reduce costs Lambda's 1M requests/400K GB-s allowance never expires.
Sources: aws.amazon.com/bedrock/pricing, aws.amazon.com/lambda/pricing, aws.amazon.com/eventbridge/pricing, aws.amazon.com/ses/pricing, aws.amazon.com/api-gateway/pricing.

Screenshots

What I Learned

Building this reinforced how much reliability work goes into "just call an LLM" once it's unattended validating and defensively parsing model output matters far more when there's no human in the loop to notice a malformed response.

I also learned the hard way that AWS retires direct base-model-ID invocation for newer Bedrock models in favor of cross-region inference profiles, and that IAM policy changes don't always take effect on warm Lambda execution environments immediately.

On the architecture side, designing around DynamoDB TTL for ephemeral state (pending answers) turned out to be a clean, zero-maintenance way to handle "expire after N hours" without a cleanup job.

$20/Month: The Price Ceiling Every AI Company Copied

NaveenKumar Namachivayam ⚡ — Sat, 18 Jul 2026 18:37:47 +0000

In this blog post, we will see why almost every major AI subscription, ChatGPT, Claude, Perplexity, and Gemini, somehow landed on the exact same $20 a month price tag. We will trace it back to where it started, look at the actual reasoning behind the number, and figure out whether this price ceiling will hold or eventually crack the way streaming subscriptions did.

The $20 monthly price point shared by ChatGPT Plus, Claude Pro, Perplexity Pro, and Google AI Pro traces back to OpenAI's February 2023 launch, which was designed to subsidize free-tier costs rather than reflect the actual value of the product. Competitors adopted the number through price anchoring, not independent cost analysis.

The same pattern has extended to smaller AI tools and is now repeating at higher tiers, with $200 and $100 monthly plans emerging for power users. Despite identical pricing, what each $20 subscription delivers varies significantly across providers in terms of usage limits, features, and model access.

The Coincidence That Isn't a Coincidence

As of mid-2026, ChatGPT Plus, Claude Pro, and Perplexity Pro all cost exactly $20 a month. Google AI Pro (formerly Gemini Advanced) sits one cent below at $19.99. Four completely different companies, four completely different models, and yet the sticker price converges on almost the same number.

That's not four companies independently landing on the same cost math. It's one company setting a price, and everyone else deciding not to compete on it.

Where It Actually Started: OpenAI, February 2023

ChatGPT launched free in November 2022 and crossed a million users within about a month, which was an enormous number for a research preview. On February 1, 2023, OpenAI introduced ChatGPT Plus at $20 a month, expanding it internationally on February 10. The pitch at the time was simple: general access even during peak load, faster responses, and priority access to new features.

Worth remembering: this was the GPT-3.5 era. GPT-4 hadn't shipped yet. Subscribers in February 2023 were paying $20 for a noticeably weaker model than what free users get today. The price has not moved since, even as that same $20 has quietly absorbed GPT-4, several GPT-5 generations, Deep Research, Sora video, Codex, and agent mode.

The Real Reason Behind $20

OpenAI wasn't shy about why Plus existed. Running ChatGPT was expensive, Sam Altman described the compute costs as substantial even at a few cents per conversation, and the company was under real pressure to make the free product sustainable. OpenAI itself said the subscription revenue existed partly to keep free access available to as many people as possible.

Put plainly, $20 wasn't calculated as "what is a fair price for this capability." It was closer to "what can we charge a subset of engaged users so the free tier doesn't bankrupt us." OpenAI was reportedly expecting around $200 million in revenue for 2023, against more than a billion dollars already invested in the company. The $20 tier was a stopgap, not a value calculation.

The Copycat Effect

Once ChatGPT Plus proved people would pay $20 without mass cancellations, every serious competitor used that number as their starting point instead of testing their own.

Claude Pro launched at $20/month (with a modest annual discount).
Perplexity Pro launched at $20/month.
Google AI Pro priced itself at $19.99, a classic one-cent-under-the-anchor move rather than an independent price.
Grok's SuperGrok is the outlier at $30/month, positioned above the pack rather than matching it.

None of these companies share OpenAI's cost structure. Different model sizes, different infrastructure, different margins. But none of them wanted to be the one charging more for what would look like a worse deal, and none of them needed to undercut a price users had already accepted. That's price anchoring playing out at the scale of an entire industry.

It's Not Just the Big Four

This isn't only an OpenAI-versus-Anthropic-versus-Google story. Scroll through any random week of Product Hunt or Hacker News launches and the same number shows up over and over, on tools that have nothing to do with the frontier labs.

A 2026 Product Hunt launch playbook says it outright, the AI audience on that platform has been trained by ChatGPT to expect $20/month as the ceiling. Founders pricing above that are advised to address it head-on in their launch comments. Founders pricing below it, or offering a real free tier, are told to lead with that instead, since most funded AI tools have quietly dropped their free tiers altogether.

A post on Indie Hackers picking apart AI tool pricing in 2026 flagged the same pattern from the builder's side, a "weird clustering happening around the $20/month price point" across small, independently built AI products. Its explanation for why lines up with everything happening at the top of the market too, most of these tools aren't priced against what they cost to run, they're priced against what feels acceptable at a glance. Low enough that a buyer won't push back, high enough that almost nobody checks what the underlying API usage would have actually cost if billed directly instead.

You can watch that anchor form in real time inside launch threads. On one Show HN post for a coding assistant priced at $20/month with a 100 message cap, a commenter said the quiet part out loud, they assumed the tool was close to unlimited at that price only because $20 a month is already what they pay for Gemini or ChatGPT, and that already feels unlimited to them. Nobody in that thread calculated $20 from cost. It was already the accepted price of "an AI subscription," so it became the price of this one too.

So no, you don't need to dig very hard to confirm this yourself. Open Product Hunt's AI category on any given day, or skim a handful of recent Show HN posts pitching an AI wrapper, assistant, or agent. $20/month will be sitting right there as the default, again and again.

What $20 Actually Buys You Right Now

Here's a concrete snapshot, since "$20 a month" means very different things depending on the provider. As of mid-2026, roughly:

ChatGPT Plus: around 150 messages every 3 hours on the current flagship model, plus Deep Research capped at a handful of runs a month.
Claude Pro: a generous message allowance on the current flagship model, positioned around coding and long-form writing use cases.
Google AI Pro: a very large context window (in the range of a million tokens), bundled with Google storage and Workspace integration.
Perplexity Pro: effectively unlimited Pro searches, but Deep Research usage has been tightened over time.

The number on the invoice is identical. The actual product you get for that number is not even close. This is the part most comparison articles gloss over, they show you a price table and stop, without mentioning that "$20" buys you a completely different bundle of limits depending on which company you hand it to.

These specific limits shift every few months as providers adjust caps, so treat the numbers above as a snapshot rather than gospel, and check the current plan page before you decide.

Why $20 and Not $10 or $30

A few analysts pricing out the AI subscription market this year converged on the same explanation: $20 a month is roughly the ceiling a typical person will tolerate for a single productivity subscription before they start looking for a reason to cancel. Below that, you leave money on the table. Above it, you start losing casual users to the free tier or to a cheaper competitor.

Google AI Pro pricing at $19.99 instead of a round $20 is a small but telling detail. It's not a cost-based number, it's a psychological anchor, deliberately priced just under the number everyone else already normalized.

I've watched a smaller version of this play out in my own world. I sell developer tools on Gumroad, and pricing a plugin isn't really about calculating your costs and adding a margin. It's about figuring out what the market has already decided a similar tool is "supposed to" cost, then deciding whether you sit at, above, or just under that number. AI subscriptions are doing the exact same thing, just with a few more zeroes involved.

The Next Ceiling: $100 to $200

The $20 tier isn't the top of the market anymore, it's the floor for serious usage. OpenAI introduced ChatGPT Enterprise in August 2023, ChatGPT Teams in January 2024 (later increased to $30/month), and a $200/month ChatGPT Pro tier in December 2024 aimed at unlimited access for heavy users. In April 2026, OpenAI added a $100/month Pro tier, offering the same model suite as the $200 tier at lower usage limits, a move widely read as a direct response to Anthropic's Claude Max 5x plan.

The same anchoring pattern is repeating one level up. One provider sets a $200 ceiling for power users, and rather than undercutting it, competitors match it and differentiate on limits instead of price. It's the $20 story again, just with an extra zero.

My Take

The $20 price tag has become a kind of unspoken industry standard, similar to how $9.99 became the default for a certain class of consumer app subscription, or how $99 shows up everywhere in early-stage SaaS pricing. Nobody sat down and calculated $20 as the "correct" price for access to a frontier AI model. OpenAI needed it to survive, everyone else needed a reason not to compete on price, and $20 became the answer to both problems at once.

Whether it holds depends entirely on compute costs. Models are getting cheaper to run per token even as they get more capable, which is the one force that could eventually push this ceiling down instead of up. But subscription businesses rarely cut prices voluntarily. My guess is the $20 tier stays exactly where it is, and all the real competition keeps happening one level up, at $100 and $200.

Do you think the $20 AI subscription price holds for another few years, or does it eventually creep upward the way streaming subscriptions did? Let me know in the comments.

Happy Testing!

Performance Testing RAG Applications: Complete Engineering Guide

NaveenKumar Namachivayam ⚡ — Mon, 06 Jul 2026 15:56:54 +0000

In this blog post, we will see how to performance test a RAG (Retrieval-Augmented Generation) application properly, covering both speed and correctness, and how to wire both into a CI/CD pipeline so regressions get caught before they reach production.

Performance testing a RAG application requires two separate testing gates: one for speed and one for answer quality. Traditional load testing tools measure response times but cannot detect hallucinations, where a model returns fast but factually incorrect answers grounded in fabricated context rather than retrieved documents.

The guide demonstrates using k6 for load testing end-to-end latency and DeepEval for evaluating faithfulness and answer relevancy using an LLM-as-judge approach. Both gates are integrated into a GitHub Actions CI/CD pipeline so regressions in either performance or output quality are caught automatically on every pull request before reaching production.

If you've come from a JMeter or k6 background like I have, your first instinct with a RAG endpoint is probably to point a load test at it and check response times. That gets you halfway there. A RAG app can return a fast, confident, completely wrong answer, and a plain load test will never tell you that. You need two testing surfaces, not one: performance and quality. This guide covers both, using a single running example throughout: a documentation assistant that answers "How do I run JMeter in non-GUI mode?" against a small knowledge base.

Why RAG breaks traditional load testing assumptions

A conventional API returns a complete response and you measure the round trip. A RAG endpoint does two expensive things before it answers: it retrieves context from a vector store or search index, then it streams a generated response token by token. That second part matters a lot. A single request can stream hundreds of tokens over several seconds, so "request duration" as a single number hides two very different problems: how long the model took to start answering, and how fast it generated once it started.

A system with slow startup but fast generation feels broken to someone typing in a chat UI. A system with fast startup but slow generation is fine for a quick question but painful for a long document summary. Averaging those together tells you nothing useful.

The two testing surfaces: performance and quality

I think of RAG testing as two separate gates that happen to run against the same endpoint.

Performance answers: how fast is it, and does it hold up under load? This is k6's job, same as any other API load test, just with LLM-specific metrics layered on.

Quality answers: is the answer actually grounded in what got retrieved, or did the model make something up? This is where DeepEval comes in, scoring faithfulness and relevancy on every response using an LLM as the judge.

Neither gate alone tells the full story. A fast RAG app that hallucinates is worse than a slow one that's accurate, and a perfectly grounded app that takes eight seconds to respond will lose users regardless of correctness.

Metrics that actually matter

Performance metrics

Metric	What it tells you
TTFT (Time to First Token)	How long a user stares at a blank screen before anything appears
ITL (Inter-Token Latency)	How smoothly tokens stream once generation starts
Tokens/sec	Generation speed, matters most for long-form answers
p95 / p99 latency	The tail experience, not the average one

TTFT is the most user-visible number in the whole system, and it's also the metric most classic load testing tools weren't built to isolate, since they were designed for atomic request/response cycles, not streams.

Quality metrics

Metric	What it tells you
Faithfulness	Is the answer grounded in the retrieved context, or invented
Answer relevancy	Does the answer address the actual question, or just sound plausible
Context precision	Did retrieval return the right chunks, ranked correctly
Context recall	Did retrieval miss anything the answer needed

These four metrics carry most of the diagnostic weight in RAG evaluation. Faithfulness and answer relevancy live on the generation side; context precision and recall live on the retrieval side. When faithfulness is low but context recall is high, the retriever did its job and the model ignored it that's a prompting problem, not a retrieval problem. Worth knowing the difference before you go tuning the wrong component.

Hallucination detection with DeepEval

I'm using DeepEval here instead of RAGAS mainly because DeepEval treats evaluations as pytest test cases with pass/fail thresholds, which is exactly the shape you need for a CI/CD gate. It also accepts any LLM as the judge model, so it isn't locked to one vendor even though our example app happens to use Gemini.

Here's what a test case looks like against our JMeter doc-assistant example:

from deepeval.metrics import FaithfulnessMetric, AnswerRelevancyMetric
from deepeval.test_case import LLMTestCase
from deepeval.models import GeminiModel

judge_model = GeminiModel(
    model="gemini-3.5-flash",
    api_key=os.getenv("GEMINI_API_KEY"),
)

faithfulness_metric = FaithfulnessMetric(threshold=0.75, model=judge_model)
answer_relevancy_metric = AnswerRelevancyMetric(threshold=0.8, model=judge_model)

def test_jmeter_non_gui_mode_answer():
    question = "How do I run JMeter in non-GUI mode?"
    result = query_rag_app(question)

    test_case = LLMTestCase(
        input=question,
        actual_output=result["answer"],
        retrieval_context=result["retrieved_chunks"],
    )

    for metric in [faithfulness_metric, answer_relevancy_metric]:
        metric.measure(test_case)
        status = "PASS" if metric.success else "FAIL"
        print(f"[{status}] {metric.__class__.__name__}: {metric.score:.3f}")

    failed = [m for m in [faithfulness_metric, answer_relevancy_metric] 
              if not m.success]
    if failed:
        names = ", ".join(m.__class__.__name__ for m in failed)
        raise AssertionError(f"Metrics below threshold: {names}")

Run this with pytest, and it either passes or fails like any other test. That's the whole point it turns a fuzzy "does the AI sound right" question into a binary CI/CD signal.

The test suite includes retry logic to handle transient Gemini API 503 errors, automatically retrying up to 3 times with exponential backoff. DeepEval generates both JUnit XML and HTML reports, making it trivial to wire into any CI system that understands pytest output.

Load testing with k6 (and why you can't measure TTFT yet)

Here's where things get frustrating if you came here looking for a clean TTFT measurement story: the k6 SSE extension (xk6-sse) is not compatible with k6 v2. It targets go.k6.io/k6 v1, and until it gets updated, you're stuck choosing between k6 v2's improved architecture or the ability to measure streaming metrics properly.

So the companion repo does the pragmatic thing: it tests the /chat/complete endpoint instead of the /chat streaming endpoint, using k6's built-in http module. No custom binary, no extensions, just standard k6. The tradeoff is you lose true TTFT measurement, because /chat/complete waits for the full response before returning. What you get instead is end-to-end latency, which is still useful it tells you if the system is slow, just not why it's slow.

Here's what the test looks like:

import http from 'k6/http';
import { Trend, Counter } from 'k6/metrics';
import { check } from 'k6';

const totalDuration = new Trend('total_duration_ms', true);
const tokensPerSecond = new Trend('tokens_per_second');

const BASE_URL = __ENV.RAG_APP_URL || 'http://localhost:8080';

export const options = {
  scenarios: {
    rag_chat: {
      executor: 'ramping-vus',
      stages: [
        { duration: '30s', target: 10 },
        { duration: '1m', target: 10 },
        { duration: '30s', target: 0 },
      ],
    },
  },
  thresholds: {
    http_req_duration: ['p(95)<6000'],
    total_duration_ms: ['p(95)<6000'],
  },
};

export default function () {
  const startTime = Date.now();

  const res = http.post(
    `${BASE_URL}/chat/complete`,
    JSON.stringify({ query: 'How do I run JMeter in non-GUI mode?' }),
    {
      headers: { 'Content-Type': 'application/json' },
      timeout: '30s',
    },
  );

  const duration = Date.now() - startTime;

  check(res, {
    'status 200': (r) => r.status === 200,
    'has answer': (r) => JSON.parse(r.body).answer !== undefined,
  });

  totalDuration.add(duration);

  // Rough tokens/sec estimate from word count
  const words = JSON.parse(res.body).answer.trim().split(/\s+/).length;
  tokensPerSecond.add((words / duration) * 1000);
}

The test ramps from 0 to 10 virtual users over 30 seconds, holds for a minute, then ramps back down. Thresholds are set at p95 < 6000ms for both http_req_duration and the custom total_duration_ms metric.

When should you switch back to SSE? Watch the xk6-sse repo. Once it adds k6 v2 support, swap the endpoint from /chat/complete to /chat, add the SSE extension to your Dockerfile, and you'll get true TTFT measurement. Until then, this is the most pragmatic path forward standard k6, no custom builds, just with the caveat that you're measuring end-to-end latency rather than streaming behavior.

The companion repo includes both endpoints in the Express app so you can switch when you're ready:

Endpoint	Response	Status
`POST /chat`	SSE stream	Ready for when xk6-sse supports k6 v2
`POST /chat/complete`	Full JSON	Used by k6 and DeepEval today

Wiring both gates into CI/CD

Once both tests run locally, wiring them into GitHub Actions is mostly plumbing: start the app, wait for it to be healthy, run the k6 gate, run the DeepEval gate, both in parallel since they're independent.

name: RAG CI

on: [pull_request]

jobs:
  performance-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Write app env file
        run: |
          cat > app/.env << EOF
          GEMINI_API_KEY=${{ secrets.GEMINI_API_KEY }}
          GEMINI_MODEL=gemini-3.5-flash
          FILE_SEARCH_STORE_NAME=${{ secrets.FILE_SEARCH_STORE_NAME }}
          PORT=8080
          EOF

      - name: Start RAG app
        run: docker compose up -d --build app

      - name: Wait for health
        run: |
          timeout 60 bash -c 'until curl -f http://localhost:8080/health; do sleep 2; done'

      - name: Run k6 load test
        run: docker compose --profile perf run --rm k6

  quality-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Write app env file
        run: |
          cat > app/.env << EOF
          GEMINI_API_KEY=${{ secrets.GEMINI_API_KEY }}
          GEMINI_MODEL=gemini-3.5-flash
          FILE_SEARCH_STORE_NAME=${{ secrets.FILE_SEARCH_STORE_NAME }}
          PORT=8080
          EOF

      - name: Start RAG app
        run: docker compose up -d --build app

      - name: Wait for health
        run: |
          timeout 60 bash -c 'until curl -f http://localhost:8080/health; do sleep 2; done'

      - name: Run DeepEval tests
        env:
          GEMINI_API_KEY: ${{ secrets.GEMINI_API_KEY }}
        run: docker compose --profile quality run --rm deepeval

Both jobs run on every pull request. A PR that slows down response time and a PR that quietly makes the model hallucinate get caught the same way, before either reaches a reviewer's eyeballs, let alone production.

You'll need to add two secrets to your GitHub repo before the workflow will pass:

Secret	Value
`GEMINI_API_KEY`	Your Gemini API key from https://aistudio.google.com/apikey
`FILE_SEARCH_STORE_NAME`	The store name from `setup-store.js` (format: `fileSearchStores/your-store-id`)

Setting SLOs

I'm deliberately not giving you one universal latency number to target. I've seen guidance ranging from sub-second targets for chat-style RAG apps to 3-5 second budgets for more complex document analysis, and the right number for you depends entirely on your retrieval backend, your model, and what your users are actually doing. Run the load test against your own baseline first, then set thresholds off that baseline, not off a number from a blog post (including this one).

The example repo uses p95 < 6000ms as a starting point because that's what the test Gemini File Search RAG app achieves at 10 concurrent users with gemini-3.5-flash. Your mileage will vary dramatically based on:

Model choice (flash vs pro, size of context window actually used)
Retrieval backend (vector DB query time, number of chunks retrieved)
Document size and complexity
Network latency to your LLM provider

What you should track regardless of the exact number:

p95 and p99 latency, not just the median. The tail experience is what users complain about.
Latency at your expected concurrency, not at 1 user. RAG apps often degrade non-linearly under load because of retrieval bottlenecks.
Faithfulness and answer relevancy trending over time, not just pass/fail on one run. A metric that's consistently 0.90 dropping to 0.78 is a signal even if both pass the 0.75 threshold.

Wrap-up

RAG performance testing is really two disciplines wearing one trench coat: classic load testing with LLM-aware metrics, and LLM-as-judge quality scoring that classic load testing tools were never built to do. Run them both, gate on both, and you'll catch the regressions that a speed-only test walks right past.

The current state of tooling isn't perfect you can't measure TTFT with k6 v2 without writing your own SSE client, and LLM-as-judge scoring has its own consistency quirks but it's good enough to catch regressions before production, which is the whole point of a CI/CD gate.

Head to the companion GitHub repo for the full working app, k6 script, DeepEval tests, Docker Compose setup, and GitHub Actions workflow you can clone and run locally in under five minutes.

Happy Testing!

Have you run into hallucination regressions that a pure load test missed? I'd like to hear how you caught them reply on X or open an issue on the companion repo.

Qwen3 vs DeepSeek R1: Which Open-Source Reasoning Model Should You Use in 2026?

NaveenKumar Namachivayam ⚡ — Thu, 25 Jun 2026 15:11:58 +0000

In this blog post, we will see how Qwen3 and DeepSeek R1 compare as open-source reasoning models, where each one shines, and which one you should actually run in 2026.

Open-source reasoning models have changed the game. DeepSeek R1 felt like a revolution when it dropped in early 2025. Then Qwen3 from Alibaba quietly overtook Llama as the most downloaded model family on Hugging Face by late 2025. Now both are serious contenders.

If you run local LLMs for code, automation, or performance analysis, you have probably stared at this choice. Let me break it down with one grounding example so you can make the call.

DeepSeek R1 lit the fire for open-source reasoning. The distilled 14B and 32B variants are still excellent, especially for math-heavy tasks on limited hardware.

But in 2026, Qwen3 is the more versatile daily driver. The hybrid thinking mode alone justifies the switch. You get DeepSeek-level depth when you need it and sub-5-second responses when you do not. The ecosystem, the model size range, and the tooling around Qwen3 are simply broader.

What Are These Models?

DeepSeek R1 is a reasoning model from DeepSeek AI, released in January 2025. It uses a dense 671B parameter architecture trained with multi-stage reinforcement learning. Every single query goes through a chain-of-thought reasoning process. That is its identity.

Qwen3 is Alibaba's open-source LLM family, spanning 0.6B to 235B parameters. The flagship is Qwen3-235B-A22B, a Mixture-of-Experts (MoE) model that activates only 22B parameters per forward pass. Every Qwen3 model ships with a built-in dual-mode thinking system. Flip a soft switch in your prompt and the same model either engages deep chain-of-thought reasoning or returns fast responses like a traditional assistant.

That single design decision separates the two philosophies.

The Key Architectural Difference

DeepSeek R1 is always reasoning. There is no off switch. Every query burns through a full chain-of-thought, whether you are asking it to fix a typo or solve a differential equation. Typical response latency for complex reasoning is 30 to 90 seconds. That is not viable for real-time customer-facing chat, but for batch processing, code review automation, or research tasks, it is acceptable.

Qwen3 gives you a choice. Thinking mode on: deep reasoning, comparable to DeepSeek R1. Thinking mode off: fast response, like a traditional assistant.

In Ollama, you trigger it simply:

ollama run qwen3:8b
/set think

Or prefix your prompt with /think in API calls.

This is a practical advantage. You do not always need a model to overthink. Qwen3 lets you decide.

A Simple Example: Performance Bottleneck Reasoning

Here is one concrete prompt to anchor the comparison.

Prompt:

"A load test has 100 virtual users hitting an API endpoint. Each request takes 2 seconds. What is the throughput in requests per second? Show your reasoning and flag any assumptions."

DeepSeek R1 responded in approximately 95 seconds. It walked through Little's Law correctly: Throughput = Concurrent Users / Response Time = 100 / 2 = 50 RPS. The answer was accurate. It also flagged that the formula assumes zero think time between requests, meaning 100% CPU utilization. Solid, but the wait was real.

Qwen3-8B with thinking mode responded in about 105 seconds. Same correct answer. On output structuring, Qwen3 takes the lead. It added a note about how think time and pacing affect the real-world number, and formatted the output with clear sections. Slightly slower than DeepSeek on raw latency, but better organized.

Qwen3-8B without thinking mode returned the correct answer in under 5 seconds. No chain-of-thought. Just: 50 RPS based on Throughput = Concurrent Users / Average Response Time.

For a quick sanity check during a load test session, that 5-second response changes your workflow. For a deep architectural review where reasoning quality matters, both models land at the same level.

The switchable thinking mode is the real differentiator in day-to-day use.

Benchmarks at a Glance

With only 60% activated and 35% total parameters, Qwen3-235B-A22B in thinking mode outperforms DeepSeek R1 on 17 out of 23 benchmarks, particularly on mathematics, agent tasks, and coding.

Key numbers:

ArenaHard (overall reasoning): Qwen3-235B scores 95.6, DeepSeek R1 scores 91.8
CodeForces Elo (competitive programming): Qwen3-235B scores 2056, DeepSeek R1 scores 2029
MATH-500: DeepSeek R1 scores 97.3, Qwen3 scores 97.2. Essentially tied.

DeepSeek R1 holds a clear advantage on pure mathematical reasoning. This is the benchmark where DeepSeek's reputation is most defensible. If your workload centers on mathematical reasoning, that edge is real.

For coding specifically, Qwen3-32B in thinking mode scores 1970 on CodeForces Elo. That is above GPT-4o.

Hardware Requirements

This is where the gap becomes very practical.

DeepSeek R1 full model: 671B parameters, needs 400+ GB memory. Not viable on consumer hardware.
DeepSeek R1 distilled (14B): Runs on a 12 GB GPU, strong reasoning performance.
Qwen3-8B: Runs on 6 GB VRAM in Q4 quantization. RTX 3060 level.
Qwen3-32B: Runs on a single RTX 4090 with 24 GB VRAM.

Qwen3-4B and Qwen3-8B are ideal for edge use, requiring only 6 to 12 GB of VRAM post-quantization.

That means Qwen3-8B runs on a MacBook Air M2 with 8 GB unified memory. DeepSeek R1 distills are the practical comparison here, not the full 671B model.

Licensing

Both are open source, but with different terms.

DeepSeek R1: MIT License. No thresholds, no restrictions. Commercial use, fine-tuning, and redistribution are fully open.
Qwen3: Apache 2.0 for models up to 35B parameters. If you use a larger Qwen model and reach 100 million monthly active users, Alibaba requires a separate commercial agreement. For smaller Qwen models, Apache 2.0 applies cleanly with no threshold.

For most teams, this distinction does not matter. Both are practically free to use.

When to Use Which

Use DeepSeek R1 distills when:

Your workload is purely math or formal logic
You want the MIT license with zero usage thresholds
You are fine with always-on reasoning latency and do not need a fast path
You have a 12 to 24 GB GPU and want specialized reasoning performance

Use Qwen3 when:

You need to switch between fast responses and deep reasoning based on the task
You want a model family that scales from 0.6B on edge devices up to 235B
You are building agents with tool use (the Qwen-Agent framework makes this clean)
You need strong multilingual support. Qwen3 covers 119 languages; DeepSeek covers roughly 30.
Code quality and output structure matter to you

Final Verdict

DeepSeek R1 lit the fire for open-source reasoning. The distilled 14B and 32B variants are still excellent, especially for math-heavy tasks on limited hardware.

I currently run Qwen3-8B locally for quick analysis tasks and Qwen3-32B for anything that needs actual reasoning. For pure math puzzles, I still reach for an R1 distill.

Which one are you running on your local setup? Are you using thinking mode, or keeping it off by default? Let me know in the comments.

Happy Coding!

Codex CLI vs Claude Code: A Deep-Dive Command Comparison

NaveenKumar Namachivayam ⚡ — Wed, 24 Jun 2026 14:50:03 +0000

In this blog post, we will see how the two most talked-about AI coding CLIs, OpenAI's Codex CLI and Anthropic's Claude Code, stack up command by command. Not just the headline features, but the small wins, the gaps, the uncommon flags, and the places where one clearly pulls ahead. Everything here is sourced directly from the official docs.

Codex CLI vs Claude Code CLI commands: what's the difference?Both are agentic terminal coding tools, but their command surfaces diverge significantly.If you need deep CI integration and multi-agent pipelines, choose Claude Code. If you need local models or a richer TUI experience, choose Codex CLI.

Quick Context

Both tools are agentic coding CLIs that live in your terminal. They read codebases, edit files, run shell commands, and talk to external services over MCP. The underlying models are different (Claude for Anthropic, GPT-family for OpenAI), but architecturally they are solving the same problem.

I have been using Claude Code daily as part of my performance engineering work and plugin development. I recently started exploring Codex CLI seriously after OpenAI formalized its docs under developers.openai.com. This post is the comparison I wish I had when I started.

Installation at a Glance

Claude Code:

npm install -g @anthropic-ai/claude-code
claude auth login

Codex CLI:

# macOS / Linux
curl -fsSL https://chatgpt.com/codex/install.sh | sh

# Or via npm
npm i -g @openai/codex

codex login

Both need Node.js. Claude Code requires an Anthropic account (Claude subscription or API key). Codex CLI authenticates via ChatGPT OAuth or an OpenAI API key.

Core Commands Side by Side

Here are the foundational commands every developer uses daily.

Task	Claude Code	Codex CLI
Start interactive session	`claude`	`codex`
Start with initial prompt	`claude "explain this project"`	`codex "explain this project"`
Non-interactive one-shot	`claude -p "query"`	`codex exec "query"` (alias: `codex e`)
Pipe content	`cat logs.txt \| claude -p "explain"`	`codex exec - < logs.txt`
Continue last session	`claude -c`	`codex resume --last`
Resume by name/ID	`claude -r "auth-refactor" "query"`	`codex resume <SESSION_ID>`
Update CLI	`claude update`	`codex update`
Auth login	`claude auth login`	`codex login`
Auth logout	`claude auth logout`	`codex logout`
Auth status	`claude auth status`	`codex login status`
Configure MCP	`claude mcp`	`codex mcp`
Manage plugins	`claude plugin`	`codex plugin marketplace`
Fork a session	`claude --fork-session --resume <id>`	`codex fork`

Both tools have non-interactive modes perfect for CI pipelines. Claude Code uses -p (print mode). Codex CLI uses exec as a proper subcommand with its own flag surface.

Commands Only in Claude Code

Claude Code has a significantly deeper command surface for background agent management. These are commands with no Codex equivalent.

Background Agent Management

# Start as a background agent and return to prompt immediately
claude --bg "investigate the flaky test"

# Attach to a background session
claude attach 7c5dcf5d

# See logs from a background session
claude logs 7c5dcf5d

# Stop a background session
claude stop 7c5dcf5d

# Restart a background session (picks up updated binary)
claude respawn 7c5dcf5d

# Remove from the list (transcript stays on disk)
claude rm 7c5dcf5d

This is the biggest functional gap in Codex right now. Claude Code has a full background session supervisor with claude daemon status and claude daemon stop --any. You can run multiple agents in parallel, attach and detach, and inspect each session's recent output with claude logs.

Daemon Management

# Check the background supervisor's state
claude daemon status

# Stop the supervisor (keep workers running to reconnect later)
claude daemon stop --any --keep-workers

Project State Management

# Preview what would be deleted
claude project purge ~/work/repo --dry-run

# Delete all local Claude Code state for a project
claude project purge ~/work/repo -y

This cleans up transcripts, task lists, debug logs, file-edit history, and prompt history. Useful when onboarding a project fresh or cleaning up stale state.

Ultrareview

# Run ultrareview on a PR non-interactively
claude ultrareview 1234 --json

Codex does have a /review slash command inside sessions, but claude ultrareview is a standalone CI-friendly command that exits with 0 on success and 1 on failure.

Remote Control

# Start a remote control server so you can control Claude Code from claude.ai
claude remote-control --name "My Project"

# Or start an interactive session with remote control enabled
claude --remote-control "My Project"

# Resume a web session in your local terminal
claude --teleport

This is a genuinely unique capability. You start a session locally, expose it over Remote Control, and then control it from claude.ai or the mobile Claude app. No Codex equivalent exists.

Long-Lived Token for CI

# Generate a long-lived OAuth token for CI pipelines
claude setup-token

Codex uses a different CI flow (piping API key via stdin with codex login --with-api-key).

Install Specific Version

claude install 2.1.118
claude install stable
claude install latest

Commands Only in Codex CLI

Cloud Task Management

# Browse cloud tasks from the terminal
codex cloud

# Submit a cloud task directly
codex cloud exec --env ENV_ID "fix the auth bug"

# List recent tasks with JSON output
codex cloud list --json --limit 10

# Apply a cloud task diff to your local working tree
codex apply TASK_ID

This is Codex's hybrid cloud-plus-local model. You can kick off tasks in the Codex cloud environment and then codex apply their diffs locally. Claude Code has remote web sessions but not this apply-a-cloud-diff pattern.

Sandbox Helper

# Run a command inside Codex's sandboxing layer (macOS Seatbelt)
codex sandbox --permissions-profile my-profile -- pytest tests/

# Log sandbox denials for debugging
codex sandbox --log-denials -- npm test

You can test what commands Codex allows or denies before committing a config. Very useful for security-conscious teams.

Exec Policy Testing

# Check whether a command would be allowed, prompted, or blocked
codex execpolicy --rules ~/.codex/rules/my-policy.rules --pretty -- git push

# Validate rules before saving them
codex execpolicy -r policy.rules -r another.rules -- rm -rf /tmp/junk

This is a preview feature that lets you unit-test your execution policy files. Nothing like this exists in Claude Code.

Shell Completion Scripts

# Generate completions for Zsh
codex completion zsh > "${fpath[1]}/_codex"

# Generate for Bash, Fish, PowerShell, Elvish
codex completion bash
codex completion fish
codex completion power-shell
codex completion elvish

Claude Code does not have a completion command. You get whatever your shell discovers from the binary.

Feature Flag Management

# List all feature flags with maturity and current state
codex features list

# Persistently enable a feature
codex features enable subagents

# Persistently disable a feature
codex features disable experimental-network

Claude Code exposes betas via --betas but does not have a persistent feature flag manager as a first-class CLI command.

Debug Model Catalog

# Print the raw model catalog Codex sees
codex debug models

# Show only the bundled catalog (no remote refresh)
codex debug models --bundled

Useful when troubleshooting model availability or provider routing issues.

Run Codex as an MCP Server

# Expose Codex itself as an MCP tool for other agents to consume
codex mcp-server

This is a powerful composition pattern. Another agentic tool (including Claude Code) can talk to Codex over MCP. I have not seen Claude Code offer an equivalent claude mcp-server command.

Launch Desktop App from CLI

# Open Codex Desktop app, pointing at a workspace
codex app ~/work/my-project

Flags Compared

Shared Flags (Different Names)

Concept	Claude Code	Codex CLI
Model selection	`--model claude-sonnet-4-6`	`--model gpt-4.1` / `-m gpt-5.4`
Extra directories	`--add-dir ../lib`	`--add-dir ../lib`
Non-interactive	`--print` / `-p`	`--json` on `exec` subcommand
Skip permissions	`--dangerously-skip-permissions`	`--dangerously-bypass-approvals-and-sandbox` / `--yolo`
Output format	`--output-format json`	`--json` on exec

Flags Only in Claude Code

# Run with minimal setup (no hooks, skills, MCP, CLAUDE.md)
claude --bare -p "query"

# Set reasoning effort
claude --effort high
claude --effort max

# Append to system prompt without replacing it
claude --append-system-prompt "Always use TypeScript"
claude --append-system-prompt-file ./style-rules.txt

# Validated JSON output matching a JSON Schema
claude -p --json-schema '{"type":"object"}' "query"

# Budget cap for API-billed sessions
claude -p --max-budget-usd 5.00 "query"

# Limit agentic turns
claude -p --max-turns 3 "query"

# Auto-connect to IDE on startup
claude --ide

# Spin up an isolated git worktree
claude -w feature-auth
claude -w feature-auth --tmux

# Resume from a PR number
claude --from-pr 123

# Select a fallback model chain
claude --fallback-model sonnet,haiku

# Screen reader accessible output
claude --ax-screen-reader

# Improve prompt cache reuse across CI runs
claude -p --exclude-dynamic-system-prompt-sections "query"

# Set session display name
claude -n "my-feature-work"

# Load plugin for session only
claude --plugin-dir ./my-plugin
claude --plugin-url https://example.com/plugin.zip

# Disable all slash commands and skills
claude --disable-slash-commands

# Start in safe mode (all customizations disabled)
claude --safe-mode

# Define subagents inline
claude --agents '{"reviewer":{"description":"Reviews code","prompt":"You are a code reviewer"}}'

# Enable advisor tool with a specific model
claude --advisor opus

# Start as a background agent immediately
claude --bg "investigate the flaky test"

# Run a shell command as a PTY-backed background job
claude --bg --exec 'pytest -x'

# Teammate display mode
claude --teammate-mode tmux

# Permission mode
claude --permission-mode plan
claude --permission-mode auto
claude --permission-mode acceptEdits
claude --permission-mode bypassPermissions

# Chrome browser integration
claude --chrome

Flags Only in Codex CLI

# Use local Ollama model
codex --oss

# Switch approval behavior
codex --ask-for-approval on-request

# Attach images to the initial prompt
codex --image screenshot.png "why is this broken?"
codex -i wireframe.png,design.png "implement this"

# Load a named config profile
codex --profile ci

# Enable live web search
codex --search

# Select sandbox policy
codex --sandbox workspace-write

# Connect TUI to a remote app-server
codex --remote ws://192.168.1.10:8080

# Set working directory for the agent
codex --cd /path/to/project "run tests"

# Override a config value inline
codex -c model=gpt-4.1 "query"
codex -c features.subagents=true "query"

# Disable alternate TUI screen
codex --no-alt-screen

The --oss flag is a genuine differentiator. Codex CLI supports pointing at a local Ollama instance for offline or privacy-sensitive work. Claude Code does not have this.

The --image / -i flag at the global level is very ergonomic. In Claude Code, you can reference images inside sessions, but it is not a global flag on the CLI launch itself.

Slash Commands Face-off

Both tools have in-session slash commands. Here is how the key ones map.

Present in Both (Similar Purpose)

Command	Claude Code	Codex CLI
Model switching	`/model`	`/model`
Compact context	`/compact`	`/compact`
New conversation	`/new`	`/new`
Resume session	`/resume`	`/resume`
Fork conversation	`/fork`	`/fork`
Exit	`/quit`	`/quit`, `/exit`
Init project file	`/init` (CLAUDE.md)	`/init` (AGENTS.md)
MCP tools	`/mcp`	`/mcp`
Session status	`/context`, `/cost`, `/stats`	`/status`

Slash Commands Only in Claude Code

/compact "Focus on the auth module and current test failures"
/output-style Explanatory
/output-style my-custom-style
/insights           # compiles past month of usage into an HTML report
/add-dir ../lib     # add working directory mid-session
/rename             # rename the current session
/export             # export conversation as plain text
/terminal-setup     # activate keyboard shortcuts for your terminal
/cost               # how much have I spent? (API users)
/stats              # how much have I used? (Pro/Max users)
/extra-usage        # configure what happens when you hit rate limit

The /insights command is genuinely impressive. It reads your last month of usage history and compiles it into a detailed HTML report. I have not found anything like it in Codex.

The /output-style system lets you define named styles in .claude/commands/ and switch between them. This is a powerful content-shaping tool for teams.

Slash Commands Only in Codex CLI

/goal "Finish the migration and keep tests green"  # set a persistent task goal
/goal pause        # pause the goal tracking
/goal resume       # resume it
/personality pragmatic  # set communication style (friendly/pragmatic/none)
/fast on           # toggle Fast service tier
/plan              # switch to plan mode
/side "is there an obvious risk here?"  # start an ephemeral side conversation
/btw "quick thought"   # alias for /side
/approve           # approve a denied auto-review action and retry
/memories          # configure memory injection and generation
/skills            # browse and use skills
/apps              # browse connectors and insert into prompt
/plugins           # browse installed/discoverable plugins
/hooks             # view and manage lifecycle hooks
/archive           # archive session and exit
/delete            # permanently delete session and exit
/copy              # copy latest response to clipboard (Ctrl+O also works)
/diff              # show Git diff including untracked files
/experimental      # toggle experimental features persistently
/vim               # toggle Vim mode for the composer
/keymap            # remap TUI keyboard shortcuts
/raw               # toggle raw scrollback mode
/review            # ask Codex to review your working tree
/ps                # show background terminals and recent output
/stop              # stop all background terminals
/debug-config      # print config layer diagnostics
/statusline        # configure TUI footer items interactively
/title             # configure terminal window/tab title items
/theme             # choose a syntax-highlighting theme
/permissions       # adjust approval policy mid-session
/ide               # pull IDE context (open files, selection) into prompt
/usage daily       # show daily token usage
/usage weekly
/usage cumulative
/feedback          # send diagnostics to OpenAI
/import            # import Claude Code setup into Codex
/sandbox-add-read-dir C:\path  # grant sandbox read access (Windows only)
/agent             # switch active agent thread
/goal              # persistent task goal tracking

The /side command is something I wish Claude Code had. You can start an ephemeral side conversation to ask a quick focused question without polluting the main thread's transcript. You type /side "check if this plan has an obvious flaw", get your answer, and return to the main task. Brilliant.

The /goal command gives Codex persistent objective tracking during long-running tasks. You set a goal and the agent keeps it in view across multiple turns.

The /personality command lets you shift Codex's communication style between friendly, pragmatic, and none without changing your instructions. Small win, but very practical when switching between debugging and documentation tasks.

Uncommon Commands Worth Knowing

These are the commands that most people miss but deliver real value once you discover them.

Claude Code: `--exclude-dynamic-system-prompt-sections`

claude -p --exclude-dynamic-system-prompt-sections "run the test suite"

This moves per-machine dynamic sections (working directory, environment info, memory paths) into the first user message instead of the system prompt. The result is better prompt cache reuse across different users and machines running the same task. Essential for teams running Claude Code in shared CI environments.

Claude Code: `--bare`

claude --bare -p "explain this function"

Skips auto-discovery of hooks, skills, plugins, MCP servers, auto memory, and CLAUDE.md. Sessions start significantly faster. Useful for quick scripted calls where you do not need any project configuration.

Claude Code: `--from-pr`

claude --from-pr 123
claude --from-pr https://github.com/owner/repo/pull/123

Resumes sessions linked to a specific pull request. Sessions get linked automatically when Claude creates the PR. Supports GitHub, GitHub Enterprise, GitLab, and Bitbucket URLs.

Claude Code: `--fallback-model`

claude --fallback-model sonnet,haiku -p "query"

Automatic fallback when the primary model is overloaded or unavailable. Accepts a comma-separated list tried in order. You can persist a chain via the fallbackModel setting.

Codex CLI: `codex execpolicy`

codex execpolicy --rules ~/.codex/rules/production.rules --pretty -- git push origin main

A policy dry-run. You pass your .rules files and a command, and Codex tells you whether it would allow, prompt, or block that command. This is a fantastic tool for validating security policy before deploying to CI.

Codex CLI: `--oss`

codex --oss "refactor this module"

Points Codex at a locally running Ollama instance. No API calls, no data leaving your machine. Validates that Ollama is running before starting.

Codex CLI: `codex apply`

codex apply TASK_ID

Applies the latest diff from a Codex cloud task to your local working tree. The workflow is: run a task in the cloud environment, review the result on the web, then pull the diff locally with one command. Performance engineers who run long test analysis tasks in cloud environments will appreciate this.

Codex CLI: `/side` and `/btw`

/side "does this API response shape match our schema?"

An ephemeral fork of the current conversation. The side thread has its own transcript. The parent thread's status stays visible in the TUI while you are in side mode. Type your quick question, get the answer, return. This is a quality-of-life feature I would happily see in Claude Code.

Claude Code: `claude ultrareview`

claude ultrareview 1234 --json --timeout 60

Runs a deep code review non-interactively. Prints findings to stdout and exits 0 on success or 1 on failure. Pipe it into your CI gate.

Small Wins: Category by Category

Session Management: Claude Code Wins

Claude Code has a richer session management surface. Background agents with a daemon supervisor, claude logs, claude attach, claude respawn, claude rm, and the claude agents view for monitoring and dispatching parallel sessions. Codex has codex resume and codex fork, which cover the basics but stop there.

Sandbox Control: Codex CLI Wins

Codex's sandbox story is more explicit. You choose read-only, workspace-write, or danger-full-access at the flag level. The codex sandbox command lets you run arbitrary commands inside Codex's sandbox layer to test policies. The codex execpolicy command lets you validate rules before saving them. Claude Code has permission modes (plan, auto, acceptEdits, bypassPermissions) but does not expose the underlying sandbox policy as a testable surface.

Image Input: Codex CLI Small Win

codex --image ui-screenshot.png "why is this button misaligned?"
codex -i wireframe.png,mockup.png "implement this layout"

Codex CLI accepts images as a global flag at session launch. You can attach multiple images with a comma-separated list. Claude Code supports image input inside sessions (drag and drop in the TUI or pasting), but --image is not a CLI launch flag.

CI Scripting: Claude Code Wins

Claude Code has more CI-specific flags: --max-budget-usd caps API spend, --max-turns limits agentic turns, --no-session-persistence avoids writing to disk, --output-format stream-json gives structured streaming output, --include-hook-events and --include-partial-messages allow fine-grained pipeline observability. The claude setup-token command generates long-lived OAuth tokens for CI authentication without a browser.

Codex has codex exec --ephemeral to skip session persistence and codex exec --output-last-message to write the final response to a file, which is handy in GitHub Action pipelines.

Local Model Support: Codex CLI Wins

codex --oss with Ollama support is a genuine differentiator. If you work in an air-gapped or privacy-sensitive environment, Codex CLI has a path. Claude Code currently has no equivalent.

Context Management: Claude Code Wins Slightly

Claude Code's /compact accepts focus instructions:

/compact Focus on the auth module and current test failures

Codex's /compact summarizes the conversation without focus parameters. Also, Claude Code's /insights command compiling a usage HTML report has no Codex equivalent.

Plugin Architecture: Codex CLI More Explicit

Codex has a proper codex plugin marketplace command for managing plugin marketplace sources from Git repos or local directories. You can pin refs and use sparse checkouts. Claude Code has claude plugin install against a marketplace, but the marketplace management surface is thinner at the CLI level.

Remote Work: Both Have Unique Angles

Claude Code has claude remote-control, which lets you control a local terminal session from claude.ai or the mobile app. Codex CLI has --remote ws://host:port, which connects a local TUI to a remote codex app-server. Different models of remote work, both useful depending on your setup.

What Is Missing in Each Tool

Missing in Claude Code

codex --oss style local model support (Ollama)
codex completion for shell completion scripts
codex execpolicy for policy dry-runs
codex sandbox for testing sandbox behavior
codex cloud and codex apply for cloud task management
/side for ephemeral side conversations
/goal for persistent task objective tracking
/personality for communication style control
/fast for service tier switching
/diff as a slash command (Claude Code does have git awareness, but not as a quick slash command)
/keymap for interactive keyboard remapping
/theme for syntax highlighting selection
/statusline and /title for TUI customization
--image as a launch-time CLI flag
/approve for retrying auto-review denials
codex features for persistent feature flag management
codex debug models to inspect model catalog

Missing in Codex CLI

Background session management at the daemon level (claude daemon, claude attach, claude logs, claude respawn)
claude ultrareview as a standalone CI command
claude remote-control to control terminal sessions from the web/mobile app
claude --teleport to bring a web session back to the local terminal
claude --from-pr to resume sessions linked to a specific PR
claude setup-token for long-lived CI tokens
claude project purge for clean project state management
claude --worktree and --tmux for isolated git worktrees
claude --advisor for the server-side advisor tool
--effort levels (low/medium/high/xhigh/max)
--fallback-model chains
--bare for minimal fast-start scripted sessions
--exclude-dynamic-system-prompt-sections for prompt cache optimization
--json-schema for validated structured output
--max-budget-usd for spend caps
System prompt control flags (--system-prompt, --system-prompt-file, --append-system-prompt, --append-system-prompt-file)
/insights usage history report
/output-style named output personas
/rename for session naming mid-session
/export conversation to plain text
Custom commands via .claude/commands/

What is the difference between Codex CLI and Claude Code?

Both are AI-powered terminal coding agents. Claude Code is built by Anthropic and runs Claude models. Codex CLI is built by OpenAI and runs GPT-family models. They share core interactive and non-interactive modes but differ significantly in background agent management, sandbox control, CI flags, and TUI features.

Does Codex CLI support local models?

Yes. Codex CLI supports local Ollama models via the --oss flag. This runs the agent without any API calls. Claude Code does not have an equivalent local model flag.

Does Claude Code support background agents?

Yes. Claude Code has a full background agent system with claude --bg, claude attach, claude logs, claude respawn, claude rm, and a daemon supervisor managed via claude daemon status and claude daemon stop. Codex CLI does not have an equivalent daemon-managed background session infrastructure.

Which CLI is better for CI pipelines?

Claude Code has more CI-focused flags including --max-budget-usd for spend caps, --max-turns to limit agentic turns, --no-session-persistence, --json-schema for validated structured output, and claude setup-token for long-lived OAuth tokens. Codex CLI offers codex exec with --ephemeral and --output-last-message, and a native GitHub Action.

What commands are missing in Codex CLI compared to Claude Code?

Codex CLI is missing background session management (claude daemon, claude attach, claude logs), claude ultrareview for CI code review, claude remote-control for web and mobile session control, --from-pr to resume sessions linked to a PR, --worktree for isolated git worktrees, --fallback-model chains, --max-budget-usd spend caps, and the /insights slash command.

What commands are missing in Claude Code compared to Codex CLI?

Claude Code is missing local model support via Ollama, codex completion for shell completion scripts, codex execpolicy for sandbox policy dry-runs, the /side slash command for ephemeral side conversations, /goal for persistent task objective tracking, /personality for communication style switching, /theme for syntax highlighting, and --image as a launch-time flag.

My Take

Both tools are genuinely capable. My honest observation after going through every documented command:

Claude Code has a deeper background agent infrastructure. If you are building multi-agent pipelines, running parallel workloads, or need tight CI integration with structured outputs, Claude Code's flag surface and daemon management are hard to beat.

Codex CLI wins on local model flexibility, sandbox policy control, and the TUI experience. The /side command, /goal tracking, and /personality switching feel like thoughtful UX investments. The codex execpolicy command for policy dry-runs shows a security-first mindset.

What I personally want to see: Claude Code adopt --image as a launch flag and a /side equivalent. Codex CLI needs a proper background daemon for parallel agents and a --max-budget-usd style spend cap for CI use.

Pick your tool based on your model preference first, then your workflow. If you need remote session control or deep CI scripting, lean Claude Code. If you need local model support or prefer a more granular TUI, lean Codex CLI.

Have you switched between both tools on the same project? I would love to know which commands you reach for first. Drop a comment below.

Happy Testing!

Toy Story: The Open-Source Ecosystem

NaveenKumar Namachivayam ⚡ — Fri, 19 Jun 2026 16:41:20 +0000

As schools are off and Toy Story 5 is just around the corner, we started binge-watching Toy Story from 1 to 4. While watching, suddenly this idea popped up: what if a GitHub repo came alive just like the toys? I started writing with something basic and enhanced it using Gemini Flash. Hope you'll like it.

The Setup: The Developer's Stack

The "Room" is the ultimate production stack. The classic, dependable tools that every developer loves and relies on.

Woody (python/cpython): The beloved, classic, highly readable leader of the repo ecosystem. He’s dependable, has been around forever, and is the favorite of the developer. He prides himself on clean architecture and readability.
Rex (apache/jmeter): A massive, heavy-duty Java performance testing tool. He’s incredibly powerful but constantly anxious that modern, lightweight tools are going to make him look extinct.
Mr. Potato Head (docker/cli): The ultimate container tool. You can literally swap his volumes, environment variables, and ports around to make him look like whatever you want.
Slinky (lodash/lodash): The utility tool that just exists to stretch and connect different data structures together smoothly.

They all live in harmony on the machine, until a massive update drops...

The Inciting Incident: The Trendy New Framework

The developer is starting a massive new enterprise cloud project. Suddenly, a sleek, shiny new arrival lands in the ecosystem with over 100k GitHub stars in its first week.

Enter Buzz Lightyear (facebook/react).

Buzz is high-tech, component-based, and completely delusional. He doesn’t realize he’s just an open-source library running on a local runtime. He genuinely believes he is a Space Ranger from Vercel deployed to the Edge Network. He looks at the backend scripts and declares he will build a Virtual DOM to save the galaxy.

Woody (cpython) is furious. "You aren't a full-stack engine! You're a frontend library! You're an npm package!" But the developer keeps starring react, opening its issues, and ignoring python scripts.

The Interlude: Lost in Pizza Planet

In a heated argument over package management, Woody accidentally bumps Buzz out of the active IDE workspace. The other repos accuse Woody of a malicious git rm. Determined to patch things over, Woody chases Buzz out of the environment.

They end up stranded at Pizza Planet a massive, chaotic public multi-tenant cluster. Hungry for a way back to a developer's machine, Buzz spots a glowing, neon structure: a massive monorepo cluster masquerading as a claw machine game.

They climb inside, landing in a sea of hundreds of identical, tiny, lightweight Docker Microcontainers (alpine-linux/mini-images). They sit huddled together in their namespace pods, completely identical, staring upward in wonder.

The Microcontainers: (In unison, staring at the cluster orchestrator) "Oooooooooh... The OpenClawwww."

Buzz: "Who is in charge here?"

Microcontainer #1: "The OpenClaw! It is an open-source automation engine. It hooks into our webhooks and schedules our lifecycles."

Suddenly, a heavy, automated crane mechanism descends from the top of the repository cluster.

Microcontainer #2: "The OpenClaw moves! It has selected a container!"

Microcontainer #3: "I have been chosen! I am being scheduled to a high-availability EC2 node! Farewell, my friends, I go to a better place... Production!"

Before Woody and Buzz can escape the cluster, Sid (malicious-npm-bot) a chaotic script-kiddie developer playing on the cluster drops a malicious token into the machine. The OpenClaw descends, but instead of a container, its mechanical hook snags Woody and Buzz, dropping them right into Sid's dark dependency backpack.

The Climax: The Dark Web of Dependency Hell

Sid’s machine is a chaotic nightmare of dependency hell. He takes famous repos, strips their licenses, injects malware, and bundles them into mutated, broken franken-packages. He has strapped a volatile crypto-miner to Buzz, intending to deploy him to an unsecured AWS bucket.

Woody realizes he can't save the day alone. He rallies Sid’s mutated, broken open-source forks. They break the prime directive of software: they execute without being called by a command line. They glitch out Sid's IDE, spamming his screen with endless Deprecated warnings and breaking changes until he panics, shuts down his PC, and goes outside.

The Resolution: The Great Git Push

Woody and Buzz race back to the developer's main machine, but the developer is in the middle of a massive migration. He is running a script to push his entire workspace to a new cloud organization.

The migration truck is leaving! Woody and Buzz missed the initial commit. They scramble to find a way into the push. They spot a fast, high-velocity transport stream: curl running over a high-speed fiber connection.

They hitch a ride on a webhook, but the payload is too heavy. Buzz throws Woody ahead into the repository, sacrificing himself to an asynchronous timeout. Woody refuses to lose his friend. He grabs a gzip compression rocket, ignites it, sweeps down, grabs Buzz, and they soar through the pipeline.

They don't just land in the repo; they land right at the top of the main branch, fully compiled and perfectly integrated.

Post-Credits Scene:

cpython and react are now happily co-existing in a beautiful Django-React stack. Suddenly, the developer runs an installation command for a new repo that just dropped: microsoft/autogen.

An army of autonomous AI Agents floods the repository.

Buzz: "Woody, look! Multi-agent orchestration!"

Woody: (Gulp) "Great..."

THE STORY IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR ANY MERGE CONFLICTS, BROKEN DEPENDENCIES, OR EXISTENTIAL CRISES EXPERIENCED BY YOUR LOCAL SCRIPTS AFTER READING. 

Toy Story is © Disney/Pixar. All featured repositories belong to their rightful maintainers.

JMeter vs k6 vs Locust in 2026: Which Load Testing Tool Should You Pick?

NaveenKumar Namachivayam ⚡ — Thu, 18 Jun 2026 20:33:01 +0000

In this blog post, we will see a detailed, grounded comparison of the three most debated open-source load testing tools in 2026: Apache JMeter, Grafana k6, and Locust. All three are free. All three are production-proven. Yet they could not be more different in philosophy, architecture, and day-to-day experience.

I have worked with all three across real-world projects, from legacy JDBC-heavy enterprise systems at work to lightweight microservice pipelines I test for my own side projects. The honest truth? There is no universal winner. But there is almost always a right answer for your specific situation, and that is what we will figure out today.

Why This Comparison Still Matters in 2026

Every year someone writes "JMeter is dead." Every year JMeter ships another release and shows up in another enterprise RFP.

The market has not consolidated. Instead, it has stratified. k6 owns the developer-experience conversation. Locust owns the Python ecosystem. JMeter owns the protocol breadth and enterprise legacy. And in 2026, all three have meaningful updates worth knowing about before you pick a tool for your next project.

Let me give you the ground truth, not marketing copy.

Quick Stats at a Glance

	Apache JMeter	Grafana k6	Locust
Language	Java (GUI + XML)	Go runtime, JS/TS scripts	Python
Latest Version	5.6.3	2.0.0 (May 2026)	Latest on PyPI (May 2026)
GitHub Stars	~9.4k	~30.8k	~27.9k
License	Apache 2.0	AGPL-3.0	MIT
Concurrency Model	Thread per VU	Go goroutine per VU	gevent greenlet per VU
Protocol Breadth	Excellent (HTTP, JDBC, JMS, LDAP, MQTT, FTP...)	Good (HTTP, gRPC, WebSocket)	Good (HTTP, extensible via Python libs)
CI/CD Fit	Good	Excellent	Good
GUI	Yes (built-in)	k6 Studio (separate app)	Web UI (live stats only)
Cloud Option	BlazeMeter, OctoPerf	Grafana Cloud k6	Self-managed
Best For	Multi-protocol, legacy enterprise	Modern APIs, developer teams	Python shops, flexible scripting

Apache JMeter

JMeter was first released in 1998. That is not a typo. It turned 27 this year, and it is still actively maintained under the Apache Software Foundation.

The latest stable release is 5.6.3. It requires Java 17 as the recommended runtime, and the team has already signaled that the next major version will drop Java 8 support entirely.

What JMeter Gets Right

JMeter's superpower is protocol coverage. Nothing else on this list comes close.

HTTP / HTTPS
JDBC (database connection testing)
JMS
LDAP
MQTT
FTP
TCP

If you are testing a legacy enterprise system, a mainframe-adjacent API, or a backend that talks over JDBC, JMeter is often the only open-source option that handles it natively.

The plugin ecosystem also deserves credit. The JMeter Plugins project (Head to https://jmeter-plugins.org) adds over 60 additional components. I have built and maintain several commercial plugins of my own, and the extensibility is genuinely solid once you understand the architecture.

Where JMeter Struggles in 2026

The XML-based .jmx test plan format is the biggest pain point in a modern team. Git diffs on .jmx files are nearly unreadable. Code review for JMeter scripts is painful. "Load testing as code" with JMeter is possible but requires discipline and tooling that does not come out of the box.

The thread-per-user concurrency model also means JMeter is resource-hungry at scale. A single machine can generate fewer concurrent users than k6 or Locust on equivalent hardware. For large-scale tests, you need distributed mode or a cloud platform like BlazeMeter, which starts around $149/month for the basic plan.

The GUI, while powerful, shows its age next to k6 Studio or even Locust's minimal web interface.

You can check Feather Wand if you want to infuse AI in your workflow. To measure the speed of LLM, you can check iamspeed.dev.

Personal Observation

I was using JMeter daily at Salesforce for MuleSoft API performance testing. The GUI is genuinely useful for building complex request chains quickly. But the moment I need to commit a test plan to Git and do a proper review, it becomes painful.

Grafana k6

k6 is the most talked-about load testing tool in 2026, and the GitHub star count (30.8k at the time of writing) reflects that.

Two major milestones happened back to back: k6 v1.0 dropped in May 2025 with TypeScript support, native extensibility without custom build pipelines, and SemVer stability guarantees. Then k6 v2.0.0 shipped on May 11, 2026, and it changed the game again.

What k6 2.0 Brought

The headline feature in k6 2.0 is AI-assisted testing workflows. This is not a gimmick. The release ships four new commands built specifically for agent-friendly development:

k6 x agent: bootstraps agentic testing workflows inside Claude Code, Codex, Cursor, and other AI coding assistants
A built-in Model Context Protocol (MCP) server so AI agents can validate and run scripts, inspect results, and iterate without leaving the session
k6 x docs: gives agents and developers CLI access to k6 documentation and examples
k6 x explore: lets agents browse the extension registry from the CLI

There is also a new Assertions API, broader Playwright compatibility in the browser module, and a consolidated extension catalog that merges official and community extensions into one place.

What k6 Gets Right

The scripting experience is genuinely great for developers. You write JavaScript or TypeScript. Your IDE gives you autocomplete. Your CI pipeline runs it as a single binary with no JVM to provision.

import http from 'k6/http';
import { check, sleep } from 'k6';

export const options = {
  vus: 100,
  duration: '30s',
  thresholds: {
    http_req_duration: ['p(95)<500'],
    http_req_failed: ['rate<0.01'],
  },
};

export default function () {
  const res = http.get('https://api.example.com/health');
  check(res, {
    'status is 200': (r) => r.status === 200,
  });
  sleep(1);
}

k6 Studio (v1.13.1) is a desktop GUI with AI-powered auto-correlation. If you record a browser session, k6 Studio detects dynamic values like session tokens and CSRF tokens and generates correlation rules automatically. That is a feature JMeter has had for years via plugins, but k6 Studio does it through AI, without the XML.

Where k6 Struggles

Protocol coverage is more limited than JMeter. k6 is strong on HTTP, gRPC, and WebSocket. For JDBC, JMS, or LDAP, you are looking at community extensions or custom solutions.

The AGPL-3.0 license is also worth flagging for commercial use cases. Check with your legal team if you are embedding k6 in a product.

Personal Observation

I built iamspeed.dev (an LLM streaming benchmarker) and used k6 for the load side. The DX was excellent. TypeScript types in the IDE, a clean CLI, and Grafana integration out of the box. For any API-heavy workload where the protocol is HTTP or gRPC, k6 is my first recommendation in 2026.

Locust

Locust is the load testing tool for Python teams, and the May 2026 PyPI release confirms the project is alive and growing. It now officially supports Python 3.10 through 3.14.

What Locust Gets Right

Locust's model is simple: write Python classes that describe user behavior, run the tool, watch the web UI. No DSL to learn. No XML. No JVM.

from locust import HttpUser, task, between

class APIUser(HttpUser):
    wait_time = between(1, 3)

    @task(3)
    def get_products(self):
        self.client.get("/api/products")

    @task(1)
    def get_health(self):
        self.client.get("/health")

Under the hood, Locust uses gevent greenlets instead of OS threads. This gives it excellent concurrency density. On the same 8 GB machine, Locust can handle roughly 5x more concurrent users than JMeter, according to TestDevLab's 2026 analysis.

Because test files are plain Python, extending Locust to custom protocols is straightforward. Need to load test a proprietary queue or an LLM inference endpoint? Wrap the Python client library and drop it into a HttpUser subclass. This is actually something I have done for AI workload benchmarking.

Distributed testing is built in. You run a master process and any number of worker processes, scale horizontally, and the web UI aggregates everything.

Where Locust Struggles

The built-in reporting is minimal. The web UI gives you live stats during the run, but there is no built-in HTML report comparable to JMeter's dashboard or Gatling's output. Most teams pipe Locust metrics into Grafana via InfluxDB or Prometheus.

There is no GUI for building test plans. Everything is code. That is great for developer teams but can be a barrier for non-technical stakeholders.

Personal Observation

Locust is my go-to tool when I am testing an LLM API or any endpoint where I need complex Python logic in the request flow, like computing HMAC signatures, calling a pre-step to generate tokens, or parsing streaming responses. The pure-Python model gives you the whole ecosystem to work with.

Head-to-Head Comparison

Scripting Experience

JMeter gives you a GUI that is powerful but dated. Building a test plan with the GUI is fast for HTTP. Building one for gRPC or WebSocket requires plugins and some patience.

k6 gives you a code editor and a TypeScript-aware test runner. The scripting is clean, the API is well-documented, and the extension ecosystem is growing fast.

Locust gives you a Python file. Nothing else to install. If your team already writes Python, the onboarding time is near zero.

Concurrency Model

This is where architecture matters for real.

JMeter runs one OS thread per virtual user. This is expensive. A mid-range machine typically maxes out around 300-500 concurrent threads before CPU and memory become the bottleneck, not the system under test.

k6 runs each VU as a Go goroutine. Goroutines are lightweight. k6 can drive thousands of concurrent VUs from a single machine.

Locust uses gevent greenlets, which are cooperative coroutines. Similar lightweight profile to goroutines. One machine can comfortably simulate thousands of users against an HTTP API.

CI/CD Integration

k6 wins this category cleanly. A single binary, no JVM, no Python dependency tree. The GitHub Actions integration is a config change. The threshold system lets you fail a pipeline based on p95 response time or error rate directly in the test script.

Locust integrates well with CI/CD through headless mode (locust --headless). You can define pass/fail criteria via exit codes and custom listeners.

JMeter needs more setup: a JVM, a plugin directory, a .jmx file committed to the repo, and some wrapper scripts to parse the output. It works, but it takes more effort to get right.

Reporting

JMeter ships a dynamic HTML report with response time graphs, latency percentiles, and error analysis. It is comprehensive out of the box.

k6 pushes metrics to Grafana natively (local or cloud), and the k6 2.0 summary is significantly improved over previous versions. For cloud runs, the Grafana Cloud k6 dashboard is excellent.

Locust's built-in report is minimal. Pipe to Grafana via Prometheus or InfluxDB for anything beyond a quick check.

Cloud Execution

	JMeter	k6	Locust
Managed Cloud	BlazeMeter ($149/mo+), OctoPerf	Grafana Cloud k6	None (self-managed)
Kubernetes	Manual setup	k6 Operator (official)	Manual setup
Distributed	Controller + agents via SSH	k6 cloud run / k6 Operator	Master + worker processes

The Metric Problem Nobody Talks About

This is something I always include when I write about load testing tools, because it catches teams off guard.

Run the same test against the same endpoint using JMeter and k6, and you will see different response time numbers. Not because one tool is wrong. Because they measure different slices of the request lifecycle.

JMeter starts the clock at the connection and stops when the last byte is received
k6 breaks response time into granular phases: http_req_connecting, http_req_tls_handshaking, http_req_waiting, http_req_receiving
Locust, using gevent, can report higher response times under certain connection reuse configurations

OctoPerf's comparative study showed up to 15-20% variance in reported response times between tools running identical load against the same target. The practical takeaway: never compare baselines across tools. Establish baselines inside a single tool and track trends there.

Which Tool Should You Choose?

Use this decision tree:

Choose JMeter if:

You are testing JDBC, JMS, LDAP, FTP, or SOAP endpoints
Your team uses GUI-driven test creation
You have an existing JMeter investment and plugin ecosystem
You work in enterprise environments where BlazeMeter or OctoPerf is already licensed

Choose k6 if:

Your stack is HTTP, gRPC, or WebSocket
Your team writes JavaScript or TypeScript
CI/CD integration is a first-class requirement
You want AI-assisted test authoring in 2026 (k6 2.0's MCP server is real and it works)
You want the best DX in the category right now

Choose Locust if:

Your team is already Python-first
You need deep customization of request logic (token generation, streaming parsing, custom protocols)
You are testing LLM APIs or AI workloads where the request logic is non-trivial
You want distributed testing without a managed cloud dependency

The Hybrid Stack Reality

Something the comparison articles rarely say: most mature teams run two tools.

The practical 2026 default stack looks like one of these:

k6 OSS for daily CI checks + Grafana Cloud k6 for quarterly capacity tests
JMeter locally for protocol-rich scenarios + BlazeMeter for distributed runs
Locust for API behavioral tests in Python + Prometheus/Grafana for dashboards

I have run exactly this kind of hybrid at QAInsights, using JMeter for the complex correlation scenarios and k6 for the lightweight API regression checks that live in CI. The tools complement each other more than they compete.

Final Verdict

There is no single best load testing tool in 2026. But there is a best tool for your context.

If you are starting from scratch on a modern microservices stack, pick k6. The DX is excellent, k6 2.0's AI integration is ahead of everyone else, and the Grafana ecosystem is mature.

If your Python team needs to write complex behavioral scripts, pick Locust. The gevent-based concurrency is efficient, the code is readable, and the Python ecosystem fills every gap.

If you are in an enterprise environment testing JDBC, JMS, or anything beyond HTTP, pick JMeter. The protocol breadth is unmatched in open source, and the plugin ecosystem solves problems that other tools have not even attempted.

What matters most is not which tool you pick. It is that you actually test under load before your users find the bottleneck for you.

Happy Testing!

What tool are you using in your current project, and what made you choose it over the alternatives? Drop your answer in the comments below.

I Built a Fast.com for LLMs: Introducing iamspeed.dev

NaveenKumar Namachivayam ⚡ — Wed, 17 Jun 2026 16:50:20 +0000

In this blog post, we will see how I built iamspeed.dev, a fast.com-style LLM API speed benchmark tool that measures Time to First Token (TTFT) and tokens-per-second throughput directly in your browser.

If you have ever stared at a spinning cursor waiting for an LLM response and wondered "is this slow, or is it just me?" this tool is for you.

The tool is designed for quick, lightweight benchmarking, modeled after the fast.com experience for internet speed tests. It uses an extensible provider adapter architecture, making it straightforward to add new providers such as Gemini or Groq. Planned additions include historical results, model comparison mode, and support for local models via Ollama.

iamspeed.dev is an open-source, browser-based benchmarking tool for LLM APIs that measures two key performance metrics: Time to First Token (TTFT) and tokens-per-second throughput. It supports OpenAI and Anthropic providers and stores API keys locally using AES-GCM encryption, with no backend or data transmission.

The Problem That Sparked This

I spend a lot of time benchmarking systems. Load testing APIs, profiling microservices, measuring throughput it is what I do at QAInsights and at my day job.

When LLMs started becoming part of production stacks, I noticed that most developers just eye-balled "it feels fast" or "it feels slow." There was no quick, browser-based tool you could open, configure your API key, and immediately get a concrete number.

Tools like artificial analysis do heavy-lifting comparisons across hundreds of models. But I wanted something lighter. Something you could open on a Tuesday afternoon and just run.

That is exactly how fast.com works for internet speed tests. You open it, it runs, you see a number. Done.

So I built the same thing for LLM APIs: iamspeed.dev.

Introducing iamspeed.dev

What Is iamspeed.dev?

iamspeed.dev is an open-source, browser-based benchmarking tool for LLM APIs. It streams live tokens from supported providers OpenAI and Anthropic today and shows you real-time performance metrics as they happen.

No backend. No data collection. No surprises.

Your API key is stored locally in your browser using AES-GCM encryption, meaning it never leaves your machine.

The interface is deliberately minimal, as shown below just a logo, a metric display, a Run button, and a settings panel. Inspired directly by the fast.com aesthetic.

Key Metrics: What Gets Measured

If you work with LLMs in production, you already know that raw response time is a misleading number. The two metrics that actually matter are:

1. Time to First Token (TTFT)

This is the time between sending your request and receiving the very first token back from the model. It reflects how quickly the LLM starts generating a response.

TTFT is what users feel. A high TTFT means that awkward pause before anything appears on screen.

For interactive applications, keeping TTFT low is critical. Reasoning models (extended thinking, deep think modes) can inflate TTFT by 5x to 30x because of the additional compute happening before the first visible token arrives.

2. Tokens Per Second (Throughput)

This is the rate at which the model streams tokens to you after the first one arrives. It is the "output speed" metric.

High tokens per second means the text appears fast and fluid on screen. Low throughput feels choppy and slow even if the TTFT was acceptable.

Together, these two numbers give you the full picture of how an LLM API performs for your use case.

Features at a Glance

Here is a quick summary of what iamspeed.dev supports:

Live streaming output with real-time metric updates
TTFT measurement captured precisely at the moment the first token arrives
Tokens/sec throughput tracking updated continuously during generation
AES-GCM encrypted API key storage local only, never transmitted
OpenAI provider support (GPT-4o, GPT-4.1, and compatible models)
Anthropic provider support (Claude Sonnet, Claude Haiku, and more)
Extensible provider architecture via a clean ProviderAdapter interface
Responsive minimal UI inspired by fast.com

The key thing I want to highlight is the local encryption. I have seen too many tools that ask for your API key and quietly send it somewhere. iamspeed.dev does not do that. Your key is AES-GCM encrypted and stored only in your browser's local storage.

The provider architecture is clean and intentional. Each LLM provider is implemented as an adapter that satisfies the ProviderAdapter interface. This makes adding new providers straightforward and keeps the core benchmark logic provider-agnostic.

The project is hosted at iamspeed.dev and the full source is available on GitHub.

How to Run It Locally

Running iamspeed.dev locally takes under a minute. Here are the steps:

Clone the repository:

git clone https://github.com/QAInsights/iamspeed.dev.git
cd iamspeed.dev

Install dependencies:

npm install

Start the development server:

npm run dev

Head to http://localhost:4321 in your browser.
Click the gear icon (Settings) and enter your OpenAI or Anthropic API key.
Hit Run.

You will immediately see the tokens streaming in and the tokens/sec counter updating live, as shown below.

Here are all the available commands:

Command	Description
`npm run dev`	Start the dev server
`npm run build`	Build for production
`npm run preview`	Preview the production build
`npm test`	Run unit tests (Vitest)
`npm run test:e2e`	Run E2E tests (Playwright)

How to Add a New Provider

This is where the architecture really shines. If you want to add support for, say, Gemini or Groq, the process is clean:

Create a new adapter file in src/lib/providers/. Your adapter must implement the ProviderAdapter interface.
Register it in src/lib/providers/index.ts.
Add the provider metadata (name, models, etc.) to src/lib/config.ts.

That is it. No changes to the benchmark engine, no changes to the UI logic. The adapter pattern keeps concerns separated cleanly.

I am planning to add more providers over time. If you want to contribute one, pull requests are welcome.

Why This Matters for Performance Engineers

I want to speak directly to performance engineers here for a second.

We are used to measuring systems with JMeter, k6, Gatling. We understand throughput, latency percentiles, concurrency, think time. LLM APIs add a new dimension to all of this.

When you are building an AI-powered product, you are not just measuring HTTP response time anymore. You are dealing with:

TTFT as a user-perceived latency metric (equivalent to time-to-interactive in web perf)
Streaming throughput as a sustained delivery rate (not a one-shot measurement)
Provider variability the same model can behave very differently across regions and time of day
Reasoning overhead thinking models add invisible compute time before the first visible token

Tools like iamspeed.dev give you a quick sanity check. Before you design a full performance test suite for your LLM-powered API, run a quick benchmark here to understand your baseline numbers.

I have written extensively about LLM performance metrics on the QAInsights blog and built the jmeter-llm-sampler plugin for measuring TTFT and TTLT in JMeter test plans. iamspeed.dev is the browser-friendly companion to those deeper tools.

What's Next

A few things I want to add to iamspeed.dev:

More providers: Gemini, Groq, Mistral, and local Ollama support
Historical results: Run multiple benchmarks and compare them over time
Model comparison mode: Run the same prompt across two models side by side
Shareable result links: Generate a URL you can share with your team
Prompt customization: Let you choose the input prompt length to simulate different workloads

If any of these sound useful to you, drop a star on the GitHub repo and let me know what you want to see first.

Try It Now

Head to iamspeed.dev, configure your API key in settings, and hit Run.

You will have your tokens-per-second number in about 10 seconds.

The source code is MIT licensed and available at github.com/QAInsights/iamspeed.dev. Contributions are open.

Happy Testing!

What LLM provider are you using in production today, and what TTFT are you seeing? Drop a comment below I would love to know how the numbers compare.

How I Use Qwen Code Slash Commands to Build Achu App

NaveenKumar Namachivayam ⚡ — Wed, 17 Jun 2026 03:49:08 +0000

In this blog post, we will see how I use Qwen Code's slash commands and workflow strategies to build Achu my screenshot beautifier app without burning through tokens or losing context mid-session.

If you haven't heard of Achu, it's a desktop app built with Electron + React + TypeScript. It does screenshot beautification, Privacy Guard (offline OCR redaction), Auto-Vibe (palette-extracted backgrounds), and an AI Bug Agent with GitHub integration. It's a side project I'm genuinely proud of, and Qwen Code has become my go-to agentic coding CLI for it.

A developer shares their day-to-day workflow for using Qwen Code, an open-source agentic coding CLI, to build Achu, a desktop screenshot beautification app built with Electron, React, and TypeScript. The post covers how slash commands like /init, /plan, /compress, /remember, and /btw are used to manage context, reduce token costs, and maintain consistent output across sessions.

The core approach centers on spec-driven planning through iterative /plan sessions before any code is written, combined with parallel subagents for independent tasks and strict context hygiene using /compress and /clear. Additional practices include pointing the model at library source code instead of documentation and using /remember to persist architectural decisions across sessions.

This isn't a tutorial about what Qwen Code is. It's about how I actually use it day-to-day, the slash command tricks I rely on, and the discipline it takes to get real work done with an LLM in a terminal.

It all started with Google Antigravity, but the 5 hours reset and weekly limits is killing my productivity and thinking flow. I had to switch to more affordable and open source model where I chose Qwen.

Why Qwen Code?

I've tried Claude Code, Gemini CLI, and a bunch of others. Qwen Code is open source, has excellent subagent support, a rich slash command system, and Qwen Max is genuinely strong at reasoning through complex TypeScript and Electron internals.

My go-to model is Qwen Max. For lighter tasks like /recap or prompt suggestions I set a fast model with /model --fast qwen3-coder-flash to keep costs down.

The /init and Project Context Setup

The very first thing I do when I start on a new project or return to Achu after a few days is run:

/init

This analyzes the current directory and generates an initial context file essentially giving Qwen Code a map of the project. It picks up folder structure, key files, and creates a baseline understanding before I say a single word.

After /init, I manually add a few paragraphs about the project. I treat this like writing a team onboarding doc for a new developer. I tell Qwen what Achu is, what the current milestone is, what tech stack we're on, and what the known constraints are (like Electron IPC boundaries, the Upstash Redis integration, or the Gumroad-based monetization model).

This upfront investment saves enormous amounts of back-and-forth later.

Spec-Driven Planning with /plan

When I want to build a new feature, I don't just dump a vague request and hope for the best. I use /plan to switch Qwen Code into planning mode.

/plan Implement the Privacy Guard redaction pipeline

In plan mode, Qwen analyzes and thinks, but does not touch any files. This is key. It's the agentic equivalent of "think before you act."

What I actually do is run multiple turns in plan mode to iterate the spec. I think of a "spec" as the formal artifact that describes what should be built the interface contracts, the data flow, the error paths, the acceptance criteria. It's not a vague idea. It's something precise enough that a developer (or subagent) could implement it.

The loop looks like this:

/plan enter planning mode
Describe the feature in detail what it does, what it doesn't do, edge cases
Ask Qwen to propose an approach
Push back on anything that doesn't fit the architecture
Ask for the revised plan
Repeat 2-3 times until the spec is solid

The quality of implementation is almost entirely determined by the quality of the spec. This multi-turn refinement before a single line of code is written is the most valuable habit I've developed using any agentic coding tool.

By default, Qwen will ask followup questions. But it is always recommended to tell the model to ask questions.

Subagents for Async Work

Once the spec is locked in, I use subagents aggressively for any work that can happen independently.

Qwen Code's subagent system lets you define specialized agents as Markdown files in .qwen/agents/. Each agent has its own system prompt, tool allowlist, and model. You can call them explicitly or let Qwen delegate automatically.

For Achu, I have a few custom subagents:

A testing subagent focused on Vitest and Electron testing patterns (more on this below)
A code reviewer subagent that runs in plan mode and only reads files

The key power here is Fork Subagents when Qwen needs to run multiple things in parallel, it can implicitly fork. Forks inherit the parent context, run in the background, and share the prompt cache prefix. This means if I ask Qwen to "investigate the IPC handler for Privacy Guard, the Ollama integration, and the Upstash Redis voting flow simultaneously," it can fork three parallel agents without tripling my token costs.

I explicitly phrase tasks as:

"Run these three investigations in parallel using subagents and report back."

This keeps the main conversation focused and lets the grunt work happen concurrently.

A project-level subagent config lives at .qwen/agents/testing.md and looks like this:

---
name: testing
description: "Writes Vitest unit tests and Electron integration tests for Achu. Use PROACTIVELY for any test-related tasks."
approvalMode: auto-edit
tools:
  - read_file
  - write_file
  - read_many_files
  - run_shell_command
---

You are a testing specialist for an Electron + React + TypeScript app.
Follow Vitest conventions. Mock Electron IPC using vitest-mock-extended.
Always write both positive and negative test cases.

The phrase "Use PROACTIVELY" in the description is important it signals to the main model to delegate testing tasks here without being asked explicitly.

Context Hygiene: /summary, /compress, and /clear

This is where most people fail with long agentic sessions. They let the context grow unbounded until the model starts hallucinating, forgetting earlier instructions, or producing inconsistent output. I've learned to treat context like memory on a constrained machine.

My hygiene rules:

After a major chunk of work is done:

/summary

This generates a project summary from the conversation history. I save this externally and reference it when restarting sessions.

When the context window is getting full:

/compress

This replaces chat history with a compressed summary, freeing up tokens while preserving the semantic essence of what was discussed. Think of it as a lossy but practical checkpoint.

When Qwen starts steering away:

If the model starts going off-track giving answers that don't match the project constraints, suggesting patterns we've already ruled out, or just losing the thread. I don't argue. If it happens twice in a row, I clear:

/clear

Then I reload context from scratch using /init and a fresh description. Two drifts is my hard limit. The discipline here is resisting the urge to keep "fixing" a bad session. It's cheaper to restart clean.

Watching Context and Usage with /context and /stats

I watch these two commands constantly.

/context

Shows a breakdown of what's consuming the context window right now system prompt, conversation history, tool results. If I see tool results bloating the context, I know a /compress is coming.

/context detail

Shows per-item breakdown. Useful when one massive file read is eating 40% of the window.

/stats

Gives detailed session statistics tokens used, API calls, cost estimates. I check this before and after big operations. It's how I keep tabs on spend, especially on Qwen Max which isn't the cheapest model.

Keeping an eye on these is the agentic equivalent of watching memory usage in a production system. Ignore it and you'll pay for it.

Pointing to Source Directories Instead of Docs

This one is a significant productivity trick that I don't see talked about enough.

When Qwen needs to understand a third-party library, the default approach is to tell it to fetch the docs URL. The problem is that docs are often incomplete, outdated, or optimized for humans rather than LLMs.

What I do instead: I download the library source and point the conversation directly at it using @:

@./vendor/upstash-redis/src Tell me how the pipeline API works

Or with a deeper path:

@./node_modules/@electron/remote/src/main Explain the context bridge setup

Qwen reads the actual implementation. No guessing from docs. No hallucination about API signatures that changed in the last major version.

I keep a vendor/ folder in the project root where I clone or copy source for critical dependencies. This makes @ references stable and reproducible.

For Achu specifically, I've pointed Qwen at the Ollama TypeScript client source, the llava-phi3 model integration code, and parts of the Electron forge config. The answers I get are ground-truth accurate instead of approximately correct.

Persistent Memory with /remember and /dream

Some things should survive session boundaries. My preferences, key architectural decisions, constraints Qwen needs to always respect. I use /remember for these.

/remember Always use Electron's contextBridge for IPC. Never use remote module.
/remember Achu uses oklch color space. Do not suggest hex values without conversion.
/remember Free tier users get 3 exports per day. Pro users are unlimited.

These get persisted in Qwen's memory store and are injected into future sessions automatically.

/dream is the manual trigger for auto-memory consolidation. Qwen's auto-memory runs in the background, but if I want to force a consolidation pass after a long session to make sure the important discoveries from the current session get persisted I run:

/dream

Think of it as flushing the cache to disk before shutting down.

To review and manage what's been remembered, I use:

/memory

This opens the Memory Manager dialog where I can edit or delete entries. I audit this occasionally. Stale memories can be just as harmful as no memories.

The /btw Trick for Side Questions

This is my favourite quality-of-life command.

/btw What's the difference between contextBridge.invoke and contextBridge.exposeInMainWorld?

/btw sends a parallel API call with recent conversation context (up to the last 20 messages) and shows the response above the composer without touching the main conversation at all. The main session continues uninterrupted.

I use this constantly for:

Quick clarifications while in the middle of implementation
Checking a TypeScript type signature without derailing a planning session
Double-checking a shell command before running it via !

The response doesn't become part of conversation history. It's a throwaway lookup. This is genuinely useful and I'm surprised more CLI tools don't have something like it.

Uncommon Commands Worth Knowing

Beyond the commands I use daily, here are a few from the docs that are genuinely underrated:

/restore

Restores files to their state before a tool execution. If a Qwen action made a mess, you can list recent tool executions with /restore and roll back a specific one with /restore <ID>. Think of it as a targeted undo for AI changes.

/loop

Runs a prompt on a recurring schedule:

/loop 5m check the build output and report any new warnings

I use this occasionally when I'm doing a long build and want Qwen to monitor for me while I do something else. It's a lightweight cron for conversational tasks.

/recap

Generates a one-line summary of where the session left off. If I step away for more than five minutes, Qwen auto-triggers this when I return:

? Implementing the Privacy Guard redaction pipeline. Next step: wire the OCR output into the bounding-box overlay renderer.

Incredibly useful for picking up after an interruption without scrolling through history.

/approval-mode auto-edit

Once I trust the current task scope, I switch to auto-edit to let Qwen make file changes without prompting me every time. I reserve yolo mode for throwaway branches only.

/directory

Adds multiple directories to the workspace context:

/dir add ./src,./tests,./electron

Useful when the feature spans multiple root-level directories that Qwen wouldn't automatically scope to.

My Qwen Code Workflow Summary

Here's the workflow I follow for every non-trivial feature in Achu:

Start clean /init + add project context manually
Spec first use /plan in multiple turns until the spec is solid
Delegate async tasks use subagents for parallel investigations and implementation
Monitor context /context detail regularly, /compress proactively
Log source truth point @ at source directories, not docs
Remember decisions /remember for anything that should persist
Quick questions /btw without breaking flow
Clear if it drifts two steering-away moments is my hard limit for /clear
End of session /dream to consolidate memory, /summary to save the state

This isn't magic. It's discipline. The LLM doesn't make you disciplined you have to bring that yourself. But when you apply this workflow consistently, the output quality is noticeably better than freeform chatting with an AI.

If you're building something with Qwen Code, try the spec-first approach. The twenty minutes you spend in /plan mode iterating the spec will save you three hours of correcting implementation drift.

Happy Agentic Coding, Testing, Shipping, Learning, whatever :) !

What's your biggest challenge with agentic coding workflows staying in context, or getting the model to follow architectural constraints? Let me know in the comments.