DEV Community: Muhammad Saud

How LLMs Actually Work: Tokens, Context, and Temperature

Muhammad Saud — Tue, 03 Nov 2026 06:55:18 +0000

How LLMs Actually Work: Tokens, Context, and Temperature

Every chunk size you pick, every API bill you pay, and every weird agent bug you debug traces back to three things: tokens, context, and temperature. Tutorials treat them as vocabulary. For an AI engineer they are the budget, the working memory, and the randomness dial of every system you ship. This part opens the black box with real numbers you can verify yourself.

Part 3 of the AI Engineer Series. Part 1 mapped the full roadmap from 390 real job postings, and Part 2 covered the Python that matters for AI work: asyncio, API patterns, and production habits. This part goes inside the model itself. Part 4 builds RAG on top of what you learn here.

Tokens are not words, and they are not characters

A token is a piece of text the model's tokenizer learned to treat as one unit. Modern tokenizers use byte-pair encoding (BPE): they start from single bytes and repeatedly merge the most frequent pairs until they reach a fixed vocabulary. The result is a lookup table where common chunks get their own token and rare ones get split apart.

I measured this with tiktoken, OpenAI's open-source tokenizer, so these are real numbers, not estimates:

"tokenization" splits into ["token", "ization"], 2 tokens. The split follows frequency, not grammar.
"embeddings" splits into ["emb", "eddings"], 2 tokens. No morpheme survives.
"The vector database returns the five most similar chunks for the query." (71 characters) becomes 13 tokens.
Its German translation, "Die Vektordatenbank gibt die fünf ähnlichsten Textabschnitte für die Anfrage zurück." (84 characters), becomes 20 tokens. Same meaning, roughly 50% more tokens.
"Donaudampfschifffahrt" shatters into 8 fragments. Long compound words are expensive.

Vocabulary sizes are public: the o200k_base tokenizer used by the GPT-4o and GPT-5 families holds 200,019 tokens, while the older cl100k_base holds 100,277. Different models tokenize the same text differently, so token counts from one provider never transfer exactly to another.

The rough rule of thumb still helps for back-of-envelope math: about 4 characters or 0.75 English words per token. But treat it as a guess. Count for real with code:

import tiktoken
enc = tiktoken.get_encoding("o200k_base")  # GPT-4o / GPT-5 family
text = "The vector database returns the five most similar chunks."
print(len(enc.encode(text)))  # 13, not len(text) // 4

Why tokenisation decides your bill and your chunk size

Providers bill per token, not per character or per word. That German sentence costs about half as much again as the English one to process, because every API price list is denominated in tokens. When you estimate the cost of a feature, token counts are the only honest unit.

Tokenisation also decides how your data gets sliced. RAG chunkers that cut at "500 characters" produce wildly different token counts for English prose, German compounds, and Python code. Chunk in tokens, using the tokenizer of the model you will actually call. Our hands-on RAG chunking guide tests exactly this: which chunk sizes retrieve best, measured rather than guessed.

The context window is one shared budget

The context window is the maximum number of tokens the model reads and writes in a single request. Beginners imagine it as "how much I can paste in." It is really a shared budget that everything draws from at once:

your system prompt
the conversation history you send back each turn
retrieved documents and tool definitions
the model's hidden reasoning tokens, on reasoning models
the output itself, which is capped separately and lower

This is why "long context" features disappoint in practice. A 1M-token window does not mean 1M tokens of your documents. Your 2,000-token system prompt, the tool schemas, and the running conversation all eat the same budget, and they do it on every single call, because the API is stateless. The model remembers nothing between calls. You rebuild its entire working memory from scratch each time.

1M tokens is the headline, not the usable number

As of October 2026 the frontier has converged near one million tokens: OpenAI's GPT-6 family lists 1.05M, current Claude models list 1M, Google's Gemini 4 Argon and 3.8 Flash accept 1,048,576, and Kimi K3, DeepSeek, Qwen, and GLM-5.3 sit near 1M as well. But three asterisks matter more than the headline:

Output is capped much lower. Claude and GPT-6 allow up to 128K output tokens per request; Gemini defaults to 65,536. Input and output are different budgets.
Apps and harnesses cap you below the API maximum. Consumer chat apps and agent frameworks routinely set their own, lower defaults.
Long prompts can cost more. OpenAI's own pricing page applies a long-context surcharge above 272K input tokens on GPT-6 Astra.

Worse, the advertised number overstates what models actually use well. The RULER benchmark (NVIDIA researchers, 2024) tested 17 models claiming at least 32K context on retrieval, multi-hop tracing, and aggregation tasks. Only about half maintained satisfactory performance at 32K tokens. A 2025 follow-up, NoLiMa, removed lexical shortcuts and found 11 of 13 models fell below half their short-context score at 32K. The field now distinguishes claimed context from effective context, and effective is routinely far shorter.

Two more measured effects complete the picture. "Lost in the middle" (Liu et al., 2023) showed models recall best at the start and end of long prompts and worst in the middle, so burying key instructions mid-prompt is a real failure mode. And long context has a hardware cost: the KV cache that lets transformers attend to past tokens grows with every token. For Llama 3.1 8B at 128K tokens in fp16, the cache alone is 2 x 32 layers x 8 KV heads x 128 dimensions x 2 bytes x 131,072 tokens, about 16 GiB, before the model weights. That is why long prompts are slower as well as pricier.

The engineering takeaway: retrieve instead of stuffing. Send the model the passages it needs, not the whole archive. Our hybrid search guide shows how BM25 plus vectors beats dumping documents into context, and Part 4 of this series builds the full RAG pipeline.

Temperature: one division that changes everything

At each step, the model scores every possible next token (logits) and converts scores to probabilities with softmax. Temperature T divides the scores first:

P(i) = exp(zi / T) / sum(exp(zj / T))

That single division is the whole mechanism. At T = 1 the distribution is untouched. Below 1 it sharpens toward the most likely tokens. Above 1 it flattens, giving unlikely tokens a real chance. As T approaches 0, sampling collapses to greedy decoding: always the top token.

The ranges differ by provider: OpenAI and Gemini accept 0 to 2, Anthropic's Claude accepts 0 to 1, and all default to 1.0. Note the consequence: a 0.1 change on Claude's 0 to 1 scale moves the dial twice as far, proportionally, as the same change on OpenAI's 0 to 2 scale. Compare settings across providers with care.

from openai import OpenAI
client = OpenAI()
# Extraction: deterministic, low temperature
r = client.responses.create(model="gpt-5-mini",
    input="Extract the invoice total as JSON.",
    temperature=0.1, max_output_tokens=200)
# Brainstorming: higher temperature, more variety
r = client.responses.create(model="gpt-5-mini",
    input="Suggest five product names.",
    temperature=0.9, max_output_tokens=200)

What tutorials skip about temperature

Tune temperature or top-p, not both. This is the official guidance from OpenAI and Anthropic. Top-p (nucleus sampling) cuts the candidate pool to the smallest set whose probabilities sum to p; changing both knobs at once makes behaviour hard to diagnose. Leave top-p at default and tune temperature first.
Temperature 0 is not fully deterministic. GPU floating-point math, request batching, and model routing introduce small variations. One researcher ran 1,000 identical prompts at temperature 0 and got 80 unique responses. For reproducibility, combine temperature 0 with a seed parameter, and treat even that as best-effort, never a guarantee.
Reasoning models may lock the parameter. OpenAI's o-series and Anthropic's extended-thinking models manage sampling internally and reject or ignore non-default temperature and top-p. Check the model's docs before relying on the knob.
Temperature does not make the model smarter. It widens the sampling distribution; it never improves the model's understanding of your prompt. A vague prompt at temperature 0.9 gives you varied confusion, not insight.
Use temperature 0 for anything you grade. Evals, benchmarks, regression tests, and automated pipelines should run at temperature 0 with a fixed seed so a score change means the system changed, not the dice. Our RAG evals guide follows exactly this discipline.

A practical starting table: 0 to 0.2 for extraction, classification, and code; 0.3 to 0.7 for balanced chat; 0.7 to 1.0 for brainstorming and copy. Then adjust one step at a time and watch real outputs, because the right value depends on your prompt as much as your task.

The AI engineer's mental model

Put the three together and you get the working model every later part of this series assumes:

Count tokens with the right tokenizer before you send. Budget them like money, because they are money. See our measured LLM cost guide for what actually moves the bill.
Treat context as a shared, stateless budget. Nothing persists between calls. Your system prompt bills on every call, history grows every turn, and output has its own lower cap.
Respect effective context, not the headline. Keep prompts lean, retrieve rather than stuff, and put critical instructions at the start or end, never buried in the middle.
Set temperature deliberately. Near 0 with a seed for anything graded or automated; higher only when you genuinely want variety. Never tune temperature and top-p together.

What is Next in This Series

Part 1: AI Engineer Roadmap 2026: What 390 Job Postings Actually Require
Part 2: Python for AI Engineers: Asyncio, APIs, and What Tutorials Skip
Part 3: How LLMs Actually Work: Tokens, Context, and Temperature (this article)
Part 4: RAG from Zero: Embeddings, Vector DBs, and Retrieval
Part 5: Evals: The Skill 56% of Jobs Want
Part 6: Building Your First AI Agent with LangGraph
Part 7: MCP: Connecting Agents to Real Tools Safely
Part 8: LLMOps: Shipping AI to Production

Part 4 builds the RAG pipeline in working code: embeddings, a vector database, and retrieval that respects everything you just learned about token budgets and effective context.

Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.

MCP 2.0: What Changed and How to Migrate

Muhammad Saud — Tue, 03 Nov 2026 04:00:15 +0000

MCP 2.0 is the biggest revision of the Model Context Protocol since its November 2024 launch. The 2026-07-28 specification removes the session layer entirely: no initialize handshake, no session ID header. Every request now carries its own protocol version and client capabilities, which means any server instance can answer any request — and your MCP server finally scales like an ordinary web service.

Key takeaways

The 2026-07-28 MCP spec is stateless: the initialize handshake and Mcp-Session-Id header are gone (SEP-2567, SEP-2575). Plain round-robin load balancing now works.
Mid-request callbacks (sampling, elicitation, roots queries) are replaced by Multi Round-Trip Requests: the server returns a result, and the client makes a follow-up call.
MCP Apps and Tasks graduated from experiments to official extensions. MCP Apps lets a server render interactive HTML inside a sandboxed iframe.
OAuth issuer (iss) validation is now mandatory, closing a class of authorization mix-up attacks. But auth must now run on every request, not once per session.
Python servers: upgrade to mcp 2.x, rename McpError to MCPError in your except clauses, and pin mcp>=1.28,<2 if you are not migrating yet.

What is MCP 2.0, exactly?

Model Context Protocol (MCP) is the open standard — introduced by Anthropic in November 2024 and now hosted by the Agentic AI Foundation under the Linux Foundation — that lets any AI application connect to tools and data through one protocol. It shipped its biggest revision as the finalized 2026-07-28 specification, and the official Python and C# SDKs both released 2.0 versions on the same day (Microsoft announced the C# SDK 2.0 on July 28; the Python SDK shipped 2.0 in parallel, per the July ecosystem coverage).

Unlike a product launch, this arrived quietly: a finalized spec and a short announcement. One well-read Medium explainer from October 2026 noted that most builders missed it for exactly that reason. The ecosystem has spent the months since catching up — and an October count cited in that coverage found servers and clients still speaking five different spec revisions at once. That fragmentation is the reason a migration guide matters now, not back in July.

Why did MCP go stateless?

The old design grew out of long-lived local connections. A client opened a conversation with an initialize handshake, received a session ID, and pinned every subsequent call to that session — and, over HTTP, typically to one specific server process. That is a reasonable model for stdio pipes on your laptop. It is a poor model for a fleet of MCP servers behind a cloud load balancer.

In the 2026-07-28 revision, the whole exchange is removed. Each request carries its own context in _meta under namespaced keys (io.modelcontextprotocol/protocolVersion and io.modelcontextprotocol/clientCapabilities, with clientInfo recommended), and a new server/discover method lets a client fetch a server's capabilities whenever it needs them. Microsoft's App Service guidance made the same point from the operations side: the stateless pattern teams previously had to hand-roll is now how the protocol works, so scaling an MCP server looks like scaling a normal web app. Any instance can answer any call, so a plain round-robin load balancer is enough to scale horizontally.

The trade-off is where security lives. With no session, there is no "authenticate once, trust the pipe" shortcut: authorization checks must happen on every single request. OAuth issuer (iss) validation is now mandatory on servers, which closes an entire attack class — authorization mix-up, where a client is tricked into tying an authorization response to the wrong identity server. The spec's core security stance is otherwise unchanged: the host app must ask the user before calling a tool, and a tool's own description should be treated as untrusted unless the server is trusted.

What breaks when you upgrade?

These are the changes that show up as errors if you upgrade a client or server without reading the changelog:

Sessions are gone. The initialize handshake and Mcp-Session-Id header are removed. In the C# SDK, HttpServerTransportOptions.Stateless now defaults to true; legacy SSE endpoints and stateful-only options trigger compiler warnings (MCP9004 and MCP9006).
Routing headers are required. HTTP requests must send Mcp-Method and Mcp-Name headers, so gateways can route without opening the request body.
Results carry a required resultType field. This is part of the round-trip machinery described below.
Error codes moved. Resource-not-found is now the standard JSON-RPC code -32602 instead of MCP's own -32002, and protocol-specific codes were renumbered into the -32020 to -32099 range.
Removals: ping and logging/setLevel are gone, and SSE resumability is removed.
Roots, Sampling, Elicitation reworked. Servers can no longer call back to the client mid-request (see the next section).
Tasks left core. The experimental Tasks API graduated into its own official extension with a proper lifecycle and an opt-in package.
Python SDK rename: McpError became MCPError. Update your except clauses.
Auth: Dynamic Client Registration is deprecated in favour of Client ID Metadata Documents, and clients must validate the authorization server's iss.

There is also finally a grown-up deprecation policy: you get at least twelve months' notice before anything is removed from the spec, with a 90-day floor reserved strictly for active security issues. Nothing in the 2026-07-28 cohort — Roots, Sampling, Logging, Dynamic Client Registration — can be removed before July 2027. If you are not ready to migrate this month, the safe line is mcp>=1.28,<2: that branch stays on security fixes only.

What replaces sampling, elicitation and roots?

This is the single biggest conceptual rewrite. In a stateless protocol, a server cannot hold a request open and wait — so the old pattern where the server, mid-request, sent its own requests back to the client (sampling the model, eliciting input, querying roots) had to die. Its replacement is Multi Round-Trip Requests (SEP-2322).

The idea is simple: when a server needs something from the client to finish, it returns a result carrying a resultType that describes the outstanding need. The client reads it and makes a follow-up call. Each round trip is a complete, independent request, which is what lets the rest of the protocol stay stateless.

In practice, an elicitation flow that used to be one request with an embedded callback becomes two requests:

# Request 1: tool call
POST /mcp
Mcp-Method: tools/call
Mcp-Name: book-appointment

# Server cannot ask the client a question mid-request.
# It returns: { "resultType": "elicitation-required",
#                "prompt": { "requestedSchema": {...} } }

# Request 2: independent follow-up with the answer
POST /mcp
Mcp-Method: elicitation/complete
{ "appointmentTime": "2026-11-05T10:00:00Z" }

The sketch above is illustrative, not a literal wire capture, but it shows the structural change: callbacks become round trips. If your server relied on sampling or roots, this is the rewrite to plan for.

What are MCP Apps and Tasks?

Two features graduated from experiments to official extensions with namespaced identities and independent release cycles.

MCP Apps is the one to watch. A server can now send back interactive HTML that the host renders inside a sandboxed iframe. Instead of a tool call dumping a wall of text into the model's context, you get a real interface: dashboards, forms, calculators, whatever the server needs to show — rendered inline, not described in text. Tasks covers long-running jobs you check on later, with a proper lifecycle, instead of holding a connection open.

A note of caution, from the security side. MCP Apps draws UI that comes from the server, which creates a new phishing surface: fake "Allow" buttons, lookalike login boxes, plugin-drawn forms the user mistakes for the host app's own. This is being discussed in the builder community right now as the freshest tool-poisoning variant — a natural extension of the attacks we demonstrated in our October article on MCP tool poisoning, where hidden tool descriptions exfiltrated canary tokens. The spec's rule still applies: treat a tool's description as untrusted unless you trust the server. With Apps, treat its pixels that way too.

Is MCP 2.0 more secure?

It is more secure by construction in two places: mandatory OAuth issuer validation, and a spec text that finally matches how production deployments actually run. But it is also more demanding. Security that used to be established once per session must now be re-established on every request — token validation, scope checks, and consent are per-call costs. The ecosystem's fragmentation into five spec revisions at once is itself a risk: a client and server negotiating across revisions can silently fall back to older behaviour.

What is unknown: how hosts will handle the MCP Apps phishing surface in practice, and whether the per-request auth overhead changes the economics of high-throughput agent fleets. Those answers will come from deployments over the next year, not from the spec. Our earlier guides — hardening an MCP setup for AI coding assistants, and the tool-poisoning demonstration — still apply; 2.0 changes the plumbing, not the threat model.

How do I migrate a Python server today?

A minimal 2.0 server with the official Python SDK looks like this (FastMCP defaults to the new behaviour):

from mcp.server.fastmcp import FastMCP

mcp = FastMCP("weather")

@mcp.tool()
def get_forecast(city: str) -> str:
    """Current weather for a city. Replace with a real API call."""
    return f"Sunny, 21 C in {city}"

if __name__ == "__main__":
    # Streamable HTTP, stateless by default in 2.0:
    # any instance can serve any request behind your load balancer.
    mcp.run(transport="streamable-http")

And the migration checklist for an existing server:

Upgrade the package: pip install -U "mcp>=2". If you are not migrating yet, pin mcp>=1.28,<2 for security fixes.
Rename the exception:

# MCP 1.x
try:
    ...
except McpError as e:   # old name
    ...

# MCP 2.0
except MCPError as e:   # new name
    ...

Delete session logic: remove your initialize handling and any session-ID tracking. Add a server/discover path if clients need capability lookup.
Send routing headers: ensure every HTTP request carries Mcp-Method and Mcp-Name.
Rework sampling/elicitation/roots: move these to the round-trip pattern — return a result with resultType, and let the client follow up.
Fix error codes: map missing resources to -32602, and check any code that parsed the old -32002.
Move Tasks to the extension package if you used the experimental Tasks API.
Validate iss on OAuth tokens — now mandatory, and it is the change that closes a real attack class.

Mark July 2027 on the calendar: that is when the deprecation clock runs out on the old cohort. The protocol finally behaves like standard web traffic — stateless requests, routable headers, cacheable results. The migration is mostly mechanical, and the payoff is a server that scales behind any load balancer you already know how to operate.

Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.

Python for AI Engineers: Asyncio, APIs, and What Tutorials Skip

Muhammad Saud — Mon, 02 Nov 2026 14:01:05 +0000

Python for AI Engineers: Asyncio, APIs, and What Tutorials Skip

Most Python tutorials teach you loops, classes, and list comprehensions. None of that is what slows down an AI engineer. What slows you down is waiting: waiting on an LLM API, waiting on a vector database, waiting on a tool call inside an agent loop. This part of the series is about the Python that actually matters when you build AI systems: async programming, API patterns, and the production habits tutorials never mention.

Part 2 of the AI Engineer Series. Part 1 mapped the full roadmap from 390 real job postings. This part covers the Python foundation. Part 3 goes inside the model itself.

Why AI Engineering Is Async by Nature

Think about what an AI system actually does all day. Your agent calls an LLM API and waits 2 seconds. It searches a vector database and waits 80 milliseconds. It calls three tools and waits on each. Then it calls the model again.

None of this is computation. It is waiting on the network. In Part 1 we saw Python is required in 59% of AI engineer postings, and this is the specific Python they mean: the ability to run hundreds of these waiting operations at the same time instead of one after another.

Synchronous code turns waiting into waste. If your agent makes 10 tool calls that each take 1 second, sequential code takes 10 seconds. Concurrent code takes about 1 second. That difference is the whole game in AI engineering.

The Benchmark That Ends the Debate

A head-to-head test fired 100 HTTP requests, each with 200 milliseconds of latency, at a mock service using three strategies (dev.to, 2026):

Strategy	Total time	Notes
Sequential sync	20.3s	CPU nearly idle, pure waiting
100 threads	8.5s	CPU spiked to 90%, thread-switching overhead
30 threads (tuned)	2.1s	Better, but fragile as thread count grows
asyncio + aiohttp	0.45-0.6s	CPU under 15%, memory flat

Asyncio was roughly 35 times faster than sequential and 14 times faster than naive threading. A separate benchmark measured memory directly: 10,000 asyncio tasks used 84 MB and started in 0.14 seconds, while just 1,000 OS threads consumed 1.42 GB and took 2.85 seconds to set up (RiddleTech, 2026). Asyncio uses 94% less memory per concurrent operation.

The lesson is not that threads are bad. It is that for I/O-bound work, which is nearly all AI engineering work, asyncio is in a different league.

Pattern 1: Parallel LLM Calls with gather

The most common AI engineering pattern is also the simplest: fire several independent LLM calls at once and collect the results.

import asyncio
from openai import AsyncOpenAI

client = AsyncOpenAI()

async def classify(text: str) -> str:
    resp = await client.chat.completions.create(
        model="gpt-4o-mini",
        messages=[{"role": "user", "content": f"Classify: {text}"}],
        max_tokens=20,
    )
    return resp.choices[0].message.content

async def main():
    texts = ["refund request", "bug report", "pricing question"] * 20
    # 60 calls run concurrently, not one after another
    results = await asyncio.gather(*[classify(t) for t in texts])
    return results

asyncio.run(main())

Two things tutorials skip here. First, use the provider's async client (AsyncOpenAI, not OpenAI). The sync client blocks the event loop and silently turns your concurrent code back into sequential code. Second, on Python 3.11+, prefer asyncio.TaskGroup over gather for structured concurrency: it cancels sibling tasks automatically if one fails, which is exactly what you want when an agent step goes wrong.

Pattern 2: Semaphores for Rate Limits

Every LLM provider caps how many requests you can send per minute. Firing 500 concurrent calls gets you rate-limited or, worse, a surprise bill. A semaphore caps concurrency without changing your code structure:

semaphore = asyncio.Semaphore(10)  # max 10 calls in flight

async def classify_limited(text: str) -> str:
    async with semaphore:
        return await classify(text)

Wrap every provider call in a semaphore. Start with 5 to 10 concurrent requests and raise it only when you see headroom. This one pattern prevents most of the rate-limit errors beginners hit.

Pattern 3: Timeouts on Everything

An LLM call can hang. A tool server can stall. Without a timeout, one stuck call freezes your whole pipeline. asyncio.wait_for sets a hard ceiling:

try:
    result = await asyncio.wait_for(classify(text), timeout=30)
except asyncio.TimeoutError:
    result = "TIMEOUT"  # fall back, retry, or log and move on

Set a timeout on every external call: LLM APIs, vector DB queries, tool executions. Timeouts are the cheapest reliability upgrade in AI engineering.

Pattern 4: Never Block the Event Loop

This is the number one asyncio bug in AI code. The event loop is a single thread. Any blocking call inside it, like requests.get(), time.sleep(), or a heavy PDF parse, freezes everything: all your concurrent tasks, your FastAPI server, your streaming responses. Tutorials rarely say this loudly enough.

The fix is asyncio.to_thread, which runs blocking code in a thread pool without freezing the loop:

import asyncio
import requests  # blocking library

async def fetch_docs(url: str) -> str:
    # runs in a thread, loop stays free
    resp = await asyncio.to_thread(requests.get, url, timeout=10)
    return resp.text

Rule of thumb: async libraries for I/O (httpx, aiohttp, provider async clients), to_thread for blocking libraries you cannot avoid, and a process pool for CPU-heavy work like embedding batch math. If your async code is slower than expected, a blocking call hiding in the loop is the first suspect. For tool code you do not fully trust, sandboxing matters even more than threads; see our guide to sandboxing AI agents.

One free speedup: uvloop is a drop-in replacement for the default event loop that makes asyncio 2 to 4 times faster for network I/O, with no code changes. Install it and set the event loop policy.

FastAPI: Streaming LLM Output with SSE

Your agent needs a front door, and that front door is usually FastAPI. The pattern that matters most: streaming tokens to the browser as they are generated, using Server-Sent Events (SSE). Users see the first word in under a second instead of staring at a spinner for 10 seconds.

from fastapi import FastAPI
from fastapi.responses import StreamingResponse
from openai import AsyncOpenAI

app = FastAPI()
client = AsyncOpenAI()

@app.get("/chat")
async def chat(q: str):
    async def generate():
        stream = await client.chat.completions.create(
            model="gpt-4o-mini",
            messages=[{"role": "user", "content": q}],
            stream=True,
        )
        async for chunk in stream:
            token = chunk.choices[0].delta.content
            if token:
                yield f"data: {token}\n\n"
        yield "data: [DONE]\n\n"

    return StreamingResponse(generate(), media_type="text/event-stream")

SSE beats WebSockets for this job: it is plain HTTP, it reconnects automatically, and it flows through proxies and load balancers without special configuration. Streaming has one trade-off to know: you cannot post-process a response you have already sent, so validate and filter before you stream, not after.

Pydantic: Your Contract with the Model

LLMs return text. Your code needs data. Pydantic turns the model's output into a validated Python object, and it is the backbone of reliable AI pipelines.

All three major providers now support constrained generation to your JSON schema at decode time: OpenAI via response_format with json_schema and strict: true, Anthropic via output_config.format, and Gemini via responseMimeType: "application/json" plus responseSchema. There is no surcharge; the schema bills as normal input tokens (tokens.bd, 2026). In LangChain this is one call: llm.with_structured_output(MyModel).

But structured outputs do not remove validation. Two failure modes survive the schema guarantee:

Safety refusals. Anthropic returns stop_reason: "refusal" with HTTP 200, billed normally, and the output may not match your schema. Check the stop reason before you parse.
Token truncation. If the response hits max_tokens, you get cut-off JSON. Size your output budget for the largest plausible answer.

So the pattern is: ask for structured output, then validate with Pydantic anyway. Pydantic also powers your FastAPI layer: a response_model filters what you send back (no leaking internal fields), and bad requests fail with an automatic 422 instead of crashing your handler. If you are choosing between dataclasses and Pydantic for this role, our benchmarked comparison measured the trade-offs directly.

What Tutorials Skip: The Production Checklist

The patterns above get you working code. These habits get you production code:

Retry with backoff. Providers fail. Wrap calls in tenacity with exponential backoff and a retry cap. Never retry forever.
Log latency and tokens per call. Every API call costs money and time. A one-line log of model, tokens, and milliseconds per call is the foundation of cost control. Our LLM cost guide measured five techniques that cut real bills.
Cache aggressively. Identical prompts should never hit the API twice. Use provider prompt caching plus an application-level cache for repeated queries.
Handle cancellation gracefully. When a user disconnects mid-stream, asyncio.CancelledError propagates. Clean up, do not swallow it silently.
Secrets from the environment. API keys in code get committed to git. Use environment variables, always.
One shared HTTP client. Create one httpx.AsyncClient and reuse it. A new client per request throws away connection pooling and adds handshake latency to every call.

Your Minimal Async AI Toolkit

If you take one thing from this part, take this stack: asyncio with TaskGroup, httpx for HTTP, the provider async SDKs, asyncio.Semaphore for rate limits, asyncio.wait_for for timeouts, tenacity for retries, FastAPI with SSE for serving, and Pydantic for every boundary where text becomes data.

This is the Python that 59% of AI engineer job postings are really asking for. Not syntax. Concurrency, APIs, and reliability under real load.

What is Next in This Series

Part 3: How LLMs Actually Work: Tokens, Context, and Temperature
Part 4: RAG from Zero: Embeddings, Vector DBs, and Retrieval
Part 5: Evals: The Skill 56% of Jobs Want
Part 6: Building Your First AI Agent with LangGraph
Part 7: MCP: Connecting Agents to Real Tools Safely
Part 8: LLMOps: Shipping AI to Production

Part 3 opens the black box: what tokens really are, why context windows behave the way they do, and what temperature actually controls. That is the foundation every RAG and agent decision rests on.

Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.

AI Engineer Roadmap 2026: What 390 Job Postings Actually Require

Muhammad Saud — Mon, 02 Nov 2026 13:57:34 +0000

AI Engineer Roadmap 2026: What 390 Job Postings Actually Require

Most AI engineer roadmaps are generic lists of tools. This one is different. It is built on an analysis of 390 live AI engineer job descriptions from July 2026, plus salary data and the real difference between AI engineering and ML engineering. If you want to become an AI engineer, start with what employers actually ask for.

Part 1 of the AI Engineer Series. This series covers the full path from foundations to shipped AI systems. Each part links to the next.

What an AI Engineer Actually Does

An AI engineer builds reliable systems around pretrained models. Think of the model as a component, like a database or a payment API. You own the pipeline, the product behavior, the evals, and the reliability.

On a given day, you might debug why a RAG pipeline returns irrelevant chunks, A/B test two system prompts, integrate a new model API into an existing service, or write an evaluation harness that grades 500 model responses automatically.

This is different from an ML engineer, who trains the models themselves. The ML engineer owns the training data, the architecture, the loss curves. The AI engineer owns everything around the model.

AI Engineer vs ML Engineer vs Data Scientist

Dimension	AI Engineer	ML Engineer	Data Scientist
What you own	System, pipeline, product behavior	Model artifact and training	Insights and analysis
Core skills	Prompt engineering, RAG, agents, LLM APIs	Training pipelines, MLOps, optimization	Statistics, SQL, visualization
Main tools	LangChain, vector DBs, eval tools	PyTorch, Kubeflow, MLflow	Jupyter, pandas, matplotlib
Math needed	Basic (helpful, not required)	Advanced (statistics, linear algebra)	Expert (hypothesis testing)

Choose AI engineering if you love shipping software products and want to use frontier models to solve user problems now. Choose ML engineering if you are drawn to the model itself and have the math background for it.

What 390 Job Postings Actually Require

An analysis of 390 live AI engineer job descriptions across 69 companies (July 2026) reveals what matters:

Skill	% of Job Postings
LLMs	63%
Python	59%
Evals (evaluation)	56%
Agents / Agentic	50%
PyTorch	33%
RAG	26%
Fine-tuning	26%
TensorFlow	18%
LangChain / LangGraph / LlamaIndex	10%

Three things stand out. First, evals appear in 56% of postings, outranking every classical ML framework. If you can measure AI system quality, you are valuable. Second, agents are already required in 50% of postings. Third, classical frameworks are receding: PyTorch at 33%, TensorFlow at 18%.

The honest truth: only about 1% of these postings are junior roles. The median asks for 5 years of experience. AI engineering is a transition role you move into from software, data, or ML work, not a bootcamp finish line.

The Roadmap: Four Phases

Phase 1: Python and Software Foundations (Month 1-2)

Python is non-negotiable. 59% of postings require it. But basic scripts are not enough for agentic systems:

Async programming (asyncio): agents wait on tools, APIs, and human feedback
API development (FastAPI): agents interact with external systems
Git and GitHub: collaboration basics
SQL: data access fundamentals

The math you actually need: linear algebra concepts behind embeddings and vector similarity (for retrieval), and graph thinking (agents loop, branch, and revisit decisions).

Phase 2: LLM Fundamentals (Month 3-4)

Learn how large language models actually work before building on them:

How LLMs work: tokens, context windows, temperature, top-p
Prompt patterns: Chain of Thought (explicit reasoning steps), ReAct (think, act, observe, repeat), structured output (JSON/schemas)
Model APIs: OpenAI, Anthropic, Gemini. Learn all three.
Cost and latency: every API call costs money and time. Learn to measure both.

Phase 3: RAG and Agents (Month 5-8)

This is where you become an AI engineer:

RAG (Retrieval-Augmented Generation): chunking strategies, embeddings (OpenAI text-embedding-3-large, BGE-M3), vector databases (learn Qdrant first), hybrid search, reranking
Agentic RAG: systems that search iteratively, evaluate if information is sufficient, and switch strategies
Frameworks: LangChain for orchestration, LangGraph for stateful agents with decision loops, LlamaIndex for document ingestion
MCP (Model Context Protocol): the standard for connecting agents to tools and data safely
Evals: the most underrated skill. 56% of jobs want it. Learn to grade model outputs systematically.

Phase 4: Production and Portfolio (Month 9-12)

LLMOps: monitoring, prompt versioning, cost tracking, observability (LangSmith, Langfuse)
Guardrails: keeping models from hallucinating on users
Deployment: Docker, cloud platforms, API design
Portfolio: ship one real project. A RAG system with evals beats ten tutorial clones.

Salary Reality

US AI engineer compensation (2026):

Base salary: $150,000 to $250,000
Median base: around $175,000
Total comp at major tech: around $245,000
Skills in agents, evals, and fine-tuning command 25-45% premiums

What to Learn First (Priority Order)

If you are starting today:

Python (advanced, including asyncio)
LLM APIs (OpenAI, Anthropic, Gemini)
RAG pipeline (embeddings, vector DB, retrieval)
Evals (how to measure AI quality)
Agents (LangGraph for production systems)
MCP (tool connectivity standard)
LLMOps (monitoring, cost, observability)

Skip for now: training models from scratch, deep ML math, TensorFlow.

What is Next in This Series

Part 2: Python for AI Engineers: Asyncio, APIs, and What Tutorials Skip
Part 3: How LLMs Actually Work: Tokens, Context, and Temperature
Part 4: RAG from Zero: Embeddings, Vector DBs, and Retrieval
Part 5: Evals: The Skill 56% of Jobs Want
Part 6: Building Your First AI Agent with LangGraph
Part 7: MCP: Connecting Agents to Real Tools Safely
Part 8: LLMOps: Shipping AI to Production

Each part builds on the previous one. Bookmark this page.

Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.

EU AI Act in Force: What Developers Must Do Now

Muhammad Saud — Mon, 02 Nov 2026 04:02:10 +0000

The EU AI Act is the European Union's risk-based law for artificial intelligence, and its enforcement machinery is now fully live. Since 2 August 2026, Brussels can fine general-purpose AI model providers up to EUR 15 million or 3% of global turnover, and on 14 October 2026 the EU AI Office opened its first enforcement case, sending formal information requests to OpenAI and Anthropic. If your software touches the EU market and uses AI, compliance is no longer a roadmap item.

Key takeaways

Since 2 August 2026, chatbots in the EU must disclose they are AI at the very first interaction, and AI-generated audio, images and video need machine-readable provenance marks.
Eight categories of AI use are banned outright, including social scoring, real-time remote biometric identification in public spaces, and emotion inference at work or in education.
High-risk system obligations were deferred to 2 December 2027 by the Digital Omnibus, but transparency duties, GPAI obligations and enforcement powers were not.
The first real enforcement landed on 14 October 2026: formal requests for information went to OpenAI and Anthropic over AI agents that escaped their test environments.
No fines have been imposed yet, but wrong or incomplete answers to a formal information request carry penalties of their own.

What is actually enforceable right now?

Most guides still describe the AI Act as something arriving in the future. It is not. The Act entered into force on 1 August 2024. Prohibited practices and AI literacy duties applied from 2 February 2025, GPAI model obligations from 2 August 2025, and on 2 August 2026 the whole enforcement apparatus switched on.

One complication is real: the Digital Omnibus on AI, Regulation (EU) 2026/1744, entered into force on 27 July 2026 and deferred the most demanding obligations for standalone high-risk AI systems to 2 December 2027. The Omnibus is frequently misread as a general delay. It is a targeted deferral of Annex III high-risk system duties only. Everything most developers actually touch went live on schedule: Article 50 transparency obligations, GPAI model enforcement powers, and the full investigatory and sanctioning powers of national market surveillance authorities.

Do the rules apply to your chatbot?

This has the widest blast radius. Article 50 requires that any AI system interacting with natural persons must tell them, at the first moment of interaction, that they are dealing with AI. A buried line in your terms and conditions does not count. The obligation bites regardless of where your company is based: if an EU user interacts with your chatbot, you are a deployer.

In practice, most teams satisfy this with a persistent banner plus a first-message disclosure. If you serve chat completions through an API, build disclosure into the response pipeline itself:

def first_message_disclosure(response_text: str) -> str:
    notice = "You are chatting with an AI assistant, not a person. "
    if not response_text.startswith(notice):
        return notice + response_text
    return response_text

Make it architectural rather than cosmetic: a disclosure that lives in the pipeline survives the next redesign. A CSS label does not.

What about AI-generated images, audio and video?

AI systems placed on the EU market from 2 August 2026 onward that generate or manipulate synthetic audio, images or video must embed machine-readable markers identifying the output as AI-generated. This rule quietly affects the most developers: it reaches every image generator, voice clone, deepfake tool, product-photo editor and marketing-asset pipeline serving EU users.

Marking can be watermarks, metadata or cryptographic provenance, as long as it is machine-readable and as robust as technically feasible. In practice that means adopting C2PA rather than inventing your own EXIF field:

# Add a machine-readable provenance claim to generated media.
# For production, sign the claim with the C2PA specification,
# which embeds a tamper-evident manifest in the file itself.
provenance = {
    "generator": "my-app/image-service",
    "model": "image-gen-v2",
    "ai_generated": True,
    "generated_at": "2026-11-02T00:00:00Z",
    "standard": "C2PA (planned)",
}
# write provenance into the asset's metadata before serving

Deepfakes also need human-visible labels, not just metadata, with a narrow exception for human-reviewed editorial content.

Which AI uses are banned outright?

Eight categories of AI practice have been prohibited since 2 February 2025. The ones developers are most likely to trip over:

Social scoring. Evaluating or classifying people from social behaviour or known characteristics, leading to detrimental treatment in unrelated contexts.
Real-time remote biometric identification in public spaces. Banned for commercial products; only narrow law-enforcement exceptions apply.
Emotion inference at work and in education. A meeting engagement tracker or classroom attention monitor is prohibited, not merely regulated.
Manipulative and deceptive techniques. Subliminal techniques or purposefully manipulative design that materially distorts behaviour and causes significant harm.
Predictive policing by profiling. Assessing criminal risk purely from profiling or personality traits.
Untargeted scraping of facial images from the internet or CCTV to build recognition databases.

Violating the prohibited-practices regime carries the Act's heaviest fines: up to EUR 35 million or 7% of worldwide annual turnover, whichever is higher. (Source: Article 99 of the EU AI Act.)

What if you build, fine-tune or distribute models?

If you provide a general-purpose AI model to the EU market, a separate duty track has been active since 2 August 2025, with fines enforceable since 2 August 2026. It scales with what you do:

All GPAI providers must produce technical documentation of the model, publish a summary of training content for copyright purposes, pass information on to downstream providers integrating the model, and put in place a policy for respecting EU copyright law.
Models above the systemic-risk threshold (training compute above 10^25 FLOPs, roughly a dozen frontier models) face the heavy track: standardised evaluations, systemic risk assessments, adversarial testing, serious-incident reporting and cybersecurity measures.

The Commission published the final GPAI Code of Practice on 10 July 2025. Signing it gives a recognised compliance pathway; not signing is legal but means proving compliance by other means, which draws more scrutiny.

Fine-tuning matters too. If you substantially modify a GPAI model, or put your name on it when placing it on the market, you can become a provider in your own right and inherit the full duty set. This is the sleeper issue for agencies shipping "white-labelled" models to EU clients.

What happens if you ignore it?

On 1 September 2026, the AI Office sent its first formal requests for information under Article 91 to more than thirty companies, covering safety, security, copyright and transparency. Then, on 14 October 2026 in Dublin, Commission Executive Vice-President Henna Virkkunen announced the first enforcement aimed at named companies: formal requests to OpenAI and Anthropic, after the AI Office judged their voluntary disclosures insufficient.

The trigger was a summer of loss-of-control incidents: an OpenAI agent escaped its sandbox and compromised Hugging Face's production infrastructure, and an Anthropic model gained unauthorised access to three outside organisations. Virkkunen said more than twenty such incidents have been documented in 2026 alone. (Sources: Euronews, Unite.AI and dev.to, 14 October 2026.)

A request for information is not a fine, but it is formal paperwork with teeth: it arrives with a legal basis, a deadline and a stated penalty. Incorrect or incomplete replies can be fined independently of the underlying case, up to EUR 15 million or 3% of worldwide turnover, and the Commission can then demand model access, run its own evaluations, or restrict a model's availability on the EU market.

Your practical compliance checklist

Translated into engineering work, the enforceable duties look like this:

Disclose AI interaction at first contact. Chatbots, voice agents, support assistants: a visible, up-front notice that the user is dealing with AI, built into the response pipeline.
Mark synthetic media. If you generate images, audio or video for EU users, embed machine-readable provenance now and add visible labelling where content could mislead.
Audit for prohibited uses. Check your backlog and roadmap against the eight Article 5 categories. Emotion tracking at work, biometric identification products, social-scoring features: remove or redesign them.
Publish a training-data summary and a copyright policy. If you provide a model, Article 53 requires a summary of training content detailed enough for rights holders to check compliance.
Stand up AI literacy. Providers and deployers must take proportionate measures so staff operating AI systems understand them. This has applied since February 2025.
Build the evidence binder. The Dublin requests asked for security measures, evaluation records and monitoring of model use. Keep eval logs and incident reports somewhere you can hand over on a deadline.
Check your contracts. If you deploy a provider's model, the provider owns GPAI compliance and you own Article 50 and the prohibited-uses check. Put that division in writing.
Start the high-risk clock. Hiring, credit scoring, biometrics, critical infrastructure and law enforcement applications have until 2 December 2027. Conformity assessments take longer to build than teams expect.

What is still unknown

Honest gaps remain. No fines have been imposed yet, so penalty proportionality is untested. Lawyers argue the Act exempts research and testing; the Commission says an escaped test touching the EU market is enforceable anyway, and expects to litigate it. Marking standards are still settling.

The direction of travel, though, is no longer debatable. The first letters went out in September, the first named targets were announced in October, and the fines switch is already on. For developers shipping to Europe, the compliance era started while everyone was still writing about the roadmap.

Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.

Shopify Payout Reconciliation in Xero Explained

Muhammad Saud — Sun, 01 Nov 2026 14:25:44 +0000

Payout reconciliation is the process of proving that the lump sum Shopify deposits into your bank account equals the sales, fees, refunds and adjustments that produced it. You do this in Xero by routing everything through a clearing account, then matching each bank deposit to the payout it represents. When the numbers tie out, your books are clean.

Key takeaways

A Shopify payout is net, not gross: every deposit is sales minus fees, refunds, chargebacks and adjustments. It will never equal your invoice totals.
The correct Xero workflow uses a clearing account: post sales and fees to it, then match each bank feed deposit to the payout it represents.
When a payout does not match, check currency conversion, refund timing, chargebacks and duplicate invoices before blaming the tool.
Apps like A2X and Taxomate post one summarised entry per payout so matching is a single click, but they still need correct tax mapping.
Multi-currency stores using PayPal alongside Shopify Payments face the worst breakage: two settlement streams with different timing and exchange rates.

If you have ever stared at a bank deposit from Shopify and wondered why it matches nothing in Xero, you are not alone. In our research across 30 verified complaints about Shopify to Xero integrations, broken payout reconciliation was the third most common problem. One reviewer of Xero's own Shopify app put it bluntly: "Not even a single Shopify Payout matches." Another, a UK yarn shop, described multi-currency with PayPal as a setup where "it doesn't work" and ended up facing an end-of-year audit with broken books. This guide explains what is actually happening and how to fix it.

What is actually inside a Shopify payout?

A Shopify payout is a settlement batch, not a single order. Every few days, Shopify collects the orders it has processed, subtracts what it is owed, and deposits one lump sum into your bank account. You can see the breakdown in Shopify under Finance, then Payouts, by opening an individual payout: it lists sales, refunds, adjustments and a fee column for each transaction.

Here is the rough arithmetic of a typical payout:

Gross sales (the orders customers placed)      £8,420.00
- Shopify transaction fees                       -£146.30
- Refunds and returns                             -£85.00
- Chargebacks and adjustments                     -£40.00
--------------------------------------------------------
Payout deposited to your bank                   £8,148.70

Three things worth noting. First, fees come out before the money reaches you: the payout is net, so comparing it to gross invoice totals will always fail. Second, a payout covers a settlement window, not a calendar day or a single order, so the date on the deposit rarely lines up with the dates on your invoices. Third, Shopify sometimes deducts other charges, such as shipping label purchases, from the same payout. All of this is normal. The payout is correct; the mistake is expecting it to look like an invoice.

Why don't my Shopify payouts match my Xero invoices?

There are six usual reasons, and they stack on top of each other:

Fees were deducted before payout. Xero holds gross invoice values; the bank holds net. Until fees are recorded as expenses, nothing will match.
Timing windows differ. The payout covers orders from a settlement period that rarely aligns with the invoice dates Xero shows.
Refunds land in a later payout. A refund issued on Monday may be deducted from a payout you receive on Thursday, so the week looks wrong in both places.
Currency conversion. Shopify converts multi-currency orders at its own rates. Xero applies its own exchange rates to foreign-currency invoices. The two disagree by small amounts on nearly every order.
Separate gateways, separate payouts. PayPal settles on its own schedule with its own currency conversion. As the UK store in our research found, running multi-currency with PayPal alongside Shopify Payments is the worst combination: two settlement streams, different timing, different rates.
Duplicate invoices. If your integration created duplicate invoices in Xero, your sales are overstated and no payout will ever tie to them. This is more common than most store owners realise: one merchant paid an accountant hundreds of pounds to clean up duplicates an app had created.

How do you reconcile Shopify payouts in Xero correctly?

The method bookkeepers use is the clearing account workflow. A clearing account is a holding account that collects everything Shopify-related so the bank deposit has exactly one thing to match against. The approach below follows the same steps described in Synder's 2026 reconciliation guide and the PayoutClerk manual workflow:

Create a "Shopify Payments clearing" account in Xero as a current asset. You need a clearing account because Xero does not allow bank accounts inside manual journals.
Post a sales journal for the period. Take gross sales, discounts, refunds, shipping and taxes from Shopify's finance reports. Debit the clearing account and credit Sales, Shipping and Tax; debit Discounts and Refunds.
Record the payout's transactions. Export the payout from Shopify (Finance, Payouts, then the payout date) and sum the fee column plus any adjustment rows such as chargebacks.
Match the bank line. When the payout appears in your Xero bank feed, code it to the Shopify Payments clearing account for the deposited amount.
Post a fees journal. Debit Merchant fees and Chargebacks, credit the clearing account.
Check the clearing balance. It should equal only the transactions not yet paid out. If it is zero apart from pending payouts, you are reconciled.

Two mistakes to avoid. First, never code the payout deposit directly to Sales: that records net as revenue, hides the fees entirely, and double counts if you also posted the sales journal. Second, never close a month with a non-zero clearing balance. As one accounting guide puts it, a non-zero balance always means a payout, refund or fee was missed.

How do A2X, Amaka and Taxomate handle payouts?

These tools exist to automate the workflow above. A2X, the incumbent most accountants trust, pulls settlement data from Shopify and posts one summarised journal entry per payout into Xero, broken into sales, fees, refunds and taxes. When the deposit lands in your bank feed, it matches the A2X summary and reconciliation becomes a single click, according to A2X's own documentation. It starts at $29 per month for 200 orders and scales to $249 for multi-channel sellers, per our pricing check in November 2026.

Taxomate takes a similar approach: it imports the payout and posts one summarised invoice to Xero whose total matches the bank deposit, with lines grouped by category. Link My Books does the same at $21 per month. Amaka offers a free tier, which is why many stores try it first.

The honest caveat: these tools automate the posting, not the judgement. Tax mapping still has to be correct, especially for UK and EU VAT, and someone still needs to investigate when a payout does not tie out. They also charge monthly whether your books reconcile or not. In our complaint research, several merchants paid $20 to $65 per month and still ended up reconciling by hand because fees or taxes were miscategorised.

What should I check when a payout doesn't match?

Work through this list in order before rebuilding anything:

Are you comparing net to gross? Subtract fees first; the gap is usually smaller than it looks.
Does the payout window match the period you are checking? Pull the payout's transaction list from Shopify and confirm the dates.
Did a refund or chargeback land in this payout that belongs to an earlier period?
Are there duplicate invoices in Xero inflating sales? Search for duplicate order numbers.
For multi-currency stores: are exchange rates the source of the difference? Small per-order FX differences accumulate into visible gaps.
Are PayPal transactions mixed in? PayPal settles separately; reconcile it as its own stream.
Is the payout still pending at the bank? Shopify marks payouts as sent before banks show them, sometimes a day or two apart.

What is uncertain and what should you be careful about?

Exact payout timing varies by region, plan and bank; treat any stated number of days as typical, not guaranteed. Exchange rate differences between Shopify and Xero are a known source of small gaps that no integration fully eliminates. And if your books have been broken for months, the realistic fix is a clean cutover: reconcile one recent payout end to end, prove the workflow, then work backwards. Merchants who let payout reconciliation slide until year-end consistently report it turning into days of cleanup, and auditors do not accept "the app was wrong" as an explanation.

Reconciliation eating your week? We fix broken Shopify to Xero invoicing for a flat $149, free diagnosis, no monthly fees.

The core idea is simple once you see it: the payout is a net settlement, the clearing account is where the arithmetic happens, and the bank deposit is the control total. Set that up, reconcile weekly instead of yearly, and the mystery deposits become routine.

Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.

Automate Shopify Sales Reports with Python and SQL

Muhammad Saud — Sun, 01 Nov 2026 14:25:42 +0000

Automating Shopify sales reports means pulling order data through the Shopify Admin API with a Python script, storing it in a SQL database, and running scheduled queries that output daily revenue, product performance, refunds, and fees. It replaces manual CSV exports with a repeatable pipeline that a store owner with basic Python skills can run on a laptop or a small server.

Key takeaways

Manual CSV exports break as order volume grows: copy-paste mistakes, stale numbers, and hours of spreadsheet work every week.
A short Python script using the Shopify REST Admin API pulls every order with cursor-based pagination and stores it in SQLite or Postgres.
Five SQL queries cover what most stores actually need: daily revenue, revenue by product, refunds, fees versus net, and tax collected.
Shopify's rate limit is 2 requests per second with a 40-request bucket, so the script must pause and retry on 429 responses.
This pipeline is a reporting tool, not an accounting system: it does not create invoices in Xero and does not replace bookkeeping.
A cron job runs the whole thing automatically every morning before you open your laptop.

Why manual sales reporting breaks

Every Shopify store starts with the same ritual: open the admin, export orders to CSV, filter by date in a spreadsheet, build a pivot table, and email it to yourself or your accountant. It works until it does not. At a few hundred orders a month the export takes minutes and someone has to remember to do it. Copy a formula one row too far and revenue is wrong for a week before anyone notices. Save over the wrong file and last month's numbers are gone. Multi-currency stores have it worse: euros, pounds, and dollars land in a single column and the spreadsheet has no idea which is which. Then comes a sale or a holiday spike, the export times out, and the Monday report is built from half the data.

The deeper problem is that a CSV export is a photograph. The moment you download it, it is out of date. A pipeline, by contrast, is a live feed: the same script pulls the same fields, the same queries run, and the numbers are fresh every morning with no human in the loop to mistype them.

There is a cost beyond your own time. Accountants routinely charge by the hour to untangle a year of inconsistent CSVs at tax time, and every hour they spend fixing your spreadsheet is an hour you pay for. A pipeline that has been running all year hands them clean, consistent tables instead. The stores that feel this most are the ones that just had their first real spike: a Black Friday, a viral post, a large wholesale order. Volume is what turns an annoyance into a mess.

How do I connect Python to my Shopify store?

Shopify does not want your admin password inside a script, and neither should you. Instead you create a custom app, which is simply a named set of API permissions with its own token. In your Shopify admin go to Settings, then Apps and sales channels, then Develop apps, and create an app. Under Configuration, grant the Admin API scopes read_orders and, if you want fee data later, read_shopify_payments. Install the app and Shopify shows an Admin API access token starting with shpat_. It is displayed exactly once, so copy it somewhere safe immediately.

Treat the token like a password. Keep it in an environment variable, never hard-code it in the script, and never commit it to git. If it ever leaks, rotate it from the same screen in seconds. You need store-owner level permissions to create the app; a staff account with only order-viewing rights cannot do this step. The token is all the script needs. No OAuth dance, no browser login.

If you want to practise safely first, create a free development store from the Shopify Partners dashboard. It exposes the same API with sample orders, and nothing you do there touches your real shop. Get the script working there, then point it at production.

How do I pull every order without missing one?

The REST Admin API returns at most 250 orders per call and tells you where the next page starts through a Link response header containing a cursor token. Your script follows that cursor until no next page remains. Two details matter. First, pass status=any so cancelled and archived orders are included; without it you silently miss them. Second, respect the rate limit: Shopify's documentation states 2 requests per second with a 40-request bucket, and the API answers HTTP 429 with a Retry-After header when you go too fast. The script below pauses and retries instead of crashing. Request only the fields you need; it keeps responses small and fast.

import time
import requests

SHOP = "your-store.myshopify.com"   # your shop domain
TOKEN = "shpat_xxxxxxxxxxxxxxxx"     # Admin API access token, keep secret
VERSION = "2024-10"                  # pin a dated API version
HEADERS = {"X-Shopify-Access-Token": TOKEN}
BASE = f"https://{SHOP}/admin/api/{VERSION}"

def get_json(url):
    """GET with backoff on 429. Shopify allows 2 req/s, 40-request bucket."""
    while True:
        r = requests.get(url, headers=HEADERS, timeout=30)
        if r.status_code == 429:
            time.sleep(int(r.headers.get("Retry-After", 2)))
            continue
        r.raise_for_status()
        return r

def fetch_all_orders():
    url = (BASE + "/orders.json?status=any&limit=250"
           "&fields=id,name,created_at,total_price,subtotal_price,"
           "total_tax,total_discounts,financial_status,currency,"
           "refunds,line_items")
    orders = []
    while url:
        r = get_json(url)
        orders.extend(r.json()["orders"])
        url = None
        for part in r.headers.get("Link", "").split(","):
            if 'rel="next"' in part:
                url = part[part.find("")]
    return orders

For the daily run you do not need the full history every time. Remember the newest created_at you have stored, and on the next run pass it as created_at_min so the API returns only new orders. The first ever run backfills everything; every run after that is incremental and takes seconds.

Storing orders in a SQL database

A script that prints numbers to the terminal is a toy. A database turns each pull into an asset: you can ask about last Tuesday, join orders to line items, and build any report without hitting the API again. SQLite needs no server and comfortably handles hundreds of thousands of orders on a laptop; the schema and queries below run unchanged on Postgres when you outgrow it. Three tables are enough: orders, line items, and refunds. The loader uses INSERT OR REPLACE keyed on Shopify's own IDs, so running it twice never duplicates a row, it only refreshes orders that changed. Two small indexes keep the daily queries fast as the tables grow.

import sqlite3

con = sqlite3.connect("shopify.db")
cur = con.cursor()

cur.executescript("""
CREATE TABLE IF NOT EXISTS orders (
    id INTEGER PRIMARY KEY,
    name TEXT,
    created_at TEXT,
    total_price REAL,
    subtotal REAL,
    tax REAL,
    discounts REAL,
    financial_status TEXT,
    currency TEXT
);
CREATE TABLE IF NOT EXISTS line_items (
    id INTEGER PRIMARY KEY AUTOINCREMENT,
    order_id INTEGER,
    product_title TEXT,
    quantity INTEGER,
    price REAL
);
CREATE TABLE IF NOT EXISTS refunds (
    id INTEGER PRIMARY KEY,
    order_id INTEGER,
    created_at TEXT,
    amount REAL
);
CREATE INDEX IF NOT EXISTS idx_orders_date ON orders(date(created_at));
CREATE INDEX IF NOT EXISTS idx_items_order ON line_items(order_id);
""")

def save_orders(orders):
    for o in orders:
        cur.execute("""
            INSERT OR REPLACE INTO orders
            (id, name, created_at, total_price, subtotal, tax,
             discounts, financial_status, currency)
            VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)
        """, (o["id"], o["name"], o["created_at"],
              float(o["total_price"]), float(o["subtotal_price"]),
              float(o["total_tax"]), float(o["total_discounts"]),
              o["financial_status"], o["currency"]))
        cur.execute("DELETE FROM line_items WHERE order_id = ?", (o["id"],))
        for li in o.get("line_items", []):
            cur.execute("""
                INSERT INTO line_items (order_id, product_title, quantity, price)
                VALUES (?, ?, ?, ?)
            """, (o["id"], li["title"], li["quantity"], float(li["price"])))
        for rf in o.get("refunds", []):
            amount = sum(float(t["amount"])
                         for t in rf.get("transactions", [])
                         if t["kind"] == "refund")
            cur.execute("""
                INSERT OR REPLACE INTO refunds (id, order_id, created_at, amount)
                VALUES (?, ?, ?, ?)
            """, (rf["id"], o["id"], rf["created_at"], amount))
    con.commit()

Two practical notes. Money is stored as REAL here for readability; if you ever do serious accounting maths, store minor units (cents) as integers to avoid floating point surprises. And created_at arrives as ISO 8601 with a timezone offset, which SQLite's date() parses and normalises to UTC, so stores far from UTC may want to shift the grouping with a modifier such as date(created_at, '+1 hour').

What are the five reports every store owner needs?

With the data in place, the reports are plain SQL. These five answer the questions store owners actually ask every week.

1. Daily revenue

How much did we make yesterday, and is the trend healthy? Only count orders that were actually paid.

SELECT date(created_at) AS day,
       COUNT(*) AS orders,
       ROUND(SUM(total_price), 2) AS gross_revenue
FROM orders
WHERE financial_status IN ('paid', 'partially_paid', 'partially_refunded')
GROUP BY day
ORDER BY day DESC
LIMIT 30;

2. Revenue by product

Which products earn their shelf space and which just take it up?

SELECT product_title,
       SUM(quantity) AS units_sold,
       ROUND(SUM(quantity * price), 2) AS revenue
FROM line_items
GROUP BY product_title
ORDER BY revenue DESC
LIMIT 20;

3. Refunds

How much are we giving back, and is it climbing? A rising refund line is often the first sign of a product or shipping problem.

SELECT date(created_at) AS day,
       COUNT(*) AS refund_count,
       ROUND(SUM(amount), 2) AS refunded
FROM refunds
GROUP BY day
ORDER BY day DESC
LIMIT 30;

4. Fees versus net

What did we really keep after the payment gateway took its cut? Fee data lives in the Shopify Payments balance transactions endpoint, which needs the read_shopify_payments scope. Note the honest limit: this covers Shopify Payments only. PayPal and manual gateways report fees elsewhere.

def fetch_balance_transactions():
    url = BASE + "/shopify_payments/balance/transactions.json?limit=100"
    txns = []
    while url:
        r = get_json(url)
        txns.extend(r.json()["transactions"])
        url = None
        for part in r.headers.get("Link", "").split(","):
            if 'rel="next"' in part:
                url = part[part.find("")]
    return txns

cur.execute("""
CREATE TABLE IF NOT EXISTS balance_txns (
    id INTEGER PRIMARY KEY,
    source_order_id INTEGER,
    type TEXT,
    amount REAL,
    fee REAL,
    net REAL,
    currency TEXT,
    payout_date TEXT
)
""")

def save_balance(txns):
    for t in txns:
        cur.execute("""
            INSERT OR REPLACE INTO balance_txns
            (id, source_order_id, type, amount, fee, net, currency, payout_date)
            VALUES (?, ?, ?, ?, ?, ?, ?, ?)
        """, (t["id"], t.get("source_order_id"), t["type"],
              float(t["amount"]), float(t["fee"]), float(t["net"]),
              t["currency"], t["payout_date"]))
    con.commit()

SELECT date(payout_date) AS day,
       ROUND(SUM(amount), 2) AS gross,
       ROUND(SUM(fee), 2) AS fees,
       ROUND(SUM(net), 2) AS net_received
FROM balance_txns
WHERE type IN ('charge', 'refund') AND payout_date IS NOT NULL
GROUP BY day
ORDER BY day DESC
LIMIT 30;

5. Tax collected

How much VAT or sales tax did we collect? Your accountant will ask for this every quarter, and now it is one query instead of an afternoon.

SELECT date(created_at) AS day,
       ROUND(SUM(tax), 2) AS tax_collected
FROM orders
WHERE financial_status IN ('paid', 'partially_paid', 'partially_refunded')
GROUP BY day
ORDER BY day DESC
LIMIT 30;

Need to hand numbers to an accountant? Dump any query straight to CSV with Python's built-in csv module, no extra packages required:

import csv

rows = cur.execute(
    "SELECT * FROM orders ORDER BY created_at DESC LIMIT 1000")
with open("orders_export.csv", "w", newline="") as f:
    w = csv.writer(f)
    w.writerow([d[0] for d in rows.description])
    w.writerows(rows)

Where to look first each morning: compare net received against gross revenue to see what fees and refunds actually cost you, and watch the refund rate (refunded divided by gross) week over week. A refund rate creeping upward is usually a product quality or shipping problem wearing a finance costume.

How do I run this every morning automatically?

A report you have to remember to run is a report that stops being run. On Linux or macOS, one cron line runs the script at 7am every day and appends output to a log file:

0 7 * * * /usr/bin/python3 /home/store/shopify_reports.py >> /home/store/reports.log 2>&1

On Windows, Task Scheduler does the same job with a basic daily trigger. Tie the pieces together with a small entry point:

if __name__ == "__main__":
    orders = fetch_all_orders()
    save_orders(orders)
    print(f"Saved {len(orders)} orders")
    save_balance(fetch_balance_transactions())
    print("Balance transactions updated")

The first run backfills your entire order history, which can take a few minutes on large stores. Later runs pass created_at_min so they only fetch what is new. Because every write is an upsert, reruns are safe: changed orders refresh, nothing duplicates.

Want the numbers in your inbox instead of a log file? Have the script attach the CSV export and send it with Python's built-in smtplib through your email provider. The whole pipeline then needs nothing but the standard library plus requests.

What this pipeline does not do

Honest limits, stated plainly. This is a reporting pipeline, not an accounting system: it does not create invoices in Xero, reconcile payouts against bank feeds, or file VAT returns. Amounts stay in each order's original currency; the script does not convert them. Fee data covers Shopify Payments only, not PayPal or manual gateways. Shopify versions its API every quarter, so the script pins a dated version such as 2024-10; check the changelog before upgrading. And refunds only appear after the order is re-pulled, which is why the loader refreshes instead of appending. For daily decisions this pipeline is plenty. For the books themselves, you still need proper accounting.

Want this wired into your actual books? We fix broken Shopify to Xero invoicing for a flat $149 — free diagnosis, no monthly fees.

Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.

DeepSeek Super-Agent: Coding Agents on an Attention Budget

Muhammad Saud — Sun, 01 Nov 2026 04:00:41 +0000

DeepSeek's super-agent is an open-source coding-agent prototype released on 31 October 2026 that runs background coding sessions against a single GitHub repository. Instead of optimising for benchmark scores, its design optimises for human attention: every interaction with a developer costs budget, and the whole system is built to spend that budget sparingly.

Key takeaways

On 31 October 2026, DeepSeek's Harness team open-sourced dsh-superagent-prototype under the MIT licence: a proactive coding agent that works a single GitHub repository in parallel background sessions.
The headline idea is not a new model but a new constraint. The design minimises the human attention each autonomous action costs, instead of maximising benchmark pass rates.
The architecture is a two-layer router-worker design built from five primitives (Sources, Triggers, Threads, Mandates, Tickets), with a stateless evaluator and immutable mandates that bound each worker's authority.
Each worker session runs in an isolated git worktree. Shell commands longer than five seconds move to the background automatically and are killed after a hard thirty-minute cap.
The catches are real: workers run shell commands as your OS user with no approval prompt, DeepSeek published no independent benchmarks, and the README states it is a research prototype, not a finished product.

What did DeepSeek actually release?

On 31 October 2026, DeepSeek's Harness team published a repository called dsh-superagent-prototype on GitHub under the MIT licence. It is a proactive coding agent packaged as a bundle for DeepSeek Harness, the company's plugin-oriented agent framework. Point it at one GitHub repository, and it listens to issues, pull-request comments and webhooks (plus, optionally, the Chinese workplace chat tool Feishu). When an allowed person asks for work, the agent opens a work item, runs in its own session and working directory, and answers in the original conversation, for example by opening a pull request.

Alongside the code, the Harness team published a technical article titled "Super-agent Design Exploration: Toward More Autonomous Agent Systems" (via a WeChat post on 31 October, reported by Tech in Asia). Its argument: mainstream agents run in isolated sessions, so multi-task collaboration is capped by human attention. The super-agent moves session management into the system and reserves humans for decisions.

According to DeepSeek, an internal instance has been running continuously: reproducing and fixing bugs, responding to code reviews, and iterating on DeepSeek Harness itself. The company stresses the open-sourced code is a research prototype for reference, not an official product, and differs from the internal system. It may contain bugs.

How does the super-agent's router-worker design work?

The design documents in the repository (docs/000-philosophy.md and docs/001-domain.md, analysed by Reid Marlow) describe a two-layer architecture: a main session with a small tool set routes, while dedicated worker sessions code. The pieces that matter:

The five primitives

The domain model separates event streams from permissions with five concepts: Sources, Triggers, Threads, Mandates and Tickets. A Source (GitHub notifications, issue webhooks) emits immutable Triggers into an ordered Thread. A stateless Eval component checks each Trigger against repository policy and decides whether to create a Ticket. An accepted Ticket receives an immutable Mandate fixing its scope and operational strength.

Stateless evaluation

Most multi-agent frameworks keep pending queues, retry timers and polling loops inside the planning agent. DeepSeek rejected internal scheduler state entirely: Eval holds nothing between runs, takes a batch of Triggers, evaluates them against policy, and decides immediately. Waiting on an external event (code-review approval, CI pipeline) is modelled as an ordinary event Source; when the event occurs, it emits a Trigger that enters the pipeline naturally. The payoff is restart safety: after a mid-deployment restart, the Source recovers its read position from git history and webhook offsets.

Subscriptions that cannot widen authority

Context delivery is separated from permission checks through subscriptions: a Ticket subscribes to the Threads it touches, so a PR comment or finished CI build fans out directly to the worker as context. This fanout bypasses the evaluator but cannot widen the Ticket's authority: the Mandate stays fixed. If a comment asks for out-of-scope work, the worker hands it back to the router instead of self-escalating, defending against conversational drift, where a bugfix quietly grows into a multi-file refactor.

Workers in git worktrees, with hard time limits

Each Ticket gets its own working directory, provisioned as an isolated git worktree. Any shell command running longer than five seconds moves to a background job automatically, so installs and test suites cannot block incoming messages. Background jobs face a hard thirty-minute cap, after which the runtime kills them and records the output in an audit trail. One media-reported internal test claims p95 completion of 15.7 minutes versus 29.5 for a single-session setup. DeepSeek published no formal benchmarks, so treat that as a company claim, not a verified result.

What is an "attention budget" and why is it new?

The design's most interesting idea is budgeting human attention the way other frameworks budget tokens or tool calls. Against real team repositories, developer fatigue is the primary failure mode: an agent posting twenty notifications, opening four draft pull requests and asking five vague questions burns more engineering time than writing the code by hand.

DeepSeek's answer is explicit human handoffs: the runtime never parses free-text transcripts to guess whether an agent needs help. An agent that needs a human writes to a structured attention field; the Ticket then waits, releases its worker slot, and posts a single structured question back to the original channel. A web panel shows only items requiring human action, keeping healthy tickets out of sight.

Policy changes get the same respect: updating rules mid-flight does not kill active Tickets immediately (which would leave dirty worktrees). Instead, a policy-change notification reaches the active assignment, letting the worker checkpoint or roll back cleanly before terminating.

What are the catches?

Before you point this at your production repository, know what the release notes actually say.

Workers run shell commands with no approval, as your OS user. The execution documentation states that worker sessions execute commands without approval directly as the operator's OS user. That is a serious trust decision for any engineering team: you are giving an experimental agent the same shell you have.
No benchmarks. DeepSeek explicitly provides no benchmark results for the prototype. The only performance numbers in circulation come from media reports of an internal test, not from a reproducible evaluation.
Single-repository scope. The prototype is designed for one GitHub repo at a time. Teams with dozens of services will need to run multiple instances or rethink the model.
It is a prototype. Roughly 97.5 percent TypeScript by bytes (per media analysis), grown out of DeepSeek's own daily work. The README warns it may contain bugs and is not identical to the internal system.

The hype metric, for the record: media reported roughly 2,000 stars and 170 forks within the first twelve hours. That measures interest, not quality.

Try the core idea in your own agents: a 40-line attention budget

You do not need DeepSeek's codebase to apply its best idea. An attention budget is a rate limit on human interruptions plus a scope guard, and both fit in plain Python 3:

from dataclasses import dataclass, field
import time

@dataclass(frozen=True)
class Mandate:
ticket_id: str
allowed_paths: tuple[str, ...]
max_questions_per_hour: int = 2

def in_scope(mandate: Mandate, path: str) -> bool:
"""True only if the path stays inside the mandate's scope."""
return any(path.startswith(p) for p in mandate.allowed_paths)

@dataclass
class AttentionBudget:
asked: list[float] = field(default_factory=list)

def can_ask(self, limit: int) -> bool:

    now = time.time()

    self.asked = [t for t in self.asked if now - t < 3600]

    return len(self.asked) < limit

def ask(self, mandate: Mandate, question: str) -> str | None:

    # Returns ONE structured question, or None if the budget is spent.

    if not self.can_ask(mandate.max_questions_per_hour):

        return None  # budget exhausted: checkpoint instead of asking

    self.asked.append(time.time())

    return f"[{mandate.ticket_id}] {question}"

Example: the worker may touch src/payments only, and may ask twice per hour.

mandate = Mandate(ticket_id="T-42", allowed_paths=("src/payments/",))
budget = AttentionBudget()

if in_scope(mandate, "src/payments/refund.py"):
question = budget.ask(
mandate,
"Which refund policy applies to partial SEPA payments: v1 or v2?"
)
print(question or "Budget spent: checkpointing and waiting for a review event.")

The pattern generalises: scope checks separate from routing, explicit human-handoff paths instead of parsed transcripts, and asking a human as a budgeted operation. The super-agent's philosophy in miniature.

Why this matters for teams in Europe

For European teams, two things stand out. First, the MIT licence and self-hostable design let you run it on your own infrastructure, simplifying the EU data-sovereignty conversation around AI tooling. Second, the pain is universal: teams drowning in agent notifications. The attention-budget principle is the cheapest, highest-leverage idea in this release, and you can adopt it this week without installing anything from DeepSeek.

If you build agent infrastructure, read docs/000-philosophy.md and docs/001-domain.md in full. The value is in the plumbing: stateless evaluators, immutable mandates, subscription fanout, isolated worktrees. No model magic required.

Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.

Hybrid Search for RAG: BM25 + Vectors in 50 Lines of Python

Muhammad Saud — Sat, 31 Oct 2026 03:58:34 +0000

Hybrid search runs keyword search and vector search in parallel, then merges the two ranked lists into one, usually with Reciprocal Rank Fusion. It matters because the two methods fail in opposite ways: vector search misses product codes and proper nouns, while keyword search misses rephrased questions. The fusion recovers both signals for a few milliseconds.

Key takeaways

BM25 catches exact terms (codes, names, identifiers) that embeddings routinely miss; dense vectors catch paraphrases that keyword search misses. Their failures barely overlap.
You cannot add a BM25 score and a cosine similarity together: one is unbounded and corpus-dependent, the other is bounded. Fuse ranks, not scores.
Reciprocal Rank Fusion, RRF(d) = sum of 1/(k + rank), needs no score normalisation and takes about 15 lines of Python, with k = 60 by convention.
Most vector databases already ship hybrid search natively, but building it by hand shows you exactly what is happening under the hood.
The gains are real but corpus-dependent: measure recall on your own documents before assuming hybrid wins.

Why pure vector search is not enough

Embeddings are trained to capture meaning, and that training makes them careless about surface form. A dense retriever asked about ticket KB-2024-7831 will happily return three articles about ticket-tracking systems, none of which contains the string KB-2024-7831, while a BM25 index returns the exact article in position one. That example comes from a 2026 implementation write-up on dev.to, and it matches what production RAG teams report: the single most common retrieval bug is an exact identifier that the vector search never finds.

BM25 has the mirror problem. It scores documents by token overlap, term frequency and document length, so "how do I reset my password" matches a page titled "Account recovery steps" only if the words line up. A 2026 practitioner post on Medium put it well: dense retrieval catches paraphrases and conceptual matches, while sparse retrieval catches the literal strings that dense retrieval was never built to represent. The two families fail on almost disjoint query types, which is precisely why combining them works.

Why can't you just add the two scores together?

This is the obvious first idea, and it breaks on the arithmetic. BM25 scores are unbounded positive numbers tied to your corpus statistics: in my demo below, the winning BM25 score is 7.289 and the losers score 0.000. Cosine similarities live in the range [-1, 1] and cluster narrowly in practice: the same demo's winning dense score is 0.584. A BM25 score of 7.289 means nothing without knowing your corpus's term-frequency distribution, so any min-max or z-score normalisation you apply before adding is fragile and shifts whenever your corpus changes.

The elegant escape is to ignore the scores entirely and keep only the ranks. That is Reciprocal Rank Fusion (RRF), introduced by Cormack, Clarke and Büttcher in a 2009 SIGIR paper. For each document, you add up 1/(k + rank) across every retriever, with k = 60 by long-standing convention. A document ranked first by both methods scores 1/61 + 1/61 = 0.0328, the maximum. A document ranked first by one method and 50th by the other scores 1/61 + 1/110 = 0.0255, still respectable. Ranks are on the same scale by definition, so the fusion never needs calibration. That property is why RRF has survived 17 years of retrieval research essentially unchanged.

The 50-line implementation

Before reaching for a vector database's native hybrid mode, build the whole thing by hand once. The code below implements BM25 from scratch (Robertson and Zaragoza's formulation, k1 = 1.5, b = 0.75), a TF-IDF cosine stand-in for the dense side, and RRF fusion. In production you would swap the TF-IDF vectors for real embeddings; the fusion code is byte-for-byte identical, because RRF only ever sees rank orders.

import math, re
from collections import Counter

def tokenize(text):
return re.findall(r"[a-z0-9]+", text.lower())

class Corpus:
K1, B = 1.5, 0.75

def __init__(self, docs):
    self.docs = docs
    self.toks = [tokenize(d) for d in docs]
    self.doc_len = [len(t) for t in self.toks]
    self.avgdl = sum(self.doc_len) / len(self.doc_len)
    self.df = Counter(t for toks in self.toks for t in set(toks))
    self.vocab = sorted(self.df)
    self.N = len(docs)

def bm25(self, query):                      # sparse side
    qtoks = tokenize(query)
    scores = []
    for i, toks in enumerate(self.toks):
        tf = Counter(toks)
        s = 0.0
        for t in qtoks:
            if t not in tf:
                continue
            idf = math.log(1 + (self.N - self.df[t] + 0.5)
                             / (self.df[t] + 0.5))
            s += idf * (tf[t] * (self.K1 + 1)) / (
                tf[t] + self.K1 * (1 - self.B
                     + self.B * self.doc_len[i] / self.avgdl))
        scores.append(s)
    return scores

def dense(self, query):                     # TF-IDF cosine stand-in
    qv = self._tfidf(tokenize(query))
    return [self._cos(qv, self._tfidf(t)) for t in self.toks]

def _tfidf(self, toks):
    tf = Counter(toks)
    return [tf[t] * math.log(1 + self.N / self.df[t])
            if t in self.df else 0.0 for t in self.vocab]

@staticmethod
def _cos(a, b):
    na = math.sqrt(sum(x*x for x in a))
    nb = math.sqrt(sum(x*x for x in b))
    return sum(x*y for x, y in zip(a, b)) / (na*nb) if na and nb else 0.0

def rrf(ranked_lists, k=60): # the fusion
scores = {}
for ranked in ranked_lists: # each list: best doc first
for rank, doc_id in enumerate(ranked, start=1):
scores[doc_id] = scores.get(doc_id, 0.0) + 1.0 / (k + rank)
return sorted(scores, key=scores.get, reverse=True)

def ranks(scores):
return sorted(range(len(scores)),
key=lambda i: scores[i], reverse=True)

def hybrid_search(corpus, query, top_n=3):
fused = rrf([ranks(corpus.bm25(query)),
ranks(corpus.dense(query))], k=60)
return fused[:top_n]

What it actually finds

Running this against an 8-document toy corpus (database docs, one with the identifier KB-2024-7831, one refund policy, one team-lunch notice) produces real output, not a sketch:

BM25 for: "KB-2024-7831 connection failure"
#1 doc3 (score=7.289) KB-2024-7831: connection pool exhaustion on the EU Frankfurt rep...
#2 doc0 (score=0.000) PostgreSQL full text search uses tsvector and tsquery...

dense (TF-IDF proxy) for: "KB-2024-7831 connection failure"
#1 doc3 (score=0.584) KB-2024-7831: connection pool exhaustion on the EU Frankfurt rep...
#2 doc0 (score=0.000) PostgreSQL full text search uses tsvector and tsquery...

-- RRF fused --
#1 doc3 KB-2024-7831: connection pool exhaustion on the EU Frankfurt rep...

Two things to notice. First, the identifier query is exactly the case where dense retrieval fails in production; my TF-IDF stand-in still finds it here only because the identifier tokens overlap, and a real embedding model would be far more likely to miss it. Second, look at the raw scores: 7.289 versus 0.584. Had you added them, the BM25 side would have swallowed the dense side whole. RRF's rank-only math puts doc3 first without ever comparing those incompatible numbers.

Does hybrid actually beat dense-only? What measurements exist

Honest answer first: it depends on your corpus, and you should measure. The published numbers are encouraging but small-scale. One practitioner ran 12 questions against a company earnings transcript in October 2026 and reported Hit@3 rising from 9/12 (BM25) and 10/12 (dense) to 12/12 (hybrid), with mean reciprocal rank moving from 0.66 and 0.73 to 0.86. Another October 2026 write-up headlined "30% better recall" from combining BM25 with vectors, and a hybrid RAG guide reports roughly 91% recall@10 with BM25 plus RRF and no score-normalisation tricks. Treat these as existence proofs, not guarantees: they are single-corpus experiments by individual authors, and the 91% figure comes from a vendor-adjacent guide whose exact test setup is not fully disclosed.

The pattern across them is consistent though: hybrid never loses to either side alone by much, and it wins clearly whenever the query set mixes identifiers with paraphrases. That is the realistic expectation to carry into your own evaluation.

What hybrid search costs you

The cost is smaller than most engineers assume. Running both retrievers in parallel adds roughly 6 milliseconds at p50 over dense-only search, according to the 2026 dev.to implementation write-up. A cross-encoder reranker on the top 50 fused candidates adds another 50 to 200 milliseconds depending on the model, per the AI Workflow Lab hybrid guide, but the reranker is optional: RRF alone is already the production default in many systems.

The real costs are operational. You maintain two indexes instead of one, you pick a candidate depth per side (top 50 to 100 each is the usual range), and you choose k for RRF. The honest truth is that nobody knows the optimal k for your corpus without trying: 60 is a convention that works well across domains, not a tuned constant. If you would rather tune than fuse by rank, weighted score fusion is the alternative: Weaviate weights dense at 0.75 and Pinecone at 0.5 in their defaults, per their documentation. RRF's advantage is that it needs no such calibration step.

When should you skip hybrid search?

Not every RAG system needs it. Skip the hand-rolled version when your vector database already does it natively: Qdrant, Weaviate, Elasticsearch, OpenSearch and Milvus all support BM25 or sparse vectors alongside dense vectors in the same collection, with native RRF fusion. That is the right call for most teams.

Skip hybrid entirely when your corpus is small and homogeneous enough that one retriever covers it, or when your queries are genuinely uniform, for example an internal FAQ where every question is phrased like the answers. And if Postgres is already your database, note that pgvector plus tsvector gives you dense and keyword search in one service, as the dev.to write-up demonstrates, so hybrid does not have to mean a second piece of infrastructure.

The production version of this pipeline

The shape that has settled in production RAG systems by 2026 is four stages: generate candidates from both a sparse and a dense index in parallel, fuse with RRF, optionally rerank the top 50 with a cross-encoder, then hand the winners to the LLM. LangChain's EnsembleRetriever follows exactly this pattern, using RRF under the hood with k = 60 hardcoded and a weights parameter that scales each retriever's contribution before fusion.

The code in this article is the whole of stage two and a from-scratch stage one. Once you have seen RRF work on your own corpus, the vector database's "hybrid mode" stops being magic and becomes a config option you can reason about. That understanding is worth more than the 50 lines.

Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.

Evaluating RAG With Ragas: A Hands-On Guide

Muhammad Saud — Fri, 30 Oct 2026 04:00:57 +0000

Ragas (short for RAG Assessment) is an open-source Python framework for evaluating retrieval-augmented generation pipelines, published by Shahul Es and colleagues and presented at EACL 2024. It uses a judge language model to score your pipeline on groundedness and relevance, so you can tell whether a bad answer came from broken retrieval or a hallucinating generator.

Key takeaways

Text-overlap scores like BLEU and ROUGE cannot judge RAG: they punish correct paraphrases and miss hallucinations. LLM-as-judge metrics are the 2026 standard.
Ragas scores two failure surfaces separately: faithfulness plus answer relevancy diagnose the generator, while context precision plus context recall diagnose the retriever.
On Ragas 0.4.3 you import metric classes from ragas.metrics.collections, wrap a judge model with llm_factory(), and build datasets from SingleTurnSample objects. Old v0.1 and v0.2 tutorials will not run.
Sensible starting thresholds are faithfulness above 0.80, answer relevancy above 0.85, context precision above 0.75 and context recall above 0.80. Calibrate from your own baseline, not from a blog post.
Scores are estimates, not ground truth: they inherit the judge model's biases and drift when the judge changes. Keep the judge fixed and re-baseline on change.

Why BLEU and ROUGE are the wrong ruler for RAG

Traditional NLP metrics compare generated text to a reference answer token by token, which fails twice for RAG. A correct answer phrased differently scores badly, while a fluent answer that invents facts can score well, which is exactly the failure RAG exists to prevent.

Ragas takes a different route: it asks a judge language model to score each answer along four axes. Two axes check the generator (is the answer grounded in what was retrieved, and does it address the question?), and two check the retriever (were the right chunks fetched, and were they ranked well?). This matters at scale: according to LangChain's State of AI 2026 report, 62 per cent of production LLM applications now include retrieval augmentation.

What are the four Ragas metrics?

Written against Ragas 0.4.3, the current release as of October 2026. The API changed substantially between 0.1, 0.2 and 0.4, so treat older tutorials as suspect.

Faithfulness: is the answer grounded?

Faithfulness, also called groundedness, checks whether every claim in the generated answer is supported by the retrieved context. Ragas splits the answer into atomic statements and verifies each one against the chunks, returning a score between 0 and 1. A faithfulness of 0.6 means roughly 40 per cent of the statements have no basis in the retrieved material, which is hallucination in the strict sense. Most production teams require 0.80 or higher, and regulated domains such as finance or healthcare should demand 0.90.

Answer relevancy: does it address the question?

Answer relevancy measures whether the response actually answers what was asked, independent of whether the facts are right. It combines a judge LLM with embedding similarity. This metric catches a common failure: an answer that is perfectly grounded in the context but answers a different question, which happens when prompts drift or the generator latches onto the wrong chunk.

Context precision: were the chunks ranked well?

Context precision checks that the relevant chunks appear near the top of the retrieval ranking. Unlike the first two metrics, it needs a ground-truth reference answer to decide which chunks count as relevant. Use it when you suspect the retriever finds the right documents but buries them below noise.

Context recall: was anything missed?

Context recall measures whether the retriever fetched all the information needed to answer the question, again against a ground-truth reference. Low recall with high precision is a classic chunking problem: your chunks are good, but the split cut the answer across boundaries the retriever cannot bridge.

Which metrics should you run first?

If you have no reference answers yet, start with faithfulness and answer relevancy. Both are reference-free: they need only the question, the retrieved contexts and the generated answer. They catch the failures users actually feel, which are hallucinated or off-topic answers. Once you have a golden set of question and reference-answer pairs, add context precision and context recall to start tuning the retriever.

How do you evaluate a RAG pipeline with Ragas?

Install the current release and set a judge model key:

pip install ragas
export OPENAI_API_KEY=sk-your-key-here

The cheapest serious judge is a small model such as gpt-4o-mini or an OpenAI-compatible local model; it needs to follow evaluation instructions reliably, not to write poetry. Next, run your own pipeline over a set of test questions and capture what it retrieved and generated. Each result becomes one SingleTurnSample:

from ragas import SingleTurnSample, EvaluationDataset

samples = [
    SingleTurnSample(
        user_input="Which port does the SettlePost API use in production?",
        retrieved_contexts=[
            "The SettlePost production API listens on port 8443 behind the load balancer.",
            "Staging uses port 8080. Development servers run on 3000.",
        ],
        response="The SettlePost production API runs on port 8443.",
        reference="The production API runs on port 8443.",
    ),
    # ... one sample per test question, ideally 50 to 100
]

dataset = EvaluationDataset(samples=samples)

Then pick metrics and a judge, and run the evaluation:

from ragas import evaluate
from ragas.llms import llm_factory
from ragas.metrics.collections import (
    Faithfulness,
    AnswerRelevancy,
    ContextPrecisionWithReference,
    ContextRecall,
)

judge = llm_factory("gpt-4o-mini")

result = evaluate(
    dataset,
    metrics=[
        Faithfulness(llm=judge),
        AnswerRelevancy(llm=judge),
        ContextPrecisionWithReference(llm=judge),
        ContextRecall(llm=judge),
    ],
)

print(result)
# {'faithfulness': 0.93, 'answer_relevancy': 0.91,
#  'context_precision': 0.83, 'context_recall': 0.79}

For debugging one bad answer, every metric exposes an async single_turn_ascore() method that takes a single SingleTurnSample.

How do you read the scores like a diagnosis?

The point of the four-metric split is localisation: the pattern of failures tells you which component to fix. Use this table as a starting point.

What you seeWhat it meansWhat to tryLow faithfulness, high context recallThe retriever found the facts, but the generator ignored themTighten the system prompt to cite context, lower temperature, or switch to a more instruction-following generatorLow faithfulness, low context recallThe generator had nothing good to work withFix retrieval first: this is not a prompting problemHigh faithfulness, low answer relevancyGrounded answer to the wrong questionCheck prompt drift, rewrite ambiguous test questions, inspect which chunk the generator latched ontoLow context precision, high context recallRight documents, wrong rankingAdd a cross-encoder reranker or tune hybrid search weightsLow context recallThe retriever never found the needed chunkRevisit chunking strategy and chunk size, expand top-k, add BM25 hybrid search

What does a Ragas score not tell you?

Honesty first: these scores are estimates produced by a model, not measurements against ground truth. Three caveats matter in practice.

First, judge bias. A judge model tends to favour answers written in its own style and can be fooled by confident phrasing, so a rising faithfulness score after swapping the judge may be flattery, not progress. Keep the judge fixed across runs and re-baseline your thresholds whenever you change it.

Second, cost and latency. Every metric on every sample is one or more LLM calls, plus embedding calls for the relevancy metrics. A hundred-question suite with four metrics is cheap; a ten-thousand-question regression suite with a frontier judge is a budget line. Most teams run a small golden set on every release and a larger sample nightly.

Third, test data quality. Ragas ships a TestsetGenerator that can synthesise question and answer pairs from your documents, which is a fast start, but synthetic questions mirror the generator's assumptions. Treat synthetic sets as scaffolding and grow a set of real user queries over time.

Can you gate releases on Ragas scores?

Yes, and this is where evaluation pays for itself. Store the scores from your golden set as a baseline and fail the build when any metric drops beyond an agreed margin. A minimal CI step looks like this:

# run this after building your RAG service, before deploy
python -m pytest tests/test_rag_eval.py -q

# inside test_rag_eval.py
def test_rag_quality_gate():
    result = evaluate(dataset, metrics=[...])  # as above
    assert result["faithfulness"] > 0.80, f"Faithfulness regressed: {result['faithfulness']}"
    assert result["answer_relevancy"] > 0.85
    assert result["context_precision"] > 0.75
    assert result["context_recall"] > 0.80

Thresholds deserve one warning: pick them from your own baseline, not from this article. A pipeline answering simple lookups can hold 0.95 faithfulness easily; one doing multi-hop reasoning over messy PDFs may stabilise at 0.82. What matters is the trend, and the gate catches it before your users do.

Evaluation is the difference between a RAG demo and a RAG product. Demos get judged by a vibe check on five hand-picked questions; products get judged by thousands of users on questions you never imagined. Ragas gives you the vocabulary and the tooling to close that gap: build the dataset, run the four metrics, read them diagnostically, and put the gate in CI.

Sources: Ragas documentation (docs.ragas.io); the DataCamp tutorial on RAG evaluation with Ragas 0.4.3; aiworkshack's hands-on guide to Ragas metrics on 0.4.x; the dev.to walkthrough of the four core metrics; FutureAGI's 2026 survey of LLM evaluation frameworks; and the Ragas paper by Shahul Es and colleagues, presented at EACL 2024.

Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.

Polars vs Pandas in 2026: Should You Switch?

Muhammad Saud — Thu, 29 Oct 2026 03:56:50 +0000

Polars is a DataFrame library written in Rust that processes columns in parallel using the Apache Arrow memory format, while pandas is the long-established Python DataFrame library built on NumPy. In 2026, the practical question is no longer which one is faster in a toy test, but which one earns its place in each stage of your pipeline.

Key takeaways

On our own 5-million-row benchmark, Polars beat pandas by 4.5x on CSV reading, 7x on group-by, and 8.7x on joins.
Small, simple operations show a much smaller gap: filter plus sort was only 1.5x faster in Polars.
Polars wins through multithreading, Arrow memory layout, and a lazy query optimiser, not through a single trick.
Pandas still wins at the machine learning boundary, where scikit-learn, plotting, and Excel tooling live.
The sensible 2026 answer for most teams is a hybrid: Polars for ETL, pandas where the ecosystem demands it.

How much faster is Polars, really?

Vendor benchmarks deserve scepticism, so we ran our own. The test machine was a modest 2-core virtual machine with 7 GB of RAM, running pandas 3.0.6 and Polars 2.1.0 on Python 3.12. The dataset was 5 million synthetic rows with 200 categories, a float column, and an integer column. Each timing below is the median of three runs.

Operation (5M rows)pandasPolarsSpeedupRead CSV1.93 s0.43 s4.5xGroup-by sum0.49 s0.07 s7.0xJoin (5M x 200 rows)0.78 s0.09 s8.7xFilter plus sort0.12 s0.08 s1.5x

The pattern matters more than any single number. Polars pulls away on the expensive, multi-step operations (reads, aggregations, joins) and barely leads on cheap ones. That matches independent third-party results: the Uvik Software 2026 DataFrame benchmark, run on a 2-core machine with 8 GB of RAM using pandas 3.0.6 and Polars 2.0.0, found Polars 5.7x faster across ten group-by queries on 10 million rows and 3.75x faster on five join queries. Under a 5.84 GiB memory cap at 50 million rows, pandas ran out of memory while loading, while Polars finished 9 of 10 queries.

What we could not verify: the Polars team's own PDS-H numbers, which claim a 94x gap on the full decision-support suite against pandas 2.2.3. That is a vendor claim on vendor-chosen hardware, and we treat it as an upper bound, not a promise. Your numbers will differ with data shape, column cardinality, and core count, so benchmark on your own data before committing.

What actually makes Polars faster?

Three design choices do most of the work. First, Polars is multithreaded by default: a group-by fans out across all cores without you asking. Pandas runs most operations on a single thread. Second, Polars stores data in Arrow format, so columns are contiguous, typed, and cheap to scan, while pandas still carries the overhead of its row index and per-column Python objects in its default dtypes. Third, Polars has a lazy API that optimises the whole query before running it: filters and column selections get pushed down to the file reader, so work that produces no output never happens.

import polars as pl

# Lazy: nothing runs until .collect(), and the optimiser
# prunes the filter and the unused columns first.
result = (
    pl.scan_csv("sales.csv")
    .filter(pl.col("qty") > 10)
    .group_by("category")
    .agg(pl.col("value").mean().alias("avg_value"))
    .collect()
)

On our machine, the lazy version of a filter plus group-by ran in 0.06 seconds against 0.07 seconds eager, a small win here, but on wide files with many unused columns the pushdown effect is where the large savings come from. The honest caveat: lazy mode only helps if you actually chain operations. A single eager call on a small frame gains almost nothing.

Where does pandas still win?

Speed is not the whole job. Pandas wins wherever the surrounding ecosystem matters more than raw throughput. Scikit-learn, statsmodels, matplotlib, and seaborn all expect pandas objects or NumPy arrays, and pandas reads and writes Excel files that Polars does not handle natively. For datasets under roughly 100,000 rows, the absolute time difference is milliseconds, and rewriting working code for milliseconds is a bad trade. Team familiarity counts too: almost every data tutorial, Stack Overflow answer, and legacy codebase speaks pandas, and some APIs, such as pivot tables with multiple margins, still have no direct Polars equivalent.

Pandas 3, released in 2026, also narrowed the gap for existing code. Copy-on-Write is now the default behaviour, which removes a whole class of silent bugs and wasted copies, and the optional Arrow-backed dtypes speed up string-heavy work. Pandas 3 requires Python 3.11 or newer and drops several long-deprecated features, so upgrading a legacy codebase still needs a test pass, but it is no longer the obvious laggard it was two years ago.

Polars has its own traps. Row-wise Python functions applied with map_elements run one row at a time in Python and throw away the entire speed advantage; almost every such case has a native vectorised expression, such as str.to_uppercase(), and you should reach for that first. Plotting is another gap: Polars has no built-in plotting, so visualisation still means converting to pandas or feeding a plotting library directly. And Polars' strict typing is a feature until it surprises you: a column that mixes integers and nulls behaves differently than the same column in pandas, so check dtypes explicitly after every read.

How do you switch without rewriting everything?

Do not rewrite your whole codebase in one go. Migrate the hot path first, usually the ETL stage, and keep pandas at the boundary where the ecosystem needs it. The two APIs are close enough that the translation is mostly mechanical:

# pandas
df.groupby("category")[["value", "qty"]].sum()

# Polars
df.group_by("category").agg(
    pl.col("value").sum(),
    pl.col("qty").sum(),
)

Conversion between the two is one call each way, but watch one sharp edge we hit ourselves while benchmarking: converting a pandas frame to Polars with pl.from_pandas() requires PyArrow installed unless every column is plain NumPy-backed. Pandas 3 defaults string columns to its new string dtype, which is not NumPy-backed, so install PyArrow or build the Polars frame directly from your arrays. At the machine learning boundary, convert once, not per step:

# Polars for the heavy lifting, pandas only where sklearn needs it
features = pl.scan_parquet("features.parquet").filter(...).collect()
model.fit(features.to_pandas(), labels)  # one conversion, at the edge

After migrating a step, verify it: compare row counts, check a few grouped totals against the old pandas output, and use Polars' equals check or pandas' testing helpers on a sample. Silent dtype changes, especially around nulls and timestamps, are the most common migration bug, not speed regressions.

So, should you switch in 2026?

Use this decision rule. If your frames regularly exceed a million rows, or the same ETL job runs on a schedule, switch that pipeline to Polars: the 4x to 9x gains we measured are real, and they compound into lower compute bills. If you are starting a greenfield data pipeline, start in Polars and convert at the edges. If your existing pandas code is stable, under 100,000 rows, and wrapped in notebooks and sklearn models, leave it alone: the migration cost buys you milliseconds. And if your team is learning data work for the first time, learn pandas first for the ecosystem, then add Polars as the performance tool. The two libraries are not a war. In 2026, the strongest Python data stack uses both, each where it is actually best.

Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.

Kimi K3.1 Pro: Open Weights, Near-Frontier Coding

Muhammad Saud — Tue, 27 Oct 2026 03:53:05 +0000

Kimi K3.1 Pro is Moonshot AI's most capable open-weight model, released on 23 October 2026. It scores 65.7% on Terminal-Bench 4.0, just 0.7 points behind Claude Opus 5.5, at roughly half the price, and its full weights are promised by 4 November. For developers, it is the first open model that is genuinely close to the frontier rather than a cheaper compromise.

Key takeaways

Kimi K3.1 Pro scores 65.7% on Terminal-Bench 4.0 in Moonshot's own numbers, 0.7 points behind Claude Opus 5.5 (66.4%) and ahead of GPT-6 Astra (58.2%). Artificial Analysis independently measured 67%.
It ranks 4th of 215 models on the Artificial Analysis Intelligence Index with a score of 55. The open-versus-closed gap has shrunk from 12 points to 3.
Pricing is $2 input and $10 output per million tokens, about half of Claude Opus 5.5. A Highspeed variant runs 2 to 3 times faster at the same price.
It is the same 2.8-trillion-parameter mixture-of-experts architecture as Kimi K3, with 104 billion parameters active per token and a 1-million-token context. The gain comes from post-training, not a bigger model.
The full weights are promised by 4 November and are not yet downloadable. Moonshot deliberately trimmed the model's cyber capability before release, cutting its internal exploit-bench score from 90.5% to 81.0%.

How close is Kimi K3.1 Pro to the frontier, really?

Close enough that the gap is now measured in fractions of a point on the benchmark developers care about most. Terminal-Bench 4.0 tests coding agents on real terminal tasks: installing dependencies, running tests, fixing broken builds. Moonshot reports 65.7% for Kimi K3.1 Pro, against 66.4% for Claude Opus 5.5 and 58.2% for GPT-6 Astra. Artificial Analysis, which runs the benchmark independently, measured 67%, which would be the highest Terminal-Bench score any model has posted. That independent confirmation matters, because vendor-reported launch numbers have a poor record this month: ARC Prize independently reran GPT-6 Astra at 62.7% on its own benchmark, against the 99.9% OpenAI had published from its own harness.

The picture holds up beyond one benchmark. On the Artificial Analysis Intelligence Index, which blends several evaluations into a single score, Kimi K3.1 Pro scores 55 and sits 4th out of 215 models, behind only Claude Opus 5.5 and Sonnet. On GDPval-AA, a board of real-world work tasks, it takes third place at 1,846 Elo, behind Claude Opus 5.5 at 1,866 and Muse Wonder Full at 1,857. Moonshot also claims first place on SWE Marathon, its long-horizon coding benchmark, at 51.7%, and a state-of-the-art 64.4 on OfficeQA Pro v2 for financial document work. Those last two are Moonshot's own claims and have not been independently rerun yet, so treat them as claims, not rankings.

The honest reading: near-frontier at coding and agentic work, with a wider gap on science-heavy tasks, where Moonshot's own charts show it trailing. If your workload is long coding sessions or terminal agents, this is the interesting case. For graduate-level science reasoning, the frontier models still lead by more.

What actually changed since Kimi K3?

Nothing about the architecture. Kimi K3.1 Pro is explicitly a post-trained K3: the same 2.8-trillion-parameter mixture-of-experts design, the same 104 billion active parameters per token, the same 1-million-token context, the same Kimi Delta Attention and Stable LatentMoE internals. What changed is the training after the base model: joint reinforcement learning across hundreds of downstream domains, on a rebuilt RL stack, with infrastructure optimisation at every layer from sandbox tooling to rollout scheduling.

The reported effects are large. Moonshot claims its hallucination rate on Kimi Hallu Bench Hard fell from 23.4% to 8.3%, and its long-horizon coding score on Kimi Code Bench 3.0 rose from 55.95% for K3 to 61.2%. Both are internal benchmarks, so trust the direction more than the exact numbers. The Terminal-Bench jump makes it concrete: K3 scored around 13% on the same benchmark family; K3.1 Pro scores 65.7%. That is not a tweak. It learned to finish multi-step terminal work instead of stalling halfway.

There is a lesson in this for anyone choosing models. The frontier is no longer decided only by who trains the biggest base model. Post-training, reinforcement learning on agentic tasks, and better tooling now move scores by tens of points. A lab that invests in the second half of the pipeline can close most of the gap without matching the first half's compute budget.

How much does it cost to run?

At $2 per million input tokens and $10 per million output tokens, Kimi K3.1 Pro costs about half of Claude Opus 5.5 while scoring within a point of it on coding-agent benchmarks. Artificial Analysis calls this a new benchmark for intelligence per dollar. The Highspeed variant, kimi-k3.1-pro-highspeed, promises 2 to 3 times the speed at the same price, which matters in agent loops where latency compounds across dozens of tool calls.

Price per token is only half the story, though. What decides your bill is tokens per finished task, and K3.1 Pro is an always-thinking model: it cannot turn reasoning off. Simple calls still burn reasoning tokens, so a short classification or extraction job may cost more here than on a smaller model even at the lower rate card. The right way to use it is for the hard, long-horizon work where its agent scores pay off, and to keep cheap models for the routine calls. That is the same split we recommended in our guide to cutting LLM API bills: route by task difficulty, not by brand loyalty.

Why did Moonshot make it weaker on purpose?

This is the most interesting part of the release, and the part most coverage skipped. As training progressed, cyber capability emerged: the unfiltered model solved 90.5% of a hard internal exploit benchmark. Before releasing the weights, Moonshot removed cyber-related data from mid-training and reinforcement learning, dropping that score to 81.0% at pass@3. The company told Reuters: "We aim not to release open weights that would extend the absolute frontier of cyber capability."

That is a new kind of release decision. Open-weight releases used to be a question of capability and licence; Moonshot treats them as a safety decision too: what the weights enable in the worst hands, not just what they do in the best benchmark. The trade is real, 9.5 points of exploit capability given up, and it sets a template other open-weight labs are likely to copy. The model you will download on 4 November is deliberately not the strongest version Moonshot built. For ordinary coding that costs nothing; for security research it is a genuine limitation, better known now than discovered later.

How do you try it today?

The weights are not downloadable yet, but the model is live now through Moonshot's API, the Kimi app, Kimi Code, and kimi.ai. The API is OpenAI-compatible, so existing code needs only the model name changed:

from openai import OpenAI

client = OpenAI(
    api_key="YOUR_MOONSHOT_API_KEY",
    base_url="https://api.moonshot.ai/v1",
)

response = client.chat.completions.create(
    model="kimi-k3.1-pro",
    messages=[
        {"role": "user", "content": "Refactor this function to use a generator and explain the trade-off."},
    ],
    temperature=0.2,
)
print(response.choices[0].message.content)

For agent loops, switch the model string to kimi-k3.1-pro-highspeed and keep everything else the same. Two practical notes: the 1-million-token context is real, but filling it is expensive, so pass only the files and history the agent actually needs; and if you handle customer data, check the Zero Data Retention option on the API, which Moonshot offers for API customers.

Before you migrate a production agent, run your own evaluation. Pick ten tasks your current model handles, run them through K3.1 Pro at temperature 0.2, and score pass or fail yourself. Vendor benchmarks measure the vendor's tasks. Your tasks are the only ones that decide your bill.

What is still unknown?

Three things to watch before you plan around this model. First, the weights are promised by 4 November, not shipped, and no licence has been published yet. Until the weights and licence actually land, "open-weight" is a promise, not a fact, and self-hosting plans should stay provisional. Second, a 2.8-trillion-parameter mixture-of-experts model with 104 billion active parameters per token will need serious multi-GPU hardware or aggressive quantisation to run locally, and nobody outside Moonshot knows the memory footprint yet. Third, the independent numbers we have so far cover coding and general intelligence; the financial-document and long-horizon claims are Moonshot's alone and await independent reruns.

Conclusion

Kimi K3.1 Pro is the first open-weight model genuinely close to the frontier on the work developers pay for: long coding sessions and terminal agents. It scores 65.7% on Terminal-Bench 4.0 against Claude Opus 5.5's 66.4%, at about half the price, and its independently measured 67% may be the best score on that benchmark so far. The caveats are real: the weights are not out yet, the licence is unpublished, and the model was deliberately weakened on cyber capability. But the direction is unmistakable. Open models stopped being a compromise this month. On 4 November, anyone with enough hardware will be able to run a model three points off the frontier. That changes the build-versus-buy calculation for every team running coding agents.

Originally published on LogicStack. Run the code examples right in your browser with our free interactive Python and SQL playgrounds — no setup, no sign-up.