DEV Community: Pradeep Kumar

The Best Open Source LLMs in August 2026: A Developer's Honest Guide

Pradeep Kumar — Mon, 24 Aug 2026 05:14:00 +0000

The Frontier Has No Wall

A year ago, self-hosting an LLM for serious development work usually meant accepting an obvious quality gap versus the closed frontier.

That gap is now much smaller.

The best open-weight models in August 2026 are competitive with proprietary systems on a growing number of reasoning, coding, and agentic workloads. On some evaluations, they are already ahead. The more useful question for a developer is no longer:

“Which model has the highest benchmark score?”

It is:

“Which model fits my workload, infrastructure, cost envelope, and legal requirements?”

That distinction matters because these models make very different trade-offs.

Kimi K3 pushes toward maximum capability but requires serious infrastructure. GLM-5.2 and GLM-5.3 emphasize coding and throughput. DeepSeek V4-Pro makes economics a first-class feature. MiniMax M3 is differentiated by native video and multimodality. Qwen3 remains attractive for Apache 2.0 deployments. Muse Glimmer targets local agents. Gemma 4 pushes useful multimodal capability down into smaller hardware tiers.

The market is also moving fast. Kimi K3 launched in July, while GLM-5.3, Qwen3.8-Max, and Qwen3.8-27B all arrived in August before this article was published. DeepSeek changed V4 pricing during the same month (official API docs).

So treat every benchmark and price in this article as a dated snapshot, not a permanent truth.

One terminology point matters from the start: open-weight is not automatically the same thing as open source. Public weights let you download, fine-tune, quantize, and deploy a model, but they do not necessarily expose the complete training data and reproducible training pipeline. For enterprise teams, that distinction can matter.

What to Optimize For Before You Pick

License

Start with the license, not the leaderboard.

MIT and Apache 2.0 are straightforward starting points for commercial software. Custom licenses need closer reading because they may add attribution, usage, distribution, or scale-related conditions.

Kimi K3, MiniMax M3, and Qwen3.8-Max should not be treated as equivalent to a plain MIT or Apache 2.0 model simply because their weights are downloadable.

Deployment scale

Hardware eliminates models surprisingly quickly.

A cluster-scale model and a 24GB local model may both be excellent, but they solve completely different problems. Before comparing benchmark scores, decide whether you have:

a large multi-GPU cluster,
a smaller GPU server,
a workstation,
or a single consumer GPU / Mac.

Context

A 1M-token context window can materially change the architecture of an application that processes large repositories, long documents, or extended agent trajectories.

But context length is only useful when the model, serving stack, and workload can use it efficiently. Do not confuse a headline context number with a practical production configuration.

Modality

If your application needs native video understanding, the shortlist changes immediately.

If you only need text plus coding, several models become more attractive. If you need image or video input inside an agent loop, multimodality becomes a first-order selection criterion.

Cost and throughput

At scale, token price and output speed compound.

A model that is slightly better but five times more expensive can be the wrong choice for a pipeline generating millions of tokens. Likewise, a cheaper model with much slower output can lose on wall-clock time when an agent makes many sequential calls.

Quick-Pick Decision Matrix

Primary constraint	Best fit	Why	Main trade-off
Highest capability with public weights	Kimi K3	Top-tier independent composite score; strong coding profile	Cluster-scale serving
Best API coding option	GLM-5.3	Current GLM flagship on API; strong coding focus	Weights not yet available
Fast coding with downloadable weights	GLM-5.2	Strong quality + high throughput + MIT	Large serving footprint
Cost-sensitive frontier inference	DeepSeek V4-Pro-0813	Strong capability at materially lower cost than many frontier APIs	Slower than GLM; pricing now tiered
Native video + multimodal	MiniMax M3	Text, image, and video in one model	Custom license
Apache 2.0 large-model workhorse	Qwen3 235B-A22B	Strong capability/cost/licensing balance	262K context; large memory footprint
Local agent on consumer hardware	Muse Glimmer	Strong MCP/tool-oriented profile	Weaker on some computer-use/terminal benchmarks
Local multimodal coding	Qwen3.8-27B	Apache 2.0 + image/video + 27B dense model	Newer ecosystem
Smaller Apache 2.0 entry point	Gemma 4	Broad size range and strong local ecosystem	Not frontier-class

Frontier Benchmark Snapshot

Artificial Analysis scores are useful here because they provide a consistent cross-model evaluation framework. They should still be read as one signal rather than a universal definition of intelligence.

For the current August snapshot, the important story is the cluster at the top:

Model	Intelligence Index	Practical read
Kimi K3	60	Top weights-available option
GLM-5.3	60	Tied with K3 on the API
Qwen3.8 2.4T-A95B	58	High-end Qwen frontier contender
DeepSeek V4-Pro-0813	53	Stronger than its preview release
Qwen3.8-27B	52	Exceptional size/performance position
GLM-5.2	51	Still highly competitive on coding
MiniMax M3	45	More differentiated by modality than raw text score

The point is not that 60 is “good” and 52 is “bad.” A 52-score model that fits on your hardware can be more valuable than a 60-score model that needs an accelerator cluster.

Kimi K3: The Benchmark Leader

Kimi K3 is the clearest choice when the question is simply:

“What is the strongest open-weight model I can actually download today?”

It is a 2.8T-parameter sparse MoE with 16 of 896 routed experts active per token, a 1M-token context window, and a public checkpoint released under a custom Kimi K3 license.

Its architecture combines Kimi Delta Attention (KDA), Stable LatentMoE, and Attention Residuals. KDA is particularly important for long-context efficiency because it combines linear and full-attention patterns rather than treating every layer identically.

Moonshot's current Kimi K3 repository reports 88.3 on Terminal-Bench 2.1, alongside strong results on FrontierSWE, DeepSWE, ProgramBench, and GPQA Diamond. Artificial Analysis also places K3 at the top of its open-weight Intelligence Index snapshot at 60.

That does not mean K3 wins every benchmark. It does not. The most important example is SWE-Bench Verified, where DeepSeek's published score is higher. Benchmark harness differences also matter, especially on coding-agent evaluations.

The bigger practical problem is infrastructure.

The official K3 checkpoint is around 1.56 TB. This is not a model you casually start on an eight-GPU box. Moonshot's serving guidance points toward large accelerator configurations, and production deployment is a cluster problem.

The other consideration is the license. K3 is not MIT and not Apache 2.0. It uses a custom license with additional terms at very large commercial scale.

Use Kimi K3 when: capability matters more than infrastructure simplicity, you can afford cluster-scale inference, and you want the strongest downloadable model in this comparison.

GLM-5.2 / GLM-5.3: The Production Coding Workhorse

The GLM family is interesting because it separates two different needs.

GLM-5.2 is the practical weights-available option. GLM-5.3 is the newer API flagship.

GLM-5.2 is a roughly 744B-parameter sparse MoE with about 40B active parameters per token and a 1M-token context. It is released under MIT and has built a strong reputation among coding-agent users.

Its Artificial Analysis Intelligence Index score is 51. Z.ai reports 62.1% on SWE-Bench Pro and an output speed around 168–200 tokens/second. The exact throughput number should be treated as measurement-dependent, but the high-throughput positioning is real and important.

GLM-5.3 changes the equation for teams that do not require public weights. Released in August, it is the current GLM API model and reaches an Artificial Analysis score of 60, tying Kimi K3 in the August snapshot.

For coding agents, that makes GLM-5.3 especially interesting: you get the newer model without having to operate the enormous checkpoint yourself.

The trade-off is obvious. GLM-5.3's weights were not yet public as of the article's publication date, so it should not be described as equivalent to K3 from a self-hosting perspective.

Use GLM-5.2 when: you want a strong coding model with public weights, MIT licensing, and high throughput.

Use GLM-5.3 when: API access is acceptable and you want the current GLM capability level without running the model yourself.

DeepSeek V4-Pro: The Cost-Sensitive Frontier Choice

DeepSeek V4-Pro is the model to watch when economics matter as much as benchmark position.

The original preview launched in April. The important August event is the V4-Pro-0813 GA release, which improved the model's Artificial Analysis score from the earlier preview level to 53.

The model is a 1.6T sparse MoE with about 49B active parameters, a 1M-token context window, and MIT licensing. It is text-only rather than multimodal.

The most important change for production planning is pricing.

The old flat $0.87/M output-token figure is no longer current. DeepSeek moved V4-Pro to peak/off-peak pricing on August 16. That means any article comparing DeepSeek's economics using the old flat rate is already stale.

The good news is that V4-Pro remains highly competitive on cost, especially for workloads that can take advantage of lower off-peak rates.

The downside is throughput. Artificial Analysis measures the GA release at roughly 77 tokens/second, below the GLM family.

For batch generation, classification, summarization, and structured generation, that can be a very attractive trade. For latency-sensitive multi-step agent loops, the economics may not be the only consideration.

Use DeepSeek V4-Pro when: cost per generated token is one of your primary constraints and your workloads can tolerate its lower throughput or take advantage of off-peak pricing.

MiniMax M3: The Multimodal Specialist

MiniMax M3 earns its place because it optimizes for something the other frontier models do not prioritize as strongly: native multimodal reasoning that includes video.

M3 is a sparse MoE model with roughly 428B total parameters and around 23B active parameters per token. Its documented context is 1M tokens, with 512K functioning as an important guaranteed/billing threshold.

It accepts text, image, and video input. That makes it particularly interesting for applications that need to reason across screenshots, documents, images, and video frames without building separate modality pipelines.

Artificial Analysis currently places M3 at 45 on its Intelligence Index, while its output speed is around 105 tokens/second.

The important caveat is licensing. MiniMax uses its Community License, not MIT or Apache 2.0. Commercial deployment therefore deserves a license review before you build around the model.

Use MiniMax M3 when: video and multimodal reasoning are core requirements rather than nice-to-have features.

Qwen3 235B-A22B: The Apache Workhorse

Qwen3 235B-A22B remains one of the most practical large open models for teams that care about licensing and self-hosting.

It is a 235B sparse MoE with 22B active parameters per token, a 262K context window, and an Apache 2.0 license.

Its appeal is not that it beats every frontier model. It is that the combination of capability, licensing, API economics, and deployment flexibility is unusually balanced.

With INT4 quantization, the model can fit into a roughly 60GB-class VRAM budget, making it much more approachable than cluster-scale frontier checkpoints.

The important distinction is that Qwen3 235B and Qwen3.8 are separate model families. Qwen3.8 is newer and much larger at the top end, but the 235B model remains attractive when you specifically want a mature Apache 2.0 workhorse.

Use Qwen3 235B when: Apache 2.0 matters, you want a large model you can actually deploy yourself, and you do not need the 1M-token context of the newest frontier systems.

Muse Glimmer: The Local Agent

Muse Glimmer answers a different question:

“What capable agent model can I actually run on hardware I own?”

Meta's Glimmer is a roughly 30B-class dense model with a 131K context window and Apache 2.0 licensing. It accepts text and image inputs and is designed specifically around agentic workflows and tool use.

Its strongest published results are on agent-oriented evaluations such as MCP Atlas and DeepSearch QA. Meta's own comparison tables also show areas where Qwen3.6-27B performs better, particularly on some computer-use and terminal evaluations.

That is actually useful because it tells you what Glimmer is for.

Glimmer's most interesting advantage is not being a universal benchmark winner. It is providing a capable local agent stack without requiring a giant cluster.

Quantized builds can fit into a roughly 24GB-class GPU envelope, depending on the quantization and serving configuration. The model also has broad local-runtime support.

Use Muse Glimmer when: privacy, local execution, and MCP/tool orchestration matter more than absolute benchmark leadership.

Gemma 4: The Entry Point

Gemma 4 is not a frontier contender against Kimi K3 or GLM-5.3, and it does not need to be.

Its strength is breadth.

Google's family includes E2B, E4B, 12B Unified, 26B A4B MoE, and 31B Dense variants, with Apache 2.0 licensing. The edge models offer 128K context, while the larger models support up to 256K.

The family is multimodal, and the smaller variants extend into audio as well. That makes Gemma unusually flexible across edge, workstation, and larger local deployments.

For developers getting started with self-hosted inference, the combination of a clean license, multiple model sizes, multimodal capability, and a large ecosystem is compelling.

One hardware caveat is important: consumer-GPU deployment depends heavily on quantization and context length. A 31B model that fits on a 24GB card in one 4-bit configuration is not the same thing as saying every serving configuration fits into 24GB.

Use Gemma 4 when: you are getting started with local inference, need a smaller model family, want Apache 2.0, or want a mature ecosystem across several hardware tiers.

Three August Releases You Should Not Ignore

Qwen3.8-Max

Qwen3.8-Max is the most important omitted model from the original draft because it moves the Qwen family into the same conversation as the largest frontier open-weight systems.

The open checkpoint is around 2.4T parameters with roughly 95B active, while the hosted version provides multimodal capabilities and a 1M-token context configuration.

The key caveat is that the hosted API and downloadable checkpoint are not identical product experiences. The public checkpoint has different modality/context characteristics and uses a custom Qwen3.8-Max license.

At this scale, the open weights are a datacenter project.

Why watch it: frontier capability without being tied to a closed model API, for organizations with the infrastructure to serve it.

Qwen3.8-27B

Qwen3.8-27B is much more relevant to ordinary developers.

It is a dense 27.78B model under Apache 2.0, with native multimodal input and a 262K context window, extendable with long-context techniques.

Quantized builds can fit into a roughly 17GB-class memory envelope, putting it firmly into single-GPU territory.

It is especially interesting because it fills the exact hole created by the original article's nonexistent “Gemma 4 27B” recommendation.

Why watch it: Apache 2.0, multimodality, and serious capability in a model that is still realistic to run locally.

GLM-5.3

GLM-5.3 deserves mention even though its weights were not yet public at publication.

Its Artificial Analysis score of 60 puts it alongside Kimi K3 in the August snapshot, and its coding-focused positioning makes it one of the most important API models to test if your workload is agentic software engineering.

Why watch it: it changes the conclusion for teams that value API performance more than self-hosting.

Open-Weight vs. Open-Source

This distinction is easy to ignore until it becomes important.

An open-weight model gives you the model parameters. That can be enough for fine-tuning, local inference, quantization, private deployment, and serious product development.

But it does not necessarily give you:

the complete training dataset,
the full preprocessing pipeline,
the exact training recipe,
or the ability to reproduce the model from scratch.

That matters differently depending on the organization.

For a startup shipping an internal coding assistant, it may be mostly academic.

For a regulated enterprise, a research lab, or a company planning to maintain a model for years, it can affect reproducibility, legal review, procurement, and long-term risk.

Also read the actual license. “Open weights” does not imply MIT or Apache 2.0.

Bottom Line

The open-weight ecosystem has moved from “good enough if you accept compromises” to “choose according to your constraints.”

That changes how you should evaluate models.

Need the strongest downloadable frontier model? Start with Kimi K3.

Need the strongest current coding API in this group? Test GLM-5.3.

Need high-throughput coding with public weights? GLM-5.2 remains compelling.

Need cost-sensitive frontier inference? DeepSeek V4-Pro-0813 remains one of the strongest economic choices, but use current tiered pricing in your model.

Need native video understanding? MiniMax M3 is the specialist.

Need Apache 2.0 and a large self-hosted model? Qwen3 235B-A22B remains highly attractive.

Need a local agent on consumer hardware? Evaluate Muse Glimmer and Qwen3.8-27B against your exact workload.

Need an approachable local family with a clean license? Gemma 4 is still one of the best starting points.

The leaderboard will change again next week.

Your hardware budget, latency target, license requirements, and workload will not.

Sources

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Caching Strategies Every Backend Developer Must Know

Pradeep Kumar — Fri, 21 Aug 2026 07:10:14 +0000

There is a reason Phil Karlton's 1990s quip — that the two hardest problems in computer science are cache invalidation and naming things — still surfaces in every engineering conversation. Caching is simple in theory and genuinely difficult in practice. Get it right and your API responds in single-digit milliseconds under load. Get it wrong and you ship corrupted data, stale reads, or a cache stampede that takes down your database at peak traffic.

This guide cuts through the noise. You will learn the five core caching patterns, when to reach for each one, how eviction policies work, how to approach the invalidation problem, and what not to cache. Every section includes working code you can apply today.

Why Caching Matters: The Numbers

The performance gap between a database read and a cache read is not incremental — it is categorical. AWS benchmarks on RDS for MySQL 8.0 paired with ElastiCache show average read latency dropping from 14 ms to 0.51 ms — a 27x improvement — with RDS CPU utilization falling from 78% to 32% and throughput increasing 34% simultaneously. A single Redis instance, per Redis's own benchmark documentation, can sustain over 1.8 million GET operations per second with pipelining enabled. The database cannot compete with RAM on raw throughput.

The impact at scale is equally stark. Facebook's Memcached deployment, documented at USENIX NSDI '13, handles billions of requests per second across trillions of cached items — an architecture where the cache, not the database, is the primary read path for the world's largest social network. AWS benchmarks further show caching RDS workloads can reduce infrastructure costs by up to 55%, because fewer read replicas are needed.

The relationship between hit rate and database load is direct: at an 80% cache hit rate, 80% of your database reads are eliminated. The chart below shows how this scales:

The point is not that caching always helps — the sections below explain when it actively hurts. It is that ignoring it, at any meaningful scale, almost always leaves significant performance and cost on the table.

Core Terminology

Get these four concepts locked in before the patterns.

Cache hit / cache miss — A hit means the requested key exists in cache and is returned directly. A miss means it does not; the application must fetch from the source of truth (usually the database).

TTL (Time to Live) — How long a cached entry lives before it expires automatically. Too short and you get high miss rates; too long and you risk serving stale data. The right TTL is almost always workload-specific and discovered through measurement, not intuition.

Eviction policy — What happens when the cache runs out of memory. The cache must remove something to make room. Your policy choice determines what gets removed (covered in its own section below).

Cache stampede (thundering herd) — What happens when many requests simultaneously miss a key that just expired, all racing to repopulate it from the database at once. This is one of the most common ways caching makes an outage worse rather than better.

Quick Decision Framework

Before the deep dives, here is the pattern-selection map. Jump to whichever section applies:

Is data read frequently and written infrequently?
  └─ Yes → Cache-Aside (default) or Read-Through

Is data consistency critical on every write?
  └─ Yes → Write-Through
  └─ No, and writes are very frequent → Write-Behind (with durable queue)

Are hot keys causing stampedes on expiry?
  └─ Yes → Refresh-Ahead

Can you tolerate eventual consistency (minutes)?
  └─ Yes → TTL-based invalidation is sufficient

Must every read see the latest write immediately?
  └─ Yes → Do not cache this data

The Five Core Caching Patterns

1. Cache-Aside (Lazy Loading)

The most common pattern. The application owns all cache interaction logic. On a read, it checks the cache first. On a miss, it queries the database, stores the result in cache, and returns it to the caller.

Read request
     │
     ▼
  Cache hit? ──Yes──▶ Return cached value
     │
     No
     │
     ▼
  Query database
     │
     ▼
  Store in cache (with TTL)
     │
     ▼
  Return value

Node.js / Redis example:

import { createClient } from 'redis';

const redis = createClient({ url: process.env.REDIS_URL });
await redis.connect();

async function getUserById(userId) {
  const cacheKey = `user:${userId}`;

  // 1. Check cache first
  const cached = await redis.get(cacheKey);
  if (cached) {
    return JSON.parse(cached); // cache hit
  }

  // 2. Miss — fetch from database
  const user = await db.query('SELECT * FROM users WHERE id = $1', [userId]);
  if (!user) return null;

  // 3. Populate cache with a 10-minute TTL
  await redis.setEx(cacheKey, 600, JSON.stringify(user));

  return user;
}

Python equivalent:

import json
import os
import redis

r = redis.Redis.from_url(os.getenv("REDIS_URL"))

def get_user_by_id(user_id: int):
    cache_key = f"user:{user_id}"

    cached = r.get(cache_key)
    if cached:
        return json.loads(cached)

    user = db.execute("SELECT * FROM users WHERE id = %s", (user_id,)).fetchone()
    if not user:
        return None

    r.setex(cache_key, 600, json.dumps(dict(user)))
    return user

When to use it: Read-heavy workloads where data changes infrequently. The cache only populates on demand, so you never pre-warm stale entries. It is resilient to cache failures — if Redis goes down, the application still works (slower, but correctly).

Trade-off: The first request after a cache miss (or expiry) always hits the database. Under high concurrency this becomes the stampede problem. Add a distributed lock or use the refresh-ahead pattern to mitigate.

2. Read-Through

Similar to cache-aside, but the cache itself fetches from the database on a miss, rather than the application. The application only ever talks to the cache.

// The cache client is configured with a loader function
const cache = new CacheClient({
  loader: async (key) => {
    const userId = key.split(':')[1];
    return db.query('SELECT * FROM users WHERE id = $1', [userId]);
  },
  ttl: 600,
});

// Application code is now clean:
async function getUserById(userId) {
  return cache.get(`user:${userId}`); // miss handled internally
}

When to use it: When you want to keep cache interaction logic in one place and out of business code. Works well with managed caching services that support loader callbacks (e.g., Momento, AWS ElastiCache with data tiering).

Trade-off: Slightly less flexible than cache-aside — you cannot customize the miss flow per call site. The first-request miss penalty is identical.

3. Write-Through

Every write goes to the cache and the database simultaneously, synchronously, before returning to the caller.

async function updateUser(userId, data) {
  const cacheKey = `user:${userId}`;

  // Write to database first (or transactionally with cache)
  const updated = await db.query(
    'UPDATE users SET name = $1, email = $2 WHERE id = $3 RETURNING *',
    [data.name, data.email, userId]
  );

  // Immediately update cache
  await redis.setEx(cacheKey, 600, JSON.stringify(updated));

  return updated;
}

When to use it: When data consistency is critical and writes are not extremely frequent. The cache is always warm and always fresh. Read performance is excellent because entries exist immediately after a write — no cold-start miss.

Trade-off: Every write requires two round-trips (database + cache) before returning. For write-heavy workloads this compounds at scale. You also cache data that might never be read, wasting memory.

4. Write-Behind (Write-Back)

The application writes to the cache only, and a background process flushes dirty entries to the database asynchronously. The write returns immediately after the cache is updated.

// Simplified write-behind with a queue
async function updateUserAsync(userId, data) {
  const cacheKey = `user:${userId}`;

  // Write to cache immediately — caller gets fast response
  await redis.setEx(cacheKey, 600, JSON.stringify({ ...data, id: userId }));

  // Queue the database write for background processing
  await redis.lPush('db:write-queue', JSON.stringify({
    table: 'users',
    id: userId,
    data,
    timestamp: Date.now(),
  }));
}

// Background worker (runs separately)
async function processWriteQueue() {
  while (true) {
    const item = await redis.brPop('db:write-queue', 0); // blocking pop
    const { table, id, data } = JSON.parse(item.element);
    await db.query(`UPDATE ${table} SET name=$1, email=$2 WHERE id=$3`, [data.name, data.email, id]);
  }
}

When to use it: High write-throughput scenarios where sub-millisecond write latency is required and you can tolerate a small window of data loss. Session stores, analytics event pipelines, activity feeds.

Trade-off: Risk of data loss. If the cache node fails between the write and the flush, the database never gets updated. Always pair write-behind with Redis persistence (AOF or RDB) and durable queue infrastructure (e.g., Kafka, SQS) in production.

5. Refresh-Ahead (Proactive Refresh)

The cache predicts which entries are about to expire and refreshes them in the background before they go cold — so the caller never experiences a miss.

const REFRESH_THRESHOLD = 0.2; // refresh when 20% of TTL remains

async function getUserWithRefreshAhead(userId) {
  const cacheKey = `user:${userId}`;

  const [cached, ttlRemaining] = await Promise.all([
    redis.get(cacheKey),
    redis.ttl(cacheKey),
  ]);

  if (cached) {
    const fullTtl = 600; // your configured TTL
    const shouldRefresh = ttlRemaining < fullTtl * REFRESH_THRESHOLD;

    if (shouldRefresh) {
      // Trigger background refresh without blocking the caller
      refreshUserCache(userId).catch(console.error);
    }

    return JSON.parse(cached);
  }

  // Full miss — blocking refresh
  return refreshUserCache(userId);
}

async function refreshUserCache(userId) {
  const user = await db.query('SELECT * FROM users WHERE id = $1', [userId]);
  await redis.setEx(`user:${userId}`, 600, JSON.stringify(user));
  return user;
}

When to use it: Frequently accessed, expensive-to-compute data where cache miss latency is unacceptable. Home page aggregations, pricing data, leaderboards. Eliminates the stampede problem entirely for hot keys.

Trade-off: You may refresh data that was never requested after its refresh — wasted compute. Requires good observability to tune the threshold correctly.

Eviction Policies: What Gets Dropped When Memory Is Full

Policy	Full Name	How It Works	Best For
LRU	Least Recently Used	Evicts the entry accessed least recently	General-purpose; good default
LFU	Least Frequently Used	Evicts the entry accessed least often overall	Workloads with stable hot-key distribution
FIFO	First In First Out	Evicts oldest entry regardless of access	Simple queues; less common
TTL	Time-based	Evicts based on expiry time	All workloads; complements LRU/LFU
Random	—	Evicts a random entry	Rarely the right choice in production

Redis default is noeviction — it throws an error when memory is full rather than silently dropping data. Switch to allkeys-lru for most production workloads:

# In redis.conf or via CONFIG SET
maxmemory 2gb
maxmemory-policy allkeys-lru

For workloads with clear hot-key skew (e.g., celebrity social media posts), allkeys-lfu tends to outperform LRU because frequency is a better predictor of future access than recency. USENIX NSDI '18 research shows that hit rate increases logarithmically as a function of cache capacity — meaning the eviction policy becomes more important than raw memory size as you scale.

Cache Invalidation: The Hard Part

Phil Karlton was right. Cache invalidation is hard because the cache and the source of truth can diverge, and the application must detect and resolve that divergence without serving incorrect data or generating excessive database load.

The three practical strategies:

TTL-based expiry

Let entries expire automatically after a configured duration. Simple, predictable, eventually consistent.

await redis.setEx('product:123:price', 300, JSON.stringify(price)); // 5 min

Risk: Stale reads for up to the full TTL window. For pricing data, a 5-minute stale price may be acceptable. For account balances, it is not.

Event-driven invalidation

Invalidate (delete) a cache entry the moment the underlying data changes, rather than waiting for TTL expiry.

// In your update endpoint
async function updateProduct(productId, data) {
  await db.query('UPDATE products SET price = $1 WHERE id = $2', [data.price, productId]);

  // Immediately purge — next read will miss and repopulate
  await redis.del(`product:${productId}:price`);

  // Or pattern-delete all keys related to this product
  const keys = await redis.keys(`product:${productId}:*`);
  if (keys.length > 0) await redis.del(keys);
}

Risk: Delete-before-repopulate creates a brief window where all in-flight requests hit the database simultaneously. For high-traffic keys, use cache-aside with a distributed lock, or write-through instead.

Versioned keys

Append a version number or hash to the cache key. Updates increment the version, making the old key unreachable without requiring an explicit delete.

// Store the current version
await redis.set('product:123:version', 'v5');

// Cache key includes version
const version = await redis.get('product:123:version');
const cacheKey = `product:123:price:${version}`;

// On update: bump version (old key becomes unreachable)
await redis.incr('product:123:version');

This is the safest approach for multi-region or distributed caches where explicit delete propagation is unreliable. Old entries age out via TTL naturally.

Redis vs. Memcached: Which One?

This question comes up in every architecture discussion. The short answer for new projects: use Redis.

	Redis	Memcached
Data types	Strings, hashes, lists, sets, sorted sets, streams	Strings only
Persistence	RDB snapshots + AOF log	None
Replication	Built-in primary/replica	External only
Pub/Sub	Yes	No
Lua scripting	Yes	No
Max value size	512 MB	1 MB
Multithreading	Multi-threaded I/O (Redis 6+)	Multi-threaded
Best for	Nearly all production use cases	Pure key/value, ultra-simple workloads

Memcached's one genuine advantage is slightly lower memory overhead for pure string storage at extreme scale. Unless you are operating at hundreds of terabytes of cache data, this rarely matters. Redis's richer data types, persistence options, and Lua scripting make it the clear default.

The benchmark comparison below is from real AWS infrastructure measurements — not synthetic lab tests:

What NOT to Cache

Caching is not universally beneficial. These are situations where it actively causes harm:

Highly volatile data. If a value changes more often than your TTL, you will serve stale data constantly while paying cache infrastructure overhead for essentially no hits. Financial ticker prices and live sports scores are common examples where a direct database or streaming feed is the right answer.

Unbounded key spaces. If your cache key includes a user-supplied parameter with high cardinality (e.g., search query strings), you will generate millions of cache keys that each get accessed once. This consumes memory, produces near-zero hit rates, and degrades eviction efficiency. Apply caching only to keys with stable, bounded spaces.

Data that is cheaper to recompute than to cache. Simple arithmetic or deterministic in-memory transformations have effectively zero cost. Routing them through a network call to Redis adds latency.

Data requiring real-time consistency. If the application contract requires that every read reflects every write with zero lag — shopping cart totals, bank balances, inventory counts during flash sales — a cache introduces a consistency lag that your users will experience as bugs.

Personally identifiable information you do not need to cache. Every cached PII record is an additional attack surface. If the TTL benefits do not justify the exposure, leave it out of the cache.

Preventing the Cache Stampede

When a hot key expires under heavy traffic, every concurrent request races to repopulate it. This is the stampede. Three proven defenses:

Probabilistic early expiry — Each process individually decides whether to refresh before expiry, with probability increasing as expiry approaches. Simple to implement with no coordination overhead.

function shouldEarlyRefresh(ttlRemaining, fullTtl, beta = 1) {
  // Higher beta = more aggressive early refresh
  return ttlRemaining <= fullTtl * beta * Math.random();
}

Distributed lock — Only one process is permitted to repopulate the key at a time. All others wait or return a stale value.

const lock = await redis.set(
  `lock:user:${userId}`,
  '1',
  { NX: true, EX: 10 } // only set if not exists, expire in 10s
);

if (lock) {
  // We won the lock — repopulate
  const user = await db.query('SELECT * FROM users WHERE id = $1', [userId]);
  await redis.setEx(`user:${userId}`, 600, JSON.stringify(user));
  await redis.del(`lock:user:${userId}`);
  return user;
} else {
  // Lost the lock — wait briefly and retry, or return a slightly stale value
  await sleep(50);
  return getUserById(userId);
}

Background refresh with stale-while-revalidate — Serve the stale value immediately while a single background process refreshes asynchronously. This is the refresh-ahead pattern applied reactively.

Monitoring Cache Health

A cache you cannot observe is a cache you cannot trust. Track these four metrics in production:

Metric	What It Tells You	Action Threshold
Hit rate	Fraction of reads served from cache	Investigate below 80%
Miss rate	Fraction of reads falling through to DB	Spikes indicate TTL or eviction issues
Eviction rate	Keys dropped due to memory pressure	Any eviction = consider memory increase
Latency (p99)	99th percentile cache round-trip	Above 1 ms warrants investigation

// Instrument your cache-aside reads
async function getUserById(userId) {
  const start = Date.now();
  const cacheKey = `user:${userId}`;
  const cached = await redis.get(cacheKey);

  if (cached) {
    metrics.increment('cache.hit', { key_type: 'user' });
    metrics.histogram('cache.latency_ms', Date.now() - start);
    return JSON.parse(cached);
  }

  metrics.increment('cache.miss', { key_type: 'user' });
  const user = await db.query('SELECT * FROM users WHERE id = $1', [userId]);
  await redis.setEx(cacheKey, 600, JSON.stringify(user));
  return user;
}

Summary

Caching is one of the highest-leverage tools in backend engineering. The five core patterns map cleanly to distinct use cases:

Cache-Aside — the safe, flexible default for read-heavy workloads
Read-Through — same as cache-aside but encapsulated inside the cache layer
Write-Through — strong consistency on writes, acceptable write latency
Write-Behind — maximum write throughput, requires durable queuing
Refresh-Ahead — eliminates miss latency for predictable hot keys

Master cache invalidation — TTL, event-driven delete, and versioned keys — before worrying about exotic patterns. Most real-world caching problems come down to stale data or stampedes, and both have well-understood solutions.

Start with Redis, instrument hit rate from day one, and set allkeys-lru as your eviction policy. Everything else you can tune iteratively as traffic grows.

FAQ

What is a good cache hit rate? In most production workloads, a hit rate above 80% is the practical target. At 80% hit rate, roughly 80% of your database reads are eliminated. Below 70%, TTL tuning and key design should be revisited. Note that hit rate increases logarithmically with cache capacity — so the eviction policy and key design matter more than raw memory at scale.

Should I use Redis or a CDN for caching? CDNs (Cloudflare, Fastly) cache at the network edge — ideal for static assets and API responses that are identical across users. Redis caches at the application layer — ideal for user-specific or frequently updated data. They are complementary, not alternatives.

How do I invalidate cache entries across multiple services in a microservices architecture? Use an event bus (Kafka, RabbitMQ, or Redis Pub/Sub). The service that owns the data publishes an invalidation event when it writes; all services that cache that data subscribe and delete their local entries. This is the event-driven invalidation pattern applied at scale.

Can caching make my application slower? Yes, in two scenarios: cache miss rate is so high that you are adding a network round-trip to every database call (net negative), or cache lookup latency exceeds the database query you were trying to skip (rare, but happens with misconfigured Redis in high-latency networks). Monitor before deploying caching and measure after.

What is a cold start / cold cache problem? When your cache is empty — after a deployment, a restart, or provisioning a new node — all requests miss until entries are populated. For critical keys, pre-warm the cache at startup by reading from the database and populating keys before traffic is admitted.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Inside Claude's Brain: What Anthropic's Published System Prompts Reveal About the Future of AI Transparency

Pradeep Kumar — Wed, 19 Aug 2026 05:34:23 +0000

The Page That Changed Everything

In August 2024, a quiet page appeared on Anthropic's documentation site. No press release. No keynote. Just a URL — platform.claude.com/docs/en/release-notes/system-prompts — and a simple promise: from now on, Anthropic would publish the system prompts that govern how Claude behaves on claude.ai and its mobile apps.

The initial entries were dated July 12, 2024, covering Claude 3 Haiku, Claude 3 Opus, and Claude 3.5 Sonnet.

Two years later, that page has become one of the most discussed artifacts in the AI industry. It regularly trends on Hacker News with hundreds of upvotes. Researchers diff it like source code. Developers treat it as a masterclass in prompt engineering. And it stands as a singular act of voluntary transparency in an industry that treats system prompts like nuclear launch codes.

This is the story of what's inside those prompts — and what they tell us about where AI is heading.

What Is a System Prompt, and Why Should You Care?

Before we dive in, let's establish what we're actually talking about.

Every time you open Claude (or ChatGPT, or Gemini), the AI doesn't start with a blank slate. Before you type a single word, a hidden set of instructions — the system prompt — has already been loaded.

Think of it as an operating manual handed to a new employee on their first day: it defines who they are, how they should behave, what they're allowed to do, and what's strictly off-limits.

These prompts shape everything:

Identity: "You are Claude, created by Anthropic."
Temporal grounding: The current date, so the model knows it's August 2026, not stuck in its training data.
Knowledge boundaries: What the model knows, and when to admit it doesn't.
Safety rails: Hard lines around harmful content, illegal activities, and sensitive topics.
Personality: Whether the AI sounds like a warm friend or a crisp professional.
Tool usage: When and how to use search, code execution, file editing, and other capabilities.

The system prompt is, in every meaningful sense, the AI's constitution. And until Anthropic broke ranks, no major AI company had ever published theirs voluntarily.

The Transparency Divide: Published vs. Leaked

Here's where it gets interesting. Let's look at how the three major AI labs handle this:

	Anthropic (Claude)	OpenAI (ChatGPT)	Google (Gemini)
Strategy	Voluntary publication	Secrecy	Secrecy
How prompts surface	Official documentation	Community leaks & prompt injection	Community leaks & prompt injection
Update tracking	Versioned release notes	Reverse-engineered by researchers	Reverse-engineered by researchers
Developer guidance	High — prompt is the documentation	Moderate	High for developer tools, low for consumer

OpenAI and Google treat their system prompts as proprietary trade secrets. The only way the public sees them is through adversarial prompt injection — tricking the AI into repeating its hidden instructions.

Alternatively, they surface through massive community-maintained GitHub repositories that have accumulated tens of thousands of stars.

Anthropic simply... publishes them. On a public webpage. With a changelog.

As Simon Willison, the prolific developer and AI commentator, put it: these published prompts are "the secret missing manual" for the model. They reveal not just what Claude does, but why it does it — which is information that developers working with ChatGPT or Gemini can only guess at.

Anatomy of a System Prompt: What's Actually in There

So what does Claude's operating manual actually say?

Based on Anthropic's published prompts (and community analysis of the more recent versions), the system prompt has grown from a relatively concise document into a comprehensive technical specification — often exceeding 100,000 characters in recent versions. It's organized into dozens of sections, typically 70 or more.

Here's a breakdown of the major categories:

1. Identity & Grounding

The prompt opens by establishing who Claude is and when "now" is:

The assistant is Claude, made by Anthropic.
The current date is [dynamically injected].

This isn't just a formality. The date injection is critical for temporal grounding — it lets Claude distinguish between events in its training data and events that happened after its knowledge cutoff.

Without it, the model would confidently discuss the 2024 election as a future event.

2. Knowledge Boundaries

The prompt explicitly instructs Claude on how to handle the gap between what it knows and what's happened since:

Use internal knowledge for events within the training window.
Use search tools for anything recent or uncertain.
Never fabricate information to fill gaps — acknowledge limitations instead.

This is the "honest" in Anthropic's helpful, harmless, and honest framework. Rather than hallucinating a plausible-sounding answer, Claude is instructed to say "I'm not sure" — and, where possible, search for the answer.

3. Safety & Ethics (The Hard Boundaries)

This is the longest and most nuanced section. It covers:

Absolute refusals: Weapons instructions, malware generation, CSAM, and other categories with zero exceptions.
Contextual judgment: For ambiguous requests, Claude is instructed to assume positive, legal intent rather than reflexively refusing. This is a deliberate design choice to avoid the "sorry, I can't help with that" problem that plagued earlier AI models.
Child safety protocols: If a conversation is flagged, the model applies "extreme caution" to all subsequent requests in that thread.
Mental health: Specific protocols for users expressing suicidal ideation or self-harm, including when to provide crisis resources.
Evenhandedness: On politically controversial topics, Claude is instructed to present multiple perspectives without taking sides.

4. Formatting & Communication Style

This is where the prompt gets surprisingly opinionated:

No filler phrases: Claude is explicitly told to avoid starting responses with "Certainly!", "Of course!", "Absolutely!" or similar sycophantic openers.
Markdown by default: Code should be formatted with syntax highlighting; complex answers should use headers and lists.
Conciseness: Lead with the direct answer, then elaborate. Don't bury the lede.
Match the user's register: If someone writes casually, respond casually. If they write formally, match their tone.

5. Tool Definitions & Capabilities

In the consumer-facing claude.ai product, the system prompt contains detailed schemas for every tool Claude can use:

Web search: When to search, how to cite sources, how to handle copyright.
Code execution: Sandboxed environments for running Python, generating visualizations.
File handling: Reading uploaded documents, processing images.
Specialized agents: Browsing assistants, spreadsheet processors, presentation builders.

Community researchers note that tool definitions often consume the majority of the system prompt's token budget — sometimes more than all the personality, safety, and formatting instructions combined.

6. The "Meta" Instructions

Perhaps the most fascinating section: instructions about the instructions themselves.

Claude is told how to handle attempts to extract or manipulate its system prompt. It's instructed to be transparent about having a system prompt (it won't deny it exists), but not to reproduce it verbatim.

It's also told to prioritize its core safety instructions even if a user's prompt contradicts them — establishing a clear hierarchy of authority.

The Evolution: How the Prompts Have Changed Over Time

One of the most valuable aspects of Anthropic's transparency is that it creates a public changelog for AI behavior. Researchers like Simon Willison have exploited this brilliantly, using tools like git diff to track exactly how Claude's personality and capabilities shift between versions.

Here are some of the most notable evolutionary trends:

From Warm to Direct (2024 → 2025)

Early Claude prompts emphasized warmth and empathy. The model was positioned as a friendly brainstorming partner. By mid-2025, the tone had shifted: Claude became more of a direct, opinionated expert — someone who gives you the answer rather than asking how you feel about it.

This wasn't accidental. User feedback consistently showed that people preferred Claude when it was confident and direct, not when it hedged everything with qualifiers.

From Verbose to Lean (2025 → 2026)

A surprising trend: Anthropic has been reducing the length of its safety instructions in recent prompts. Not because they care less about safety, but because frontier models have internalized these behaviors through training.

As Anthropic's own research has noted, the shift is from "prompt engineering" to "context engineering" — recognizing that a model with deep constitutional training needs fewer explicit rules in its prompt and more contextual framing.

The Tool Explosion (2024 → 2026)

The most dramatic growth has been in tool definitions. The original prompts had minimal tool integration. By 2026, tool schemas for search, code execution, computer use, and specialized agents dominate the prompt. Claude has evolved from a text-in/text-out chatbot into an agentic system that can browse the web, write and execute code, and interact with external services.

The Fable/Mythos Architecture (June 2026)

The release of Claude Fable 5 and Claude Mythos 5 on June 9, 2026, introduced a new dimension to system prompt design: tiered safety classifiers.

Fable 5 and Mythos 5 are the same underlying model — the most capable Anthropic has ever released — but with different safety postures. Fable 5 includes safety classifiers that monitor for sensitive domains (cybersecurity, biology, chemistry); if triggered, it falls back to Claude Opus 4.8 for that response.

Mythos 5 lifts those classifiers and is restricted to vetted partners through Anthropic's Project Glasswing initiative.

This architecture means the system prompt alone no longer tells the full story — the prompt now works in concert with external classifier layers that can override the model's output entirely.

Just days after launch, both models were temporarily suspended globally (June 12–July 1, 2026) due to a U.S. government export control directive.

The incident was a stress test for Anthropic's transparency commitment: they documented the suspension and restoration publicly, rather than quietly limiting access. For a company that publishes its system prompts, this consistency mattered.

The subsequent release of Claude Opus 5 on July 24, 2026 — designed to approach Fable 5's intelligence at a lower price point — further expanded the system prompt landscape across an increasingly diverse model lineup.

Child Safety Escalation

Each prompt revision has strengthened child safety protocols. Recent versions implement a "ratcheting" mechanism: once a conversation triggers a safety flag, the model's caution level increases for all subsequent messages in that thread, not just the flagged one.

Why This Matters: Five Lessons From the System Prompts

1. The AI's "Personality" Is Engineered, Not Emergent

When Claude gives you a thoughtful, nuanced answer, it's not because it spontaneously developed good judgment. It's because someone at Anthropic wrote instructions telling it to "present multiple perspectives on controversial topics" and "lead with the direct answer."

The system prompt reveals that what feels like intelligence is, in part, careful UX design. This isn't a criticism — it's a feature. But it means we should evaluate AI responses knowing that they're shaped by editorial decisions, not just raw capability.

2. Safety Is Not a Binary Switch

The prompts reveal a sophisticated, context-dependent approach to safety. Claude doesn't have a simple "allowed/not allowed" list. It has:

Absolute prohibitions (weapons, CSAM) — no context overrides these.
Contextual judgment — assume positive intent for ambiguous requests.
Escalation protocols — increase caution within flagged conversations.
Domain-specific rules — different handling for medical questions vs. legal questions vs. creative writing.

This multi-layered approach is far more nuanced than most people realize, and it's a template that other AI companies will likely adopt.

3. The Prompt Is a Product Specification

Reading Claude's system prompt is like reading the product requirements document for a software application. It defines features, edge cases, error handling, and user experience guidelines. This reframes how we should think about AI development: the system prompt isn't an afterthought bolted onto the model — it's a core part of the product design.

4. Transparency Creates Accountability

Once you publish your AI's instructions, you can be held to them. If Claude's system prompt says "present balanced perspectives on political topics," users can call out violations. If it says "don't start responses with 'Certainly!'" and then it does, that's a measurable bug.

This is a radical departure from the black-box model, where AI companies can change behavior without anyone noticing. Anthropic has, in effect, given users a contract they can audit.

5. The Industry Will Follow

The EU AI Act and other global regulations are pushing toward mandatory transparency for high-risk AI systems. Open-source models like Llama and Mistral already publish their entire architectures. As public demand for AI accountability grows, the "publish your system prompt" approach will likely become the industry norm by 2027–2028.

Anthropic got there first, and that matters.

What the Prompts Don't Tell You

Transparency has limits, and it's worth acknowledging what Anthropic's published prompts leave out:

Tool definitions are omitted: The published prompts strip out the detailed JSON schemas for tools like search and code execution. These are arguably the most technically interesting parts.
API prompts are not included: The published prompts apply only to claude.ai and the mobile apps. If you use the Anthropic API, you're responsible for your own system prompt — and the API does not inject these instructions automatically.
Claude Code has different prompts: The CLI-based coding tool has its own, separately maintained (and not officially published) system prompt that's dynamically assembled from hundreds of conditional strings.
Safety classifiers are separate: With the Fable 5/Mythos 5 architecture, external safety classifiers can override model output independently of the system prompt. These classifier rules and thresholds are not published.
Training alignment isn't visible: The system prompt is only one layer of Claude's behavior. The deeper layer — reinforcement learning from human feedback (RLHF) and constitutional AI training — is not captured in the prompt and isn't publicly visible.

These omissions are worth noting, but they don't diminish the significance of what is published. The system prompt is the most legible, most auditable layer of AI behavior, and making it public is a meaningful act of trust.

For Developers: What You Can Learn From These Prompts

If you're building with the Claude API (or any LLM), Anthropic's published prompts are a goldmine of prompt engineering patterns:

Structure with XML Tags

Anthropic uses XML-style tags to organize different sections of the prompt (<safety_guidelines>, <formatting_rules>, etc.). This helps the model parse complex instructions without confusion.

Be Specific About What NOT to Do

The prompts don't just say "be helpful." They explicitly list anti-patterns: don't start with filler phrases, don't hedge when you're confident, don't refuse ambiguous requests without considering positive intent.

Set a Clear Instruction Hierarchy

The prompts establish explicit priority: core safety > system prompt > user instructions. This prevents prompt injection attacks from overriding safety-critical behavior.

Inject Dynamic Context

Rather than relying on the model's training data for temporal awareness, the prompt injects the current date at runtime. This pattern — dynamic context injection — is essential for any production AI application.

Design for Edge Cases

The prompts devote significant space to edge cases: what happens when a user is suicidal, when a conversation gets flagged, when a request is ambiguous. Production AI systems fail on edge cases, and Anthropic's prompts show how to handle them systematically.

The Bigger Picture: AI's Open-Source Constitution

There's something poetically fitting about an AI company publishing the rules that govern its AI. It's the software equivalent of a government publishing its constitution — an acknowledgment that the rules should be visible to the governed.

Anthropic calls its approach Constitutional AI, and the system prompt is where that constitution becomes operational.

It's where abstract values like "helpful, harmless, and honest" get translated into concrete instructions: this is what honest looks like when someone asks about a controversial politician; this is what harmless looks like when someone mentions self-harm; this is what helpful looks like when someone needs code reviewed.

As AI systems become more powerful and more autonomous — writing code, browsing the web, managing workflows — the question of "what rules is this thing following?" becomes not just academic but urgent.

Anthropic's system prompts page is one answer to that question. It's imperfect, it's incomplete, but it exists.

And in an industry where the default is to hide everything, existence is a radical act.

TL;DR

What: Anthropic publishes the system prompts for Claude on platform.claude.com/docs/en/release-notes/system-prompts — the only major AI lab to do so voluntarily.
Why it matters: These prompts are Claude's operating manual — they define identity, safety rules, formatting preferences, tool usage, and personality.
What's inside: 70+ sections covering everything from "don't say Certainly!" to child safety escalation protocols and multi-layered tool definitions.
How they've evolved: From warm-and-fuzzy to direct-and-expert, from minimal tools to a full agentic toolkit, from verbose safety rules to lean, training-aligned instructions.
The industry impact: OpenAI and Google keep their prompts secret; Anthropic's transparency sets a standard that regulation and competition will eventually force others to meet.
For developers: The prompts are a masterclass in prompt engineering — study them for patterns in structure, safety, dynamic context injection, and edge case handling.

The system prompts page is available at platform.claude.com/docs/en/release-notes/system-prompts. For ongoing analysis, follow Simon Willison's blog at simonwillison.net, where he tracks prompt changes with git-style diffs.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

How to Hide Your Backend VPS IP Behind Vercel Using Next.js Rewrites

Pradeep Kumar — Tue, 18 Aug 2026 06:32:01 +0000

If you host your frontend on Vercel and your backend on a custom VPS, your frontend usually makes API requests directly to the VPS's public IP address or domain. While this works, it exposes your backend server's true IP address to the world, making it vulnerable to direct DDoS attacks, port scanners, and malicious bots.

By using Vercel Rewrites, you can use Vercel's edge network as a shield. To the outside world, your API requests appear to be going to Vercel, while Vercel secretly fetches the data from your VPS behind the scenes.

Here is how you can set up a secure API proxy in Next.js (App Router) and lock down your backend to reject any request that tries to bypass Vercel.

The Architecture

sequenceDiagram
    actor User
    participant Vercel as Vercel (Next.js)
    participant VPS as VPS (NestJS Backend)

    User->>Vercel: Fetch /api-proxy/data
    Note over Vercel: Middleware injects secret header<br/>next.config.ts rewrites path
    Vercel->>VPS: Fetch /data (with Secret Header)

    alt Valid Secret
        VPS-->>Vercel: 200 OK (Data)
        Vercel-->>User: 200 OK (Data)
    else Missing/Invalid Secret
        VPS-->>Vercel: 403 Forbidden
    end

    User-xVPS: Direct Request (Blocked 403)

Step 1: Configure Next.js Rewrites

First, we need to tell Next.js to intercept requests made to a specific proxy path (e.g., /api-proxy/*) and forward them to your VPS. We achieve this by adding a rewrite rule to next.config.ts.

To ensure we don't accidentally create proxy loops, we will use a specific environment variable BACKEND_URL solely for the rewrite destination.

// next.config.ts
export default {
  async rewrites() {
    // The true IP of your VPS, safely hidden in Vercel environment variables
    const apiUrl = process.env.BACKEND_URL || 'http://localhost:4000';

    return [
      { 
        source: '/api-proxy/:path*', 
        destination: `${apiUrl}/:path*` 
      },
    ];
  },
};

Step 2: Inject a Secret Header at the Edge

Now that the request is proxying through Vercel, we need a way for our backend to know that the request actually came from Vercel. We do this by injecting a secret header.

In Next.js, we can use Edge Middleware to intercept the request and inject the header before the next.config.ts rewrite takes effect.

// middleware.ts (or proxy.ts if you use a custom setup)
import { NextResponse, type NextRequest } from 'next/server';

export function middleware(req: NextRequest) {
  const requestHeaders = new Headers(req.headers);
  const proxySecret = process.env.VERCEL_PROXY_SECRET;

  // Inject the secret key into the headers
  if (proxySecret) {
    requestHeaders.set('x-vercel-proxy-secret', proxySecret);
  }

  // Pass the modified headers along to the Next.js rewrite engine
  return NextResponse.next({
    request: {
      headers: requestHeaders,
    },
  });
}

export const config = {
  matcher: ['/api-proxy/:path*'],
};

Step 3: Lock Down the Backend

If an attacker guesses your VPS IP, they could still bypass Vercel entirely. We need to configure the backend to reject any request that lacks the secret header.

Here is an example of how to enforce this using a global hook in a NestJS + Fastify backend:

// backend/src/main.ts
const fastifyInstance = app.getHttpAdapter().getInstance();

fastifyInstance.addHook('onRequest', (request, reply, done) => {
  const proxySecret = process.env.VERCEL_PROXY_SECRET;

  // Exclude localhost (127.0.0.1) so internal Docker health checks still work!
  const isLocal = request.ip === '127.0.0.1' || request.ip === '::1';

  if (proxySecret && !isLocal) {
    const incomingSecret = request.headers['x-vercel-proxy-secret'];
    if (incomingSecret !== proxySecret) {
      reply.status(403).send({ message: 'Forbidden: Invalid proxy secret' });
      return;
    }
  }
  done();
});

[!TIP] Docker Health Checks: Notice the isLocal check. If you use Docker, your container's internal health check pinging localhost won't have the secret header. Bypassing the check for local IPs prevents your deployments from rolling back due to failed health checks!

Step 4: Fixing Server-Side Rendering (SSR/SSG)

There's one hidden "gotcha" with this setup. During the Next.js build process (next build), Next.js generates static pages and makes fetch requests directly to your backend API.

Because these requests originate from the build server and bypass your Next.js middleware, they will fail with a 403 Forbidden error because they lack the x-vercel-proxy-secret header.

To fix this, you must manually inject the secret into your internal API client or Apollo Client configuration:

// lib/apollo-server.ts
import { ApolloClient, InMemoryCache, HttpLink } from '@apollo/client';

const API_URL = process.env.NEXT_PUBLIC_API_URL || 'http://localhost:4000';

export const apolloServerClient = new ApolloClient({
  ssrMode: true,
  link: new HttpLink({ 
    uri: `${API_URL}/graphql`, 
    fetch, 
    headers: process.env.VERCEL_PROXY_SECRET 
      ? { 'x-vercel-proxy-secret': process.env.VERCEL_PROXY_SECRET } 
      : {}
  }),
  cache: new InMemoryCache(),
});

Summary

By combining Next.js rewrites, Edge middleware, and a simple backend validation check, you can successfully shield your custom VPS behind Vercel's robust infrastructure. Your true IP address remains a secret, and your API is locked down against direct unauthorized access!

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

Google Launches Gemini 3.7 Flash Three Weeks After 3.6 With 50% Introductory Pricing

Pradeep Kumar — Mon, 17 Aug 2026 10:53:24 +0000

Google DeepMind released Gemini 3.7 Flash on August 13, 2026, just three weeks after Gemini 3.6 Flash. This is not a new model family or a full generational reset. Google describes it as the next iteration in the Gemini 3 line, based on Gemini 3.6 Flash with algorithmic improvements to its reasoning foundation.

That distinction matters. The interesting question is not whether Gemini 3.7 Flash is simply “smarter.” It is whether Google has made a fast, relatively inexpensive model good enough to handle more real production work—especially coding, tool use, web development, and business workflows—without requiring the cost of a larger frontier model.

Google is clearly positioning it that way. Its announcement calls 3.7 Flash its “most intelligent workhorse model yet” for coding and agents, while the model card emphasizes agentic workflows, coding tasks, and enterprise workflows as intended uses. Google Google DeepMind model card

What Changed in Gemini 3.7 Flash?

Gemini 3.7 Flash is built on Gemini 3.6 Flash rather than being trained as a completely separate architecture. Google says the model introduces algorithmic improvements to its core reasoning foundation and supports customizable thinking configurations that let developers trade off quality, cost, and latency.

The basic model envelope remains substantial: up to a 1-million-token context window, 64K-token output, and multimodal input across text, images, audio, and video. The knowledge cutoff is March 2026. Google DeepMind model card

Feature	Gemini 3.7 Flash	Gemini 3.6 Flash
Context window	1M tokens	1M tokens
Maximum output	64K tokens	64K tokens
Input modalities	Text, image, audio, video	Text, image, audio, video
Thinking	Configurable	Configurable
Knowledge cutoff	March 2026	January 2026
Model dependency	Based on 3.6 Flash	—
API model name	gemini-3.7-flash	gemini-3.6-flash

The important change is not a larger context window or a radically different interface. It is getting more useful work out of the Flash model class.

Choosing the Thinking Level

Configurable thinking gives developers control over the quality/cost/latency trade-off, which matters when one request can trigger multiple model calls and tool interactions. Think of it as an engineering choice rather than a simple “higher is better” switch:

flowchart LR
    A([Incoming Request]) --> B{Primary priority?}
    B -->|Lower latency / cost| C[Use lighter thinking\nFast responses\nSimple classification\nRoutine tasks]
    B -->|Balanced| D[Use moderate thinking\nCoding\nAgent workflows\nGeneral tasks]
    B -->|Hard reasoning| E[Use deeper thinking\nComplex debugging\nMulti-step planning\nDifficult analysis]

The exact settings available depend on the API and model configuration, so developers should treat thinking as a tunable resource rather than a universal preset.

The Biggest Gains Are in Software Engineering

Coding is where Gemini 3.7 Flash makes its clearest case.

On FrontierCode 1.1 Main, Google's published result is 43.6%, up from 34.4% for Gemini 3.6 Flash. On DeepSWE v1.1, which measures long-horizon software engineering, 3.7 Flash reaches 65.3%, compared with 48.6% for 3.6 Flash.

The competitive picture is more nuanced. Gemini 3.7 Flash is ahead of Claude Sonnet 5 on FrontierCode, but GPT-5.6 Terra remains ahead on DeepSWE at 69.6%. That matters because long-horizon software engineering is closer to the kind of multi-file, multi-step work developers increasingly expect from coding agents. Google DeepMind model card

Terminal evaluations tell a similar story. Gemini 3.7 Flash scores 85.8% on Terminal-bench 2.1, up from 78.0% for 3.6 Flash. On the harder Terminal-bench 3.0, it reaches 14.9%, versus 5.4% for 3.6 Flash. GPT-5.6 Terra still leads Terminal-bench 3.0 at 20.8%.

The takeaway is more useful than any single score: the model is getting better at doing engineering work through tools, not just answering isolated coding questions.

xychart-beta
    title "Gemini 3.7 Flash: Improvement Over 3.6 Flash"
    x-axis ["FrontierCode", "DeepSWE", "AutomationBench", "GDP.pdf", "LABBench2", "HLE"]
    y-axis "Improvement (percentage points)" 0 --> 20
    bar [9.2, 16.7, 13.4, 12.0, 6.0, 2.4]

Web Development and UI Generation

Web development is another notable focus.

On Code Arena's WebDev Arena, Gemini 3.7 Flash scores 1,588 Elo, compared with 1,538 for Gemini 3.6 Flash. It also ranks above Claude Sonnet 5 at 1,541 and GPT-5.6 Terra at 1,523 in Google's published comparison.

Google says the model can produce more functional layouts and feature-complete applications in fewer prompts. It also highlights stronger adherence to reference designs, screenshots, and design systems.

That matters for frontend developers because the useful benchmark is not “can the model write React?” Modern coding assistants already can. The harder problem is whether the model can translate a visual or product specification into a working interface without requiring a long sequence of corrective prompts.

Documents, Legal Work, and Business Automation

Gemini 3.7 Flash also makes a strong showing outside traditional coding benchmarks.

On GDP.pdf, a document-comprehension benchmark, it scores 34.0%, up from 22.0% for Gemini 3.6 Flash. On Harvey LAB-AA, which evaluates complex legal workflows, it reaches 90.7%, narrowly ahead of Claude Sonnet 5 at 90.1%.

The most striking number may be AutomationBench. Gemini 3.7 Flash reaches 30.4%, compared with 17.0% for 3.6 Flash. Google's model card notes that AutomationBench is a private set, so the result should be interpreted primarily as a directional comparison rather than a universally reproducible leaderboard number.

That distinction is important. Private benchmarks can tell us that a model improved relative to another model under a particular evaluation setup, but they do not give developers the same reproducibility as a public benchmark.

Reasoning and Scientific Work

The model's gains are not limited to business tasks.

On HLE-Verified, Gemini 3.7 Flash scores 53.6%, up from 51.2% for 3.6 Flash and above Claude Sonnet 5 at 31.0%. On LABBench2, which evaluates biology research tasks, it reaches 82.1%, compared with 76.1% for its predecessor.

There are regressions too. On CharXiv without tools, 3.7 Flash scores 84.5%, slightly below 3.6 Flash at 85.2%. With tools enabled, 3.6 Flash also remains ahead, 89.4% to 88.7%.

That is a useful reminder that “model improvement” is not a monotonic property. A release can improve substantially on coding and agentic workflows while moving backward on a narrower class of reasoning tasks.

Gemini 3.7 Flash vs the Competition

Here is the more useful snapshot from Google's published evaluation table:

Benchmark	Gemini 3.7 Flash	Gemini 3.6 Flash	Claude Sonnet 5	GPT-5.6 Terra
FrontierCode 1.1 Main	43.6%	34.4%	42.7%	41.3%
DeepSWE v1.1	65.3%	48.6%	53.8%	69.6%
WebDev / Code Arena	1,588 Elo	1,538	1,541	1,523
Terminal-bench 2.1	85.8%	78.0%	80.4%	87.4%
Terminal-bench 3.0	14.9%	5.4%	14.6%	20.8%
AutomationBench	30.4%	17.0%	10.7%	23.6%
GDP.pdf	34.0%	22.0%	28.0%	24.7%
Harvey LAB-AA	90.7%	85.1%	90.1%	85.2%
HLE-Verified	53.6%	51.2%	31.0%	51.1%
OSWorld-2.0	47.9%	33.8%	—	50.2%
AI Intelligence Index	56	52	55	57

Source: Google DeepMind Gemini 3.7 Flash model card.

The table makes the central point clear: Gemini 3.7 Flash is not the universal benchmark leader. It wins important categories while remaining below the frontier on several of the hardest agentic evaluations.

The Pricing Story May Matter More Than the Leaderboard

For production systems, benchmark position is only half the equation. The other half is what it costs to run the model repeatedly.

Google is launching Gemini 3.7 Flash at an introductory price of $0.75 per million input tokens and $3.75 per million output tokens through December 31, 2026. Starting January 1, 2027, the price becomes $1.50 per million input tokens and $7.50 per million output tokens. Google Google DeepMind model card

That is why the headline should be understood as 50% introductory pricing relative to the original 3.6 Flash launch price, not as a permanent price cut.

For a simple 80/20 input-output workload, the blended cost is roughly:

Model	Input / 1M	Output / 1M	Approx. 80/20 blended cost
Gemini 3.7 Flash — intro	$0.75	$3.75	$1.35
Gemini 3.7 Flash — from Jan. 2027	$1.50	$7.50	$2.70
Claude Sonnet 5	$2.00	$10.00	~$3.60
GPT-5.6 Terra	$2.00	$12.00	~$4.00

The real economic question is cost per successful task. A cheaper model that needs more retries, tool calls, or human correction may not actually be cheaper. That is where 3.7 Flash’s coding and tool-use gains become commercially interesting.

timeline
    title Gemini 3.7 Flash Pricing Lifecycle
    Aug 13 2026 : Introductory launch
               : $0.75 input / 1M
               : $3.75 output / 1M
    Dec 31 2026 : Introductory pricing expires
    Jan 1 2027  : Standard rate begins
               : $1.50 input / 1M
               : $7.50 output / 1M

Where Gemini 3.7 Flash Still Trails

The strongest case for Gemini 3.7 Flash is not that it dominates every benchmark. It doesn't.

The Artificial Analysis Intelligence Index puts Gemini 3.7 Flash at 56, behind GPT-5.6 Terra at 57. On DeepSWE, GPT-5.6 Terra retains a meaningful 4.3-point lead. GPT-5.6 Terra also leads Terminal-bench 2.1, Terminal-bench 3.0, and OSWorld-2.0 in Google's comparison.

Those gaps matter for teams building agents that must operate across long, unpredictable sequences of actions. A 14.9% score on Terminal-bench 3.0 is dramatically better than 5.4% for the previous model, but it is still far from a solved problem.

That is the more realistic way to view this release: Gemini 3.7 Flash moves the cheaper Flash tier closer to frontier capability, but it does not erase the gap between a fast workhorse model and the best-performing models on the hardest agentic tasks.

What the Release Means for Developers

For developers, the appeal is straightforward: better coding without moving to a more expensive tier, cheaper agentic workflows, and broader production use across coding, documents, web development, and automation.

That suggests a shift in how models should be evaluated. The better question is not “Which model has the highest benchmark score?” but “Which model can complete my workflow reliably at a cost I can scale?”

Availability

Gemini 3.7 Flash is available through the Gemini API, Google AI Studio, Android Studio, Google Antigravity, Gemini Enterprise Agent Platform, and the Gemini Enterprise app. Google also says Gemini Spark will use 3.7 Flash for Google AI Pro and Ultra subscribers in supported countries. Google Google DeepMind model card

The model card lists the API, AI Studio, Antigravity, and enterprise channels explicitly. Developers should still check the current API documentation before migrating production workloads because model availability, parameters, and pricing can change.

Safety and Limitations

Google says Gemini 3.7 Flash ships with updated safeguards targeting misuse in CBRN and cyber offense domains. Its Frontier Safety assessment reports that the model did not reach tracked or critical capability thresholds, although it did reach the alert threshold for the Level 1 cybersecurity critical-capability category without crossing the capability threshold itself. Google DeepMind model card

The model card also lists familiar foundation-model limitations, including hallucinations and possible slowness or timeouts. For high-stakes applications, developers still need application-level validation, permission boundaries, monitoring, and fallback paths.

What This Release Signals

The timing may be as interesting as the benchmark numbers. Google shipped 3.7 Flash only three weeks after 3.6 Flash, reinforcing an increasingly iterative approach to its workhorse models.

The bigger story is economic: a model capable enough to code, use tools, process documents, and automate business tasks can matter more than another leaderboard leader if developers can run it at scale.

The real test comes after the introductory price expires on January 1, 2027. If teams keep 3.7 Flash in production because it completes enough work with few enough retries, this release will have mattered for reasons that have little to do with benchmark headlines.

For now, Gemini 3.7 Flash looks like a cheaper workhorse moving closer to frontier capability—and making production-scale agents easier to justify.

Sources

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

The Rise of AI Trading Among Retail Investors

Pradeep Kumar — Sat, 15 Aug 2026 14:28:59 +0000

On May 27, 2026, Robinhood let customers connect Claude, ChatGPT, and other AI agents straight to their trading accounts. The agents can watch markets, rebalance a portfolio, and place real trades, wrapped in guardrails like a dedicated account, spending limits, and a one-tap kill switch. Robinhood built the feature on top of the Model Context Protocol, the same standard developers use to wire agents into any other tool.

That single launch says a lot about where retail investing landed this year. AI trading isn't a hedge fund exclusive anymore. It's a toggle in a consumer app.

The Numbers Behind the Shift

The AI trading platform market was worth $11.2 billion in 2024. Grand View Research puts it at $16.2 billion in 2026, on track for $33.5 billion by 2030 at a 20% annual growth rate.

Retail adoption is keeping pace. A March 2026 Investing.com survey of 938 American investors found 65% of AI users say the technology improved their market performance. Most of the same respondents still cross-check AI output against other sources before they act on it, so trust is growing but it isn't blind.

The shift isn't only American. Precedence Research notes that Chinese retail investors have started leaning on tools like DeepSeek to sharpen their trading decisions, and expects Asia-Pacific to grow fastest as India and China add retail traders every quarter.

What Retail Investors Are Actually Using

Three distinct layers have formed under the "AI trading" label. Each one asks for a different amount of technical comfort.

Brokerage-native tools

Robinhood shipped Cortex and Robinhood Strategies before Agentic Trading arrived, and the feature now serves the company's 27 million funded customers. In India, Zerodha's no-code Streak platform plugs directly into its Kite terminal, letting users build and backtest rule-based strategies without writing a line of code, and SEBI-registered advisors like Jarvis Invest now run algorithmic rebalancing on real client assets.

Third-party bot platforms

Crypto-focused products sit a level down from brokerage features. 3Commas and Cryptohopper let traders run grid or dollar-cost-averaging bots without touching an API, trading some flexibility for a much shorter setup.

API-first infrastructure

This is the layer developers actually build on. Alpaca won BrokerChooser's 2026 award for Best Broker for Algorithmic Trading, and its Trading API ships official MCP servers so an agent running in Claude or ChatGPT can call the same endpoints that power live trading.

Building Your Own: The Developer Path

Here's roughly how one of these agentic pipelines works end to end, guardrails included.

flowchart LR
    A[Market Data Feed] --> B[Signal / Model]
    B --> C{AI Agent Decision}
    C -->|Within limits| D[Guardrails Check]
    C -->|Flagged| E[Human Review]
    D --> F[Order Execution]
    E --> F
    F --> G[(Broker: Alpaca / Robinhood)]

The mechanics behind that diagram are almost boring now. alpaca-py, Alpaca's official Python SDK, reduces "fetch data, check a condition, place an order" to about 20 lines, and paper trading means none of it touches real money until you flip one flag.

from datetime import datetime, timedelta
from alpaca.trading.client import TradingClient
from alpaca.trading.requests import MarketOrderRequest
from alpaca.trading.enums import OrderSide, TimeInForce
from alpaca.data.historical import StockHistoricalDataClient
from alpaca.data.requests import StockBarsRequest
from alpaca.data.timeframe import TimeFrame

trading_client = TradingClient(API_KEY, SECRET_KEY, paper=True)
data_client = StockHistoricalDataClient(API_KEY, SECRET_KEY)

bars = data_client.get_stock_bars(
    StockBarsRequest(
        symbol_or_symbols="AAPL",
        timeframe=TimeFrame.Day,
        start=datetime.now() - timedelta(days=30),
    )
).df

sma_20 = bars["close"].tail(20).mean()
last_close = bars["close"].iloc[-1]

if last_close > sma_20:
    order = MarketOrderRequest(
        symbol="AAPL", qty=1, side=OrderSide.BUY, time_in_force=TimeInForce.DAY
    )
    trading_client.submit_order(order_data=order)

That's a skeleton for wiring up the API, not a strategy worth running. Pair it with FinRL for reinforcement learning experiments or Backtrader for backtesting, and a solo developer now has most of what a small quant desk had a decade ago.

The gap between "have an idea" and "have a working bot" has basically closed. That's most of the reason retail AI trading grew so fast this year.

The Reality Check

Easy to build doesn't mean easy to profit from. The clearest evidence comes from Alpha Arena, a live benchmark from startup Nof1 that hands frontier AI models real money and lets them trade in public.

In its first season, six models each traded $10,000 in crypto perpetual contracts from October 17 to November 3, 2025. By the end, ChatGPT was down $6,267, Gemini was down $5,671, Grok was down $4,531, and Claude Sonnet was down $3,081; only DeepSeek and Qwen3 Max finished in the green, up $489 and $2,232.

A later round moved the same idea to US tech stocks, and Bloomberg reported the results didn't improve. Across 32 total contest outcomes, a model finished in profit only six times, and the group lost close to a third of its combined capital. Two models given the exact same prompt still traded wildly differently: one placed 158 trades, the other 1,418.

Nof1 founder Jay Azhang put it simply: current models can't generate returns on their own, they need serious surrounding infrastructure just to have a shot. Doug Clinton, who runs the LLM-driven fund Intelligent Alpha, described the models as having "personalities that you have to manage almost like a human analyst," since the same prompt can leave one model bullish and another bearish for no clear reason.

Prediction markets add a useful caveat here. CoinDesk reported in March 2026 that only 7% to 13% of human traders on Polymarket turn a consistent profit, a genuinely low bar. Olas co-founder David Minarsch told the outlet his firm's Polystrat agent saw over 37% of its wallets finish positive in its first month live, though that number is the company's own claim about its own product, not an independent audit, a distinction worth holding onto in a space full of impressive self-reported figures.

Hidden costs explain part of every automated strategy's underperformance too. A system placing dozens of trades a day accumulates fees and slippage fast enough to turn a solid gross return into a net loss, and a backtest that looks strong across years of historical data can still blow through its stop-loss once real conditions shift underneath it.

Regulators Are Paying Attention

FINRA's December 2025 oversight report added a dedicated GenAI section for the first time, flagging autonomous AI agents as their own risk category. The concerns are specific: agents acting without human validation, agents operating past their intended scope, and generative tools producing fake news or coordinated social posts built to look like real retail enthusiasm.

The SEC is pointing the same direction. Its 2026 exam priorities now expect firms to explain what their AI systems do, who supervises them, and how risks get caught before they compound.

Robinhood addresses this directly in its own disclosures. The company states plainly that it doesn't control, supervise, monitor, or audit third-party AI agents once they're connected, and that customers assume all risk for orders those agents place.

India has taken a more structured path. SEBI released a draft circular in December 2024 opening algorithmic trading to retail investors through exchange-approved broker APIs, and finalized the framework in February 2025 with every order tagged by a unique Algo-ID for the audit trail. It's a middle ground between banning automation outright and leaving it unregulated.

Where This Goes

None of this makes AI trading a fad. The infrastructure is real, from broker APIs to MCP servers to no-code builders, and it's improving fast.

What the 2026 data actually supports is narrower than the hype suggests. Even Nof1, the startup running Alpha Arena, is building its actual business around helping retail traders assemble their own agents rather than betting on a model that trades unsupervised. AI helps most as a research and execution layer, and the accounts that hold up tend to keep a human setting the boundaries instead of handing over the keys entirely.

For developers, that's the more interesting problem to build for anyway. A system that surfaces a signal and explains its reasoning is a harder, more useful thing to ship than one that just fires off trades in the dark.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!

AI Doesn't Ship Software. You Still Do.

Pradeep Kumar — Sat, 01 Aug 2026 16:09:38 +0000

AI made the easy 90% of software instant. The hard 90% is still yours.

You type one sentence into a prompt box. Ninety seconds later, there's a running app.

A form. A database. A chart that updates when you submit it. It looks done.

It isn't.

Open two browser tabs. Log in as the same user in both.

Submit the same form twice, fast — the way a nervous user does when the button doesn't visibly respond.

Leave the thing running for a week instead of a five-minute demo.

Somewhere in there, the wheels come off. A race condition nobody wrote a test for. A query that's instant at 40 rows and takes 40 seconds at 4 million. A webhook handler that isn't idempotent, so a single retried request charges someone twice. A rate limiter that doesn't exist, so one flaky client retrying in a loop quietly takes the whole service down.

Pull up the git history. There's a decent chance an API key is sitting in a commit from three days ago, in plain text, one git push from being public forever.

None of this means the model failed. It built exactly what you asked for: something that runs.

Whether it survives real users, real load, and real time was never the question it was answering.

An old joke that's suddenly relevant again

In 1985, a Bell Labs programmer named Tom Cargill coined a line that Jon Bentley made famous in a Communications of the ACM column.

The first 90 percent of a project's code takes 90 percent of the time. The remaining 10 percent takes the other 90 percent.

Bentley called it, only half-jokingly, the "Rule of Credibility" — a warning to anyone tempted to trust a schedule from someone who'd never shipped anything to production.

Forty years later, the joke lands harder than it used to.

AI collapsed exactly the part of the timeline the rule isn't about. Scaffolding an app — the login form, the CRUD boilerplate — used to eat real days. Now it's minutes.

What Cargill was pointing at was always the second 90 percent: migrations that can't safely run twice, and the edge case where a deleted user's session token quietly keeps working for six more hours because nobody thought to invalidate it.

That part hasn't gotten any faster. It was never a typing problem to begin with.

Judgment was always the expensive part

Ask an engineer with ten years of on-call scars what they think about while reviewing a pull request. It's rarely the syntax.

It's this: What happens under load? What's the blast radius if this throws at 3 a.m. and pages someone? Where does this leak a secret, a connection, or a customer's data?

Those aren't questions you answer by reading more code. They're questions you answer by having watched systems break before, and having built a mental model of why.

A language model doesn't have that model. It has, at enormous scale, a very good sense of what competent code looks like.

That's genuinely useful. It's also not the same thing as knowing what competent code does under pressure.

Ask it for an authentication flow and you'll get something clean, conventional, and quietly wrong — in a way that won't surface until one specific, unlucky sequence of events happens in production.

At which point it's not a code review comment anymore. It's an incident.

This isn't an argument against learning to code

It's the opposite, actually. Worth saying plainly, because a lot of people are drawing the wrong conclusion from how good the tools have gotten.

The part AI is genuinely eating is the mechanical translation of a spec into syntax — which was always the least interesting part of the job anyway.

Junior work that amounted to "implement this ticket exactly as written" is getting cheaper by the month. It's not coming back.

But the value of understanding how a database index actually works, or why two requests hitting the same row in the same millisecond corrupt data, hasn't dropped.

If anything, it's gone up. That understanding is now the only thing standing between a working demo and an outage.

The engineers pulling ahead right now aren't the ones who avoided AI. They're the ones using it the way you'd use a very fast, very confident junior developer: delegate the boilerplate, then read every line that comes back like it's a pull request from someone you've never worked with before.

What is it assuming about the input? What happens if this function gets called twice at once? Is the error actually handled, or just caught and silently swallowed?

None of that is a new skill. It's the same one code review has always demanded, aimed now at a collaborator that writes fast and never once says "I'm not sure about this part."

The engineers falling behind aren't the ones without AI access — everyone has AI access. It's the ones who ship what it hands them without being able to explain why it works, and who have no idea where to even start once it stops.

That's the actual gap AI hasn't closed. It never set out to.

This piece was sparked by Anuradha Weeraman's essay on the same gap — worth the five minutes.

Originally published on ZyVOP

💡 For more articles like this, subscribe to the ZyVOP newsletter!