DEV Community: Anoop Kumar

System Design Fundamentals I Wish I Had Learned Earlier

Anoop Kumar — Fri, 21 Aug 2026 15:17:19 +0000

Most system design resources focus on interview preparation — how to design Twitter in 45 minutes, how to draw boxes and arrows convincingly. This article is different. These are the concepts I wish I had internalized earlier because they changed how I approach every technical decision, including the ones I make building TokenPulse.

The fundamental trade-off: consistency vs availability

The CAP theorem states that a distributed system can guarantee at most two of three properties: Consistency, Availability, and Partition tolerance. Since network partitions are unavoidable in real systems, the practical trade-off is between consistency and availability.

Consistency means every read receives the most recent write. If user A writes data and user B immediately reads it, B sees A's write.

Availability means every request receives a response — not necessarily the most recent data, but a response.

Most web applications choose availability over strict consistency, accepting eventual consistency — the guarantee that data will become consistent eventually, even if reads may return slightly stale data during the propagation window.

The practical example: if you update your profile picture on a social platform, some users might see the old picture for a few seconds or minutes while the change propagates. This is acceptable. If your bank account balance is wrong for even one second, that is not acceptable.

Knowing which side of this trade-off your application falls on should drive every database and caching decision you make.

Horizontal vs vertical scaling — and when each applies

Vertical scaling (scaling up) means adding more resources to a single machine — more CPU, more RAM, faster storage. It is simple, requires no architectural changes, and works until you hit the hardware limit or the cost becomes prohibitive.

Horizontal scaling (scaling out) means adding more machines and distributing load across them. It requires your application to be stateless — requests from the same user can be handled by any server.

The question developers often ask is "which should I use?" The correct answer is: vertical first, horizontal when necessary.

Vertical scaling is dramatically simpler. You resize an instance and you are done. Horizontal scaling requires load balancers, session management, distributed caching, and careful thinking about where state lives.

The mistake is assuming you need horizontal scaling before you have validated the need. A well-optimized single server can handle enormous traffic. Stack Overflow ran on five servers for years at significant scale. Optimize your code and your queries before you add machines.

When you do need horizontal scaling, the key constraint is statelessness. Your application servers must not store session state locally. Sessions go in Redis. Files go in object storage. Database writes go to a shared database. Each server is identical and interchangeable.

Caching — the most important performance tool

Caching is storing the result of an expensive computation so you can serve it quickly on subsequent requests. It is the single highest-leverage performance improvement in most systems.

There are four places you can cache:

Client-side (browser cache): HTTP cache headers (Cache-Control, ETag, Last-Modified) tell browsers how long to cache static assets. A properly configured CDN and cache policy means your CSS and JavaScript files are served from the user's local cache — zero network round-trips.

CDN cache: Content delivery networks cache responses at edge nodes geographically close to users. Dynamic content that does not change per-user — marketing pages, blog posts, public API responses — should be cached at the CDN.

Application cache (Redis/Memcached): Server-side in-memory caching for database query results, computed aggregations, and anything expensive to regenerate. Redis is the standard choice — it is fast, supports data structures, and handles expiration automatically.

Database query cache: Most databases cache query results internally. This is largely automatic, but it means repeated identical queries are fast and schema changes or cache invalidation can cause temporary performance drops.

The hardest problem in caching is cache invalidation — knowing when to expire stale data. Three strategies:

TTL (time-to-live): Data expires after a fixed duration. Simple, but data may be stale within the TTL window.

Write-through: When data is written to the database, the cache is updated simultaneously. Consistent, but every write hits both database and cache.

Cache-aside: Application checks cache first. On a miss, reads from database and populates the cache. On a write, invalidates the cache entry. The most common pattern for read-heavy workloads.

Database indexing — what it is and why it matters

Without an index, a database must scan every row in a table to find matching records. This is a full table scan, and its cost grows linearly with the number of rows.

An index is a separate data structure (typically a B-tree) that allows the database to find matching rows in O(log n) time instead of O(n) time. For a table with one million rows, this is the difference between scanning 1,000,000 rows and scanning roughly 20.

The columns to index:

Primary keys — indexed automatically
Foreign keys — columns used in JOIN conditions
Columns in WHERE clauses — especially high-cardinality columns (many distinct values)
Columns in ORDER BY — if you frequently sort by a column The trade-off: indexes make reads fast and writes slow. Every INSERT, UPDATE, or DELETE must also update every relevant index. For read-heavy applications (most web applications), this trade-off is almost always worth it. For write-heavy applications (logging, analytics ingestion), index carefully.

Composite indexes — indexes on multiple columns — matter for query performance. An index on (user_id, created_at) supports queries that filter by user_id and sort by created_at, but only if the columns appear in that order in the WHERE clause.

Message queues — decoupling for resilience

A message queue is a buffer between a producer (something that generates work) and a consumer (something that processes work). The producer puts a message in the queue and moves on. The consumer reads from the queue and processes at its own pace.

Why this matters:

Handling traffic spikes. If your API receives 10,000 requests in one second, a queue absorbs the burst and processes it steadily rather than overwhelming your database.

Decoupling services. If the email service is down, you do not want to fail the user's registration request. Put the "send welcome email" task in a queue. When the email service recovers, it processes the queue.

Retry logic. If a queue consumer fails to process a message, the message stays in the queue (or moves to a dead letter queue) and is retried. Without a queue, failed tasks are lost.

The practical example from TokenPulse: when a user joins the Pro waitlist, the API endpoint puts two tasks in a queue — "send notification email" and "write to Google Sheets." If Google Sheets is temporarily unavailable, the task is retried automatically. The user's HTTP response does not wait for either task to complete.

Load balancing strategies

A load balancer distributes incoming requests across multiple servers. The distribution strategy matters:

Round robin: Requests go to servers in order — 1, 2, 3, 1, 2, 3. Simple, works when servers are identical and requests take similar time.

Least connections: Requests go to the server with the fewest active connections. Better when requests have variable processing time.

IP hash: The same client IP always routes to the same server. Useful when you cannot make your application fully stateless and need session affinity.

Weighted: Some servers receive more traffic than others — useful when servers have different capacities.

For most Next.js applications on Vercel, load balancing is handled automatically. Understanding the strategies matters when you operate your own infrastructure or need to reason about why requests are distributed the way they are.

The read replica pattern

Write operations go to a primary database. Read operations go to one or more replica databases that are kept in sync with the primary.

This pattern solves a specific problem: most web applications have dramatically more reads than writes. A read-heavy query that scans a large table on the primary database competes with write operations for I/O. Moving reads to a replica eliminates this contention.

The trade-off is replication lag — replicas may be slightly behind the primary. For most reads (show me recent posts, display the product catalog), this is acceptable. For reads that immediately follow a write (show me the thing I just created), you need to either read from the primary or implement a mechanism to wait for replication.

How to think about trade-offs like a senior engineer

Every system design decision is a trade-off. Senior engineers do not make universally correct decisions — they make decisions appropriate to their specific constraints.

The questions to ask for any architectural decision:

What is the actual scale? Do not design for 10 million users if you have 100. Design for 10x your current scale and revisit when you get there.

What is the cost of being wrong? If a caching strategy is wrong, you serve stale data. If a database schema is wrong, you have a migration. Different stakes require different confidence levels before deciding.

What can you change later? Some decisions are easily reversible — switching a caching library, changing an index. Others are difficult — changing your data model, switching databases. Spend more time on irreversible decisions.

What do you actually know? Optimize for known bottlenecks, not hypothetical ones. Use monitoring and profiling to find real problems before solving imagined ones.

If you are building AI-powered applications and want visibility into your token usage costs and rate limits, TokenPulse is a free Chrome extension that tracks usage across Claude, ChatGPT, Gemini, DeepSeek and Grok — no API key required.

What I Learned Building a Developer Tool for AI Usage — 6 Weeks of Engineering and Product Lessons

Anoop Kumar — Fri, 21 Aug 2026 15:15:18 +0000

Six weeks ago I started building TokenPulse after getting cut off mid-debugging session by Claude's rate limit one too many times. The extension is now live with active users across five platforms. Here is what I actually learned — the engineering decisions, product mistakes, and things I would do differently.

The problem was real but I almost built the wrong solution

My initial instinct was to build a browser extension that scraped Claude's rate limit data and sent it to a backend that users could check via a dashboard. I spent three days on the backend architecture before I stopped and asked the obvious question: why would someone open a separate dashboard to check information they need while they are already inside Claude?

The right solution was obvious once I asked it: inject the information directly into the page they are already on. A slim bar above the input box. Always visible, never intrusive, no tab switching required.

The lesson: solve the problem in the context where it actually occurs. Do not add steps.

MV3 was harder than the documentation suggested

Chrome's Manifest V3 requirements were the first major technical challenge. The biggest constraint is Content Security Policy — no inline scripts, no eval, no remotely loaded JavaScript.

This broke my initial implementation in three ways:

Inline event handlers. Every onclick="..." attribute in my HTML failed silently. The fix is addEventListener for everything, wired after DOMContentLoaded. Simple in hindsight, took me a day to fully debug.

Service worker lifecycle. MV3 background scripts are service workers — Chrome can kill them at any time. Code that assumes the background script is alive between messages will fail intermittently and be almost impossible to reproduce. Everything has to go through chrome.storage.local.

Message channel management. The chrome.runtime.onMessage listener has a specific contract: return true if you are going to call sendResponse asynchronously, return false (or nothing) if you are not. Getting this wrong causes the dreaded "message channel closed before response was received" error that appears randomly and is hard to trace.

The dual selector problem

Every platform — Claude, ChatGPT, Gemini, DeepSeek, Grok — updates its frontend regularly. I learned this the hard way when a ChatGPT update broke my content script three days after I published.

The solution is defensive selector arrays with fallbacks:

const INPUT_SELECTORS = [
  '#prompt-textarea',           // primary
  '[data-id="prompt-textarea"]', // fallback 1
  'div[contenteditable="true"]', // fallback 2
]

function findInput() {
  for (const selector of INPUT_SELECTORS) {
    const el = document.querySelector(selector)
    if (el) return el
  }
  return null
}

Never rely on a single selector. Always have at least two fallbacks. Check for null before every DOM operation.

I shipped a feature nobody asked for and skipped one everyone wanted

I spent significant time building a detailed token tips panel — a collapsible section in the popup with optimization advice. Nobody uses it. Zero feedback mentions it. It adds visual weight and code complexity for no user value.

Meanwhile, the most common piece of feedback in the first two weeks was "can you show me how much I've spent this week across all platforms?" — a weekly summary. I had cost tracking per conversation but no weekly rollup.

The lesson is not "do user research before building" — I had done that. The lesson is that users cannot tell you what they want until they are using the product. The tips panel sounded useful in theory. The weekly summary revealed itself from actual usage patterns.

Ship the minimum, watch what users do, build what they actually reach for.

The notification system needed three rewrites

My first notification implementation fired every time the threshold was crossed, which meant multiple notifications within the same session. Users turned it off immediately.

My second implementation stored the last notified percentage and only fired when the user crossed a higher threshold. Better — but it never reset, so once you hit 90% and your session reset, you would not get notified at 75% in the next session.

The third implementation tracks the last notified threshold per window, resets it when usage drops below the threshold, and fires once per crossing per window. This is the version that users actually keep enabled.

The algorithm sounds simple when written out. Getting there took three iterations and specific user feedback about each failure mode.

Choosing Vercel over AWS was correct

I spent two days considering hosting options for the marketing site before choosing Vercel. The alternative I seriously considered was AWS — EC2 with nginx and PM2.

Vercel was correct for three reasons:

Zero operational overhead. No instance management, no SSL renewal, no process monitoring. Deployment is git push.
The credentials problem doesn't change. Whether you deploy on Vercel or EC2, you still need Google Sheets API credentials in environment variables. Switching infrastructure doesn't make that easier.
Revenue is zero. Until the product generates money, infrastructure cost is a constraint. Vercel hobby tier is free. EC2 t3.micro is $8-10/month. The right time to consider AWS is when you have specific requirements Vercel cannot meet — custom runtime environments, GPU access, VPC networking, specific regional compliance. For a Next.js marketing site with API routes, those requirements do not exist yet.

The Chrome Web Store review process is slower than you expect

First submission: 3 days to review, rejected for missing privacy policy disclosures in the store listing.

Second submission after fixing: 2 days to review, approved.

This means plan for a minimum 5-7 day runway between "finished" and "published." If you are building toward a launch date, submit to the Chrome Web Store at least a week before you want to launch.

Also: the Chrome Web Store's description field is SEO. The keywords in your listing title and description affect where your extension appears in the store's own search. "TokenPulse — Claude Rate Limit & Token Tracker" performs better than "TokenPulse" alone.

What I would do differently

Build the popup last. The in-page bar is the core value. I built the popup first because it felt more like a "real" extension. The bar is what users actually use 90% of the time.

Write the content scripts with a test harness from day one. Testing content scripts means opening the browser, loading the extension, navigating to the target page, and observing behavior. This loop takes 2-3 minutes per test. A mock DOM environment for unit testing would have saved significant time on the notification system rewrites.

Set up error tracking earlier. I added error logging after shipping. I had no visibility into content script failures for the first two weeks. Users were silently experiencing broken functionality that I only discovered through direct feedback.

Launch on one platform, add others after. I launched with Claude and ChatGPT support simultaneously. The dual-platform debugging effort at launch was chaotic. Claude alone first, then ChatGPT, would have been cleaner.

TokenPulse is free and open source at github.com/anu-ship-it/TokenPulse. If you build with AI tools daily and want visibility into your usage, install it here — works on Claude, ChatGPT, Gemini, DeepSeek and Grok with no API key.

Happy to answer questions about any of the technical decisions in the comments.

Building a Full-Stack AI Application with Next.js — Architecture and Implementation

Anoop Kumar — Fri, 21 Aug 2026 15:13:55 +0000

Building an AI application is easy. Building one that handles streaming responses, manages costs, scales cleanly, and doesn't leak API keys takes more thought. This guide covers the architecture decisions and implementation patterns I have learned from building TokenPulse and several other AI-powered tools with Next.js.

The stack

Next.js 15 (App Router)
TypeScript
Tailwind CSS
Vercel (hosting + edge functions)
Resend (transactional email)

This stack handles most AI application requirements without introducing unnecessary complexity. Next.js App Router gives you server components, route handlers, and streaming out of the box.

Project structure

app/
  layout.tsx          — root layout, metadata, providers
  page.tsx            — landing page
  api/
    chat/route.ts     — AI streaming endpoint
    usage/route.ts    — usage tracking
  dashboard/
    page.tsx          — server component
    client.tsx        — client component with state
lib/
  ai.ts              — AI client configuration
  auth.ts            — session handling
  db.ts              — database client
components/
  chat/
    ChatWindow.tsx    — client component
    Message.tsx       — pure component
    StreamingText.tsx — streaming display

The key architectural decision: keep AI calls in server-side route handlers, never in client components. API keys stay server-side. The client receives streamed responses but never touches credentials.

Setting up the AI client

// lib/ai.ts
import Anthropic from '@anthropic-ai/sdk'

// Single instance — reused across requests
export const anthropic = new Anthropic({
  apiKey: process.env.ANTHROPIC_API_KEY,
})

// Type-safe model constants
export const MODELS = {
  fast:    'claude-haiku-4-5',
  default: 'claude-sonnet-4-5',
  smart:   'claude-opus-4-5',
} as const

export type ModelKey = keyof typeof MODELS

Never instantiate the client per-request. The SDK handles connection pooling internally and a single instance is more efficient.

Streaming responses with App Router

Streaming is the single most important UX improvement for AI applications. Without streaming, users stare at a blank screen for 3-10 seconds. With streaming, they see content appear token by token.

// app/api/chat/route.ts
import { anthropic, MODELS } from '@/lib/ai'

export const runtime = 'nodejs' // required for streaming

export async function POST(req: Request) {
  const { messages, model = 'default' } = await req.json()

  // Validate input
  if (!messages?.length) {
    return Response.json({ error: 'Messages required' }, { status: 400 })
  }

  // Create streaming response
  const stream = anthropic.messages.stream({
    model: MODELS[model as keyof typeof MODELS] ?? MODELS.default,
    max_tokens: 4096,
    messages,
  })

  // Convert to ReadableStream for the Response
  const readable = new ReadableStream({
    async start(controller) {
      for await (const chunk of stream) {
        if (
          chunk.type === 'content_block_delta' &&
          chunk.delta.type === 'text_delta'
        ) {
          controller.enqueue(
            new TextEncoder().encode(chunk.delta.text)
          )
        }
      }
      controller.close()
    },
  })

  return new Response(readable, {
    headers: {
      'Content-Type': 'text/plain; charset=utf-8',
      'Transfer-Encoding': 'chunked',
      'X-Content-Type-Options': 'nosniff',
    },
  })
}

Client-side streaming consumption

// components/chat/ChatWindow.tsx
'use client'
import { useState, useRef } from 'react'

export default function ChatWindow() {
  const [messages, setMessages] = useState<Message[]>([])
  const [streaming, setStreaming] = useState(false)
  const abortRef = useRef<AbortController | null>(null)

  async function sendMessage(content: string) {
    const userMessage = { role: 'user' as const, content }
    const newMessages = [...messages, userMessage]
    setMessages(newMessages)
    setStreaming(true)

    // Add empty assistant message to fill in
    setMessages(prev => [...prev, { role: 'assistant', content: '' }])

    abortRef.current = new AbortController()

    try {
      const res = await fetch('/api/chat', {
        method: 'POST',
        headers: { 'Content-Type': 'application/json' },
        body: JSON.stringify({ messages: newMessages }),
        signal: abortRef.current.signal,
      })

      if (!res.ok) throw new Error('Request failed')

      const reader = res.body!.getReader()
      const decoder = new TextDecoder()

      while (true) {
        const { done, value } = await reader.read()
        if (done) break

        const text = decoder.decode(value, { stream: true })

        // Append to last message
        setMessages(prev => {
          const updated = [...prev]
          updated[updated.length - 1] = {
            role: 'assistant',
            content: updated[updated.length - 1].content + text,
          }
          return updated
        })
      }
    } catch (err) {
      if ((err as Error).name !== 'AbortError') {
        console.error('Stream error:', err)
      }
    } finally {
      setStreaming(false)
      abortRef.current = null
    }
  }

  function stopStreaming() {
    abortRef.current?.abort()
  }

  return (
    <div>
      {messages.map((msg, i) => (
        <div key={i} className={msg.role === 'user' ? 'user' : 'assistant'}>
          {msg.content}
        </div>
      ))}
      {/* Input and controls */}
    </div>
  )
}

Rate limiting API routes

Without rate limiting, a single user can exhaust your AI credits in minutes. Implement token bucket rate limiting at the route level:

// lib/ratelimit.ts
const requests = new Map<string, { count: number; reset: number }>()

export function rateLimit(
  identifier: string,
  limit = 20,
  windowMs = 60_000
): { success: boolean; remaining: number } {
  const now = Date.now()
  const record = requests.get(identifier)

  if (!record || now > record.reset) {
    requests.set(identifier, { count: 1, reset: now + windowMs })
    return { success: true, remaining: limit - 1 }
  }

  if (record.count >= limit) {
    return { success: false, remaining: 0 }
  }

  record.count++
  return { success: true, remaining: limit - record.count }
}

// In your route handler
import { rateLimit } from '@/lib/ratelimit'

export async function POST(req: Request) {
  const ip = req.headers.get('x-forwarded-for') ?? 'unknown'
  const { success, remaining } = rateLimit(ip)

  if (!success) {
    return Response.json(
      { error: 'Rate limit exceeded' },
      { status: 429, headers: { 'Retry-After': '60' } }
    )
  }
  // ...
}

For production, use Upstash Redis with their @upstash/ratelimit package — the in-memory approach above does not work across multiple serverless function instances.

Cost tracking per request

Track costs at the API layer so you have data on actual spend:

// After streaming completes, get usage data
const finalMessage = await stream.finalMessage()
const usage = finalMessage.usage

const cost = calculateCost(
  usage.input_tokens,
  usage.output_tokens,
  model
)

// Log to your database or analytics
await logUsage({
  userId: session?.userId,
  model,
  inputTokens: usage.input_tokens,
  outputTokens: usage.output_tokens,
  cost,
  timestamp: new Date(),
})

function calculateCost(
  inputTokens: number,
  outputTokens: number,
  model: string
): number {
  const pricing: Record<string, { input: number; output: number }> = {
    'claude-haiku-4-5':  { input: 0.80,  output: 4.00  },
    'claude-sonnet-4-5': { input: 3.00,  output: 15.00 },
    'claude-opus-4-5':   { input: 15.00, output: 75.00 },
  }

  const p = pricing[model]
  if (!p) return 0

  return (inputTokens / 1_000_000) * p.input +
         (outputTokens / 1_000_000) * p.output
}

Environment variables — never expose them

Three rules:

1. Never use NEXT_PUBLIC_ prefix for AI API keys. NEXT_PUBLIC_ variables are bundled into the client JavaScript and visible to anyone who opens DevTools.

2. Always validate on startup. Add a check in your root layout or a startup file:

// lib/env.ts
const required = [
  'ANTHROPIC_API_KEY',
  'DATABASE_URL',
]

for (const key of required) {
  if (!process.env[key]) {
    throw new Error(`Missing required environment variable: ${key}`)
  }
}

3. Use .env.example committed to git, .env.local never committed. Your .gitignore should always include .env.local.

Deployment on Vercel

One vercel.json setting that matters for streaming:

{
  "functions": {
    "app/api/chat/route.ts": {
      "maxDuration": 60
    }
  }
}

Default Vercel function timeout is 10 seconds on hobby plans — not enough for long AI responses. Set maxDuration to 60 on pro plans, or implement response chunking to stay within the limit.

Also add export const runtime = 'nodejs' to any route that uses streaming. The Edge runtime does not support all Node.js APIs that the AI SDKs depend on.

The architecture decision that matters most

Keep AI logic in server-side route handlers. The temptation is to call AI APIs directly from client components using the AI SDK's client-side helpers. Resist it.

Server-side AI calls give you:

API key security
Cost logging at the server level
Rate limiting before the request reaches the AI provider
Error handling in one place The extra round-trip through your API layer is worth every one of these.

The full source code for TokenPulse's website is available at github.com/anu-ship-it/TokenPulse. If you are building AI tools and want to track token usage in real time, TokenPulse is free and works with no API key.

How Context Windows Actually Work in Large Language Models

Anoop Kumar — Fri, 21 Aug 2026 15:06:58 +0000

The context window is the most misunderstood concept in practical LLM usage. Most developers treat it as a hard boundary — "the model can see X tokens" — but the reality is more nuanced and more consequential for how you structure your prompts.

This is a technical explanation of what context windows actually are, how attention mechanisms work, and why the useful context limit is significantly smaller than the advertised number.

What a context window contains

Every time you send a message to a language model, the entire conversation history is sent with it. The model does not remember previous messages — it receives them all, in order, as a single input sequence.

That sequence contains:

Your system prompt or instructions
Every user message in the conversation
Every model response in the conversation
Any documents, code, or files you have pasted
Tool use results if the model used function calling All of this is concatenated into a single sequence of tokens and processed together. The context window is the maximum length of this sequence.

Claude Sonnet 4 processes up to 200,000 tokens at once. GPT-4o handles 128,000. Gemini 1.5 Pro handles 1,000,000. These numbers sound enormous until you realize that a single verbose debugging session can consume 30,000-50,000 tokens.

How attention works — the mechanism behind context

To understand why context windows behave the way they do, you need to understand attention — specifically, self-attention in transformers.

When a model processes your input, every token attends to every other token in the sequence. The attention score between two tokens determines how much the model's representation of token A is influenced by token B.

Conceptually:

attention_score(query_A, key_B) = dot_product(Q_A, K_B) / sqrt(d_k)

Where Q_A is the query vector for token A and K_B is the key vector for token B. Higher scores mean more influence.

The critical insight: attention is computed across the full context window, but the scores are not uniform. Recent tokens tend to have higher attention scores because the model has been trained on sequences where recent context is more relevant. Earlier tokens receive lower attention scores and contribute less to the model's representations.

This is not a bug — it is the correct inductive bias for most language tasks. In normal text, what just happened is more relevant than what happened 10,000 tokens ago.

Why quality degrades before the hard limit

The practical consequence of attention distribution is that context quality is not binary. It does not work at 100% until 200,000 tokens and then fail. It degrades gradually as the context grows.

The degradation pattern:

0-40% capacity: Full quality. The model attends effectively to everything in the context. Constraints established early are respected. Earlier decisions are remembered.

40-60% capacity: Slight degradation. The model may occasionally miss a constraint from the beginning of the conversation. Earlier architectural decisions get less weight.

60-80% capacity: Noticeable degradation. Constraints from early in long conversations get ignored. The model may contradict earlier decisions. Code quality tends to decrease.

80-100% capacity: Significant degradation. The model effectively operates on a shorter effective context than the full window. Early content is present but not meaningfully attended to.

The practical takeaway: the useful context limit for complex tasks is roughly 60% of the advertised context window.

The "lost in the middle" problem

Research on long-context LLMs has identified a specific failure mode called "lost in the middle" — models perform well on information at the beginning and end of long contexts, but poorly on information in the middle.

This has a direct implication for how you structure long prompts:

Less effective:

[System instructions - beginning]
[Background context - middle, long]
[Specific question - end]

More effective:

[System instructions - beginning]
[Specific question - near beginning]
[Relevant background context - end]

Information you want the model to attend to most strongly should be either at the very beginning or the very end of the context — not buried in the middle of a long conversation.

Practical implications for developers

Restart conversations deliberately

The most effective strategy for long-running tasks is to restart conversations when the context reaches 60% capacity.

Ask for a summary before restarting:

Summarize the key decisions, constraints, current state, and 
any important code patterns established in this conversation. 
Keep it under 300 words.

Start a new conversation with that summary. You preserve everything that matters and reset the context overhead. The summary typically condenses 30,000-50,000 tokens of conversation into 1,000-2,000 tokens.

Front-load the constraints that matter most

If you have critical constraints — "never use class components," "all functions must be async," "this code runs on Node 18" — put them at the beginning of the system prompt AND repeat them at the end of your most recent message.

The repetition sounds redundant but is effective: the constraint appears at two positions that both receive strong attention — the beginning and the most recent content.

Use separate conversations for separate problems

Every unrelated message adds tokens without adding relevant context. If you are debugging three separate functions in a single conversation, each exchange about function A is reducing the effective context available for function C.

One problem per conversation is more token-efficient and produces higher quality responses.

Monitor context usage in real time

Most platforms hide context window usage from users. TokenPulse injects a live bar above the input box on Claude, ChatGPT, Gemini, DeepSeek and Grok showing your current context window percentage. When it hits 60%, that is the signal to summarize and restart.

The 1M token context window — does it actually help?

Gemini 1.5 Pro's 1,000,000 token context window is a genuine technical achievement. But the attention degradation problem scales with context length — a 1M token context window does not give you 5x the effective context of a 200k window.

For tasks where you need to search across a very large codebase or document set, large context windows genuinely help. For typical development tasks — debugging, code review, architecture discussion — the effective range of 200k is more than sufficient when used well.

The practical benchmark: if you are regularly hitting context limits on Claude or GPT-4o, the problem is almost always conversation management, not context window size.

Summary

Context windows contain the entire conversation history sent on every request
Attention mechanisms weight recent tokens more heavily than earlier ones
Quality degrades noticeably at 60-70% capacity, not just at the hard limit
Information in the middle of long contexts gets less attention than beginning and end
Restart at 60% with a summary, not when the model tells you the conversation is too long
One problem per conversation is more efficient than long multi-topic sessions

What Are AI Tokens and Why Should Developers Care?

Anoop Kumar — Fri, 21 Aug 2026 15:04:31 +0000

If you have used Claude, ChatGPT, or any large language model API, you have encountered tokens. They show up in pricing pages, rate limit errors, and API documentation. But most explanations of tokens are either too vague ("roughly 4 characters") or too academic (subword tokenization, BPE encoding).

This article explains tokens at the level a developer actually needs — what they are, why they matter for the things you care about, and how to think about them when building with AI APIs.

What a token actually is

A token is the basic unit of text that a language model processes. It is not a character, and it is not a word — it sits somewhere in between.

Modern language models use a technique called Byte Pair Encoding (BPE) to split text into tokens. The tokenizer learns which character sequences appear frequently together and groups them into single tokens. Common words become single tokens. Rare words get split into multiple tokens. Punctuation, spaces, and special characters all become tokens too.

Some examples using GPT-4's tokenizer:

"hello"        → 1 token
"tokenization" → 3 tokens  (token + ization split further)
"ChatGPT"      → 3 tokens
" the"         → 1 token   (space included)
"2026"         → 1 token
"\n"           → 1 token

The 4-characters-per-token approximation works because English text averages roughly 4 characters per token across typical writing. Code, which has more symbols and shorter identifiers, tends to be closer to 3 characters per token. Dense technical writing can be 5 or more.

Why tokens matter for developers

1. Cost

Every major AI API charges by token. Claude Sonnet 4 costs $3.00 per million input tokens and $15.00 per million output tokens. GPT-4o costs $2.50 per million input and $10.00 per million output.

A typical back-and-forth debugging session might consume 10,000-50,000 tokens total. At Claude Sonnet pricing, that is $0.03 to $0.15 per session — which sounds trivial until you have a team of 10 developers each running 5 sessions per day.

The output token cost matters more than most developers realize. When you ask Claude to write a 500-line function, you are paying 5x more per token for the output than the input. Prompts that ask for verbose explanations cost significantly more than prompts that ask for concise code.

2. Context window limits

Every model has a context window — the maximum number of tokens it can process in a single request, including both input and output. Claude Sonnet 4 has a 200,000 token context window. GPT-4o has 128,000. Gemini 1.5 Pro has 1,000,000.

This means a 200,000 token context window is approximately:

800,000 characters of text
150,000 words
About 600 pages of a typical book
A large codebase (though not an entire monorepo) In practice, you hit the context limit much faster than these numbers suggest because:

Your system prompt consumes tokens
Every message in the conversation accumulates
Every response the model generates adds to the total
Large code pastes consume tokens proportional to their length A single paste of a 500-line file is roughly 25,000-30,000 tokens — 12-15% of Claude's context window in one shot.

3. Rate limits

AI providers enforce rate limits in tokens per unit time, not requests per unit time. Claude has a 5-hour session limit and a 7-day weekly limit, both measured in token consumption. OpenAI's API has tokens-per-minute (TPM) and tokens-per-day (TPD) limits.

This means two things developers often get wrong:

Message count is not what matters. Sending 100 short messages consumes far fewer tokens than sending 10 messages with large code pastes. Rate limits are about total token consumption, not request frequency.

Output tokens count too. If you ask a model for a very long, detailed response, you are consuming output tokens against your rate limit just as much as the input.

How to estimate tokens without calling the API

For quick estimates, the 4-characters-per-token approximation is sufficient:

function estimateTokens(text) {
  return Math.ceil(text.length / 4)
}

// Examples:
estimateTokens("Hello, world!")          // → 4
estimateTokens("function authenticate(") // → 6
estimateTokens(largeCodeFile)            // → roughly accurate ±10%

For precise counts, use the official tokenizers:

OpenAI: tiktoken library (Python and JavaScript)

import { encoding_for_model } from 'tiktoken'
const enc = encoding_for_model('gpt-4o')
const tokens = enc.encode(text).length

Anthropic: Claude's API has a count_tokens endpoint:

const response = await anthropic.messages.countTokens({
  model: 'claude-sonnet-4-5',
  messages: [{ role: 'user', content: text }]
})
console.log(response.input_tokens)

For browser extensions and client-side tools that cannot call the API, client-side estimation with the ±8% approximation is the practical approach — it is accurate enough for showing users whether they are at 20% or 80% of their context window.

Practical token optimization for developers

Be specific, not verbose. "Refactor this function to handle null values" consumes far fewer tokens than a paragraph explaining the same thing. Models understand concise instructions.

Paste targeted context, not entire files. Instead of pasting a 500-line file, paste the specific function plus 10-15 lines of surrounding context. You get equally good answers and use a fraction of the tokens.

Summarize long conversations. After a long debugging session, ask the model to summarize key decisions and findings in under 200 words, start a new conversation with that summary, and continue. You preserve context that matters and reset token overhead.

Use faster models for iteration. If you are iterating on boilerplate, formatting, or simple transformations, use Haiku or GPT-4o mini. They consume the same rate limit quota but cost 10-20x less per token. Reserve the premium models for complex reasoning.

Monitor your actual usage. Most developers significantly underestimate their token consumption because the interface hides it. Tools like TokenPulse show real-time token counts, context window percentage, and estimated cost directly in the browser — no API key required.

The key numbers to remember

Model	Context Window	Input Cost / 1M	Output Cost / 1M
Claude Sonnet 4	200k tokens	$3.00	$15.00
Claude Opus 4	200k tokens	$15.00	$75.00
GPT-4o	128k tokens	$2.50	$10.00
Gemini 2.0 Flash	1M tokens	$0.10	$0.40
DeepSeek V3	128k tokens	$0.27	$1.10

Understanding tokens is the foundation of working effectively with AI APIs. Once you internalize that everything is measured in tokens — cost, limits, context — the behavior of these systems becomes much more predictable.

I Built a Chrome Extension to Track AI Token Usage — Here's How It Works

Anoop Kumar — Fri, 21 Aug 2026 12:49:04 +0000

Six weeks ago I got cut off mid-debugging session by Claude's rate limit with no warning. Two hours of context gone. I started looking for a tool that would show me how close I was before it happened. Nothing existed that worked across more than one platform without requiring an API key.

So I built one.

TokenPulse is a Chrome extension (MV3) that injects a live token bar above the input box on Claude, ChatGPT, Gemini, DeepSeek and Grok. It tracks context window usage, rate limits, cost estimates, and daily history — all from your existing browser session, no API key required.

Here's how it works technically.

Architecture overview

Content Scripts (per platform)
        ↓
Background Service Worker
        ↓
Chrome Storage API (local)
        ↓
Popup UI
        ↓
Desktop Notifications

The extension runs a content script on each supported domain. Each script is responsible for:

Reading token usage data from that platform
Injecting the visual bar above the input box
Sending data to the background service worker via chrome.runtime.sendMessage The service worker aggregates data, writes to chrome.storage.local, checks notification thresholds, and serves data to the popup on demand.

How Claude's rate limits are read

Claude is the only platform that exposes real rate limit data through its internal API. When you use claude.ai, the browser session makes requests to a usage endpoint that returns exact utilization percentages and reset timestamps.

The content script intercepts this data by hooking into the platform's network requests using a MutationObserver to detect when Claude updates its state, then reading the cached response.

The response looks roughly like:

{
  five_hour: {
    utilization: 0.82,
    reset_at: "2026-07-15T14:14:00Z"
  },
  seven_day: {
    utilization: 0.34,
    reset_at: "2026-07-21T21:00:00Z"
  }
}

This gives exact percentages — not estimates. The popup shows these directly.

Client-side token estimation for other platforms

ChatGPT, Gemini, DeepSeek and Grok don't expose usage data the same way. For these, TokenPulse estimates token usage from the conversation DOM.

The estimation approach:

function estimateTokens(text) {
  // ~4 characters per token — standard approximation
  return Math.ceil(text.length / 4)
}

function getConversationTokens() {
  const messages = document.querySelectorAll('[data-message-author-role]')
  let total = 0
  messages.forEach(msg => {
    total += estimateTokens(msg.textContent || '')
  })
  return total
}

Accuracy is approximately ±8% — sufficient for knowing whether you're at 20% or 80% of your context window.

Injecting the token bar

The bar injection uses a MutationObserver to watch for the input box appearing in the DOM, then inserts a container element immediately above it.

const observer = new MutationObserver(() => {
  const inputBox = document.querySelector(PLATFORM_INPUT_SELECTOR)
  if (inputBox && !document.getElementById('tp-bar')) {
    injectBar(inputBox)
  }
})

observer.observe(document.body, {
  childList: true,
  subtree: true
})

The bar is a thin div with a gradient fill that updates via CSS transition whenever new token data arrives:

function updateBar(pct) {
  const fill = document.getElementById('tp-bar-fill')
  if (fill) fill.style.width = `${pct}%`
}

The CSS transition handles the smooth animation — no JavaScript animation loops needed.

Cost estimation

Cost is estimated by multiplying token count by current model pricing:

const PRICING = {
  'claude-sonnet-4': { input: 3.00, output: 15.00 },  // per 1M tokens
  'claude-opus-4':   { input: 15.00, output: 75.00 },
  'gpt-4o':          { input: 2.50, output: 10.00 },
  'gemini-2.0-flash':{ input: 0.10, output: 0.40 },
  'deepseek-v3':     { input: 0.27, output: 1.10 },
}

function estimateCost(tokens, model) {
  const price = PRICING[model]
  if (!price) return 0
  return (tokens / 1_000_000) * price.input
}

Model detection reads the platform's model selector from the DOM.

The notification system

Notifications fire at configurable thresholds (default: 75%, 90%, 100%) using Chrome's notifications API. The threshold system is designed to fire once per crossing and reset when usage drops:

async function shouldNotify(key, currentPct, settings) {
  const thresholds = [75, 90, 100].filter(t => settings[`notify_${t}`])
  const crossed = thresholds.filter(t => currentPct >= t).pop() || 0
  const lastNotified = await getLastNotified()
  const last = lastNotified[key] || 0

  if (crossed <= 0 || crossed <= last) return null

  lastNotified[key] = crossed
  await saveLastNotified(lastNotified)
  return crossed
}

This prevents notification spam — you get one notification when you cross 75%, another when you cross 90%, and they reset when your usage drops back below the threshold.

MV3 service worker constraints

Chrome's Manifest V3 requirement means the background script is a service worker — not a persistent background page. Service workers can be killed by Chrome at any time, which creates two constraints:

No persistent state in memory. Everything goes through chrome.storage.local. The service worker reads from storage on every message handler invocation rather than keeping state in variables.

Async message handling. The chrome.runtime.onMessage listener must handle the async/sync distinction carefully. Fire-and-forget messages (like saving usage data) return false immediately. Messages that need a response return true and call sendResponse after the async work completes.

chrome.runtime.onMessage.addListener((msg, sender, sendResponse) => {
  if (msg.type === 'SAVE_USAGE') {
    // Fire and forget — no response needed
    Storage.saveUsage(msg.usage)
    return false
  }

  if (msg.type === 'GET_ALL_DATA') {
    // Async response needed
    Promise.all([
      Storage.getUsage(),
      Storage.getHistory(),
      Storage.getSettings(),
    ]).then(([usage, history, settings]) => {
      sendResponse({ usage, history, settings })
    })
    return true // keep channel open
  }
})

What I learned

MV3 CSP is strict. No inline scripts, no eval, no remote code execution. Every event handler must be attached via addEventListener — no onclick attributes. This caught me early and took a full debugging session to unpack.

Platform DOM structures change without notice. Claude, ChatGPT and Gemini update their frontends regularly. Selectors that work today break in a week. The solution is multiple fallback selectors and defensive ?. access throughout.

Service worker timing is unpredictable. Chrome can kill and restart the service worker between messages. Code that assumes the worker is alive from a previous message will fail intermittently and be very hard to debug.

Local storage is fast enough. I was worried chrome.storage.local would be too slow for real-time updates. In practice, reads complete in under 5ms and the popup feels instant.

The extension is open source at github.com/anu-ship-it/TokenPulse. If you're building a Chrome extension that reads from AI platforms, feel free to look at how the content scripts are structured.

Install TokenPulse free — works on Claude, ChatGPT, Gemini, DeepSeek and Grok with no API key.

How Claude's rate limits actually work — and how I track them in real time

Anoop Kumar — Sun, 09 Aug 2026 22:09:34 +0000

I was two hours into a debugging session with Claude when it just stopped.

No warning. No countdown. No indication I was close. Just a message telling me I'd reached my usage limit.

Two hours of context — gone. I had to start over.

That was the moment I started actually trying to understand how Claude's rate limits work. What I found surprised me, and I haven't seen it explained clearly anywhere.

Claude has two completely separate rate limits

Most developers assume there's one limit. There are two, and they operate independently:

The 5-hour session limit tracks your message volume within any rolling 5-hour window. This is the one that catches most developers off guard because it resets on a rolling basis — not at a fixed time.

The 7-day weekly limit tracks cumulative usage across a rolling 7-day period. This one resets 7 days after your first message in the window — not on Sunday, not at midnight.

You can be at 0% on the weekly limit and 90% on the session limit. You can max both on the same day if you work in concentrated bursts.

Why "rolling" matters more than you think

The rolling reset is the part that trips people up most.

If you sent your first message at 9:14am on Tuesday, your 5-hour window resets at 2:14pm — not at 10am, not at noon, not at midnight. If you send your first message on Monday at 11pm, your 7-day limit resets the following Monday at 11pm.

This means the mental model of "it resets Sunday night" or "it resets every morning" is wrong for most users. The reset time is personal to your usage pattern.

The practical consequence: you can't plan around a fixed reset schedule. You need to know the actual countdown.

What Claude exposes through its internal API

This is the part most developers don't know exists.

When you use Claude through the browser at claude.ai, the interface makes requests to an internal usage endpoint that returns your actual utilization data — not estimates, not approximations, but the exact numbers Claude uses to decide when to cut you off:

json

{
  "five_hour": {
    "utilization": 0.82,
    "reset_at": "2026-07-15T14:14:00Z"
  },
  "seven_day": {
    "utilization": 0.34,
    "reset_at": "2026-07-21T21:00:00Z"
  }
}

That utilization field is a percentage — 0.82 means you're at 82% of your 5-hour limit. The reset_at field is the exact UTC timestamp when that window resets.

This data is available to anyone using Claude through the browser. You don't need an API key. You don't need special access. Your existing browser session already has permission to read it — because Claude itself uses it to show rate limit warnings.

Why Claude's rate limits are measured in tokens, not messages

The limits aren't per-message — they're per-token. A short message consumes far fewer tokens than a long one with a code paste.

A 500-line file is approximately 25,000-30,000 tokens. A typical detailed response might be 1,000-2,000 tokens. A debugging session where you paste large code snippets can burn through your 5-hour window in under an hour.

The models also matter. Claude Opus is significantly more expensive per token than Claude Sonnet or Haiku, which affects how quickly you consume your quota.

The degradation problem — quality drops before the limit

Here's something that isn't documented anywhere: Claude's response quality degrades before you hit the hard limit.

The mechanism is attention. Large language models weight recent tokens more heavily than distant ones. In a very long conversation, Claude can technically "see" everything you've written, but its effective attention to content from early in the conversation weakens as more content is added.

Practically, this means you'll notice Claude starting to ignore constraints or forget decisions established earlier in the conversation — usually around 60-70% of the context window. By the time you're at 80%, you're often getting meaningfully worse answers than you would in a fresh conversation with a good summary.

The right time to restart is at 60% of the context window — not when Claude tells you the conversation is too long.

How I track this in real time

After getting cut off enough times, I built a Chrome extension that reads this data directly from Claude's internal API and shows it in the browser.

TokenPulse injects a slim bar above Claude's input box showing:

1. Context window percentage in real time
2. 5-hour session utilization (exact percentage from Claude's API)
3. 7-day weekly utilization (same)
4. Countdown to each reset
5. Estimated cost per conversation, per day, per week

It also works on ChatGPT, Gemini, DeepSeek and Grok — though those platforms don't expose rate limit data the same way Claude does, so those use client-side estimation.

No API key required. No account. It reads your existing browser session — the same one Claude already uses to pull this data for its own interface.

Practical strategies based on this understanding

Check both limits before starting heavy sessions. If you're at 70% of your 5-hour window, either work quickly or wait for the reset. Starting a 2-hour debugging session at 70% almost guarantees getting cut off.

Start new conversations for each distinct problem. Every message accumulates context. If you're debugging three separate functions, three conversations is more efficient than one long one.

Restart at 60% context, not when Claude tells you to. By the time Claude warns you, quality has already degraded. Summarize at 60%, start fresh, paste the summary.

Use Haiku and Sonnet for iteration, Opus for final decisions. All models consume the same rate limit quota. Matching model capability to task complexity extends how long you can work before hitting limits.

Time heavy sessions around your actual reset time. Check your reset countdown before starting an intensive session. If your 5-hour window resets in 20 minutes, waiting is often worth it.

The extension is free and open source if you want to look at how it reads Claude's usage data:

Chrome Web Store: token-pulse.in
GitHub: github.com/anu-ship-it/TokenPulse

Happy to answer questions about how the rate limit detection works in the comments.