DEV Community: James Anderson

Inside the Tokenizer: Why the Same Prompt Costs Different Amounts on Every Model

James Anderson — Tue, 18 Aug 2026 11:06:50 +0000

If you build with LLMs, you pay by the token. Not by the word, not by the character — the token. And yet most of us treat the tokenizer as a black box: text goes in, a number comes out, the bill arrives.

That black box is worth opening. Once you understand how tokenization works, a lot of otherwise-mysterious LLM behavior starts to make sense: why the same sentence costs 3 tokens on Claude and 4 on GPT, why your Spanish chatbot costs more than the English one, why models are weirdly bad at arithmetic, why some prompt styles quietly burn your budget — and, crucially, how to actually calculate what a feature will cost before you ship it.

Let's open it.

What a Token Actually Is

A token is not a word and not a character. It's a chunk of text — usually a subword — that the model treats as a single unit. Before a model reasons about anything, your text is split into these chunks, and each chunk is mapped to an integer ID. The model only ever sees those integers.

A rough rule of thumb for English:

1 token ≈ 4 characters ≈ 0.75 words

So ~100 tokens is about 75 words, and a page of English prose (~500 words) is roughly 650-750 tokens.

But that's just an average for English. The real count depends entirely on how the text gets chunked — and that's decided by an algorithm called BPE.

How Tokenizers Are Built: BPE in Plain Terms

Nearly every major model today — GPT, Claude, Gemini, Llama, Mistral — uses some flavor of Byte-Pair Encoding (BPE) or a close relative.

BPE started life as a data-compression trick in 1994 and was adapted for language models in 2016. The idea is genuinely simple:

Start with text broken into the smallest units (bytes/characters).
Count every adjacent pair of units.
Merge the single most frequent pair into a new combined unit.
Repeat until you hit a target vocabulary size.

Each merge gets recorded, in order, into a permanent list. At inference time, the tokenizer just replays those merge rules deterministically on your text.

The key consequence: frequency during training decides everything. Common words become single tokens; rare words get split into pieces. This is why the is one token but tokenization might be two or three (token + ization), and why the splits don't follow English grammar — they follow whatever was statistically common in the training text.

Modern tokenizers also start from raw bytes (the 256 possible byte values) rather than characters. That's what lets them handle anything — emoji, Chinese, symbols, typos — without ever hitting an "unknown word." Worst case, a weird character just falls back to several byte-level tokens.

One design tension worth knowing: a bigger vocabulary means fewer tokens per sentence (cheaper, shorter sequences) but a larger embedding table and more memory. GPT-2 learned ~50,000 merges; models like GPT-4o's o200k_base use roughly 200,000. That jump is a big part of why newer models are more token-efficient per word.

Count It Yourself: `tiktoken`

For OpenAI models, the tokenizer is open source, so you can get exact counts locally:

# pip install tiktoken
import tiktoken

enc = tiktoken.get_encoding("o200k_base")  # GPT-4o / newer

samples = [
    "Hello, world!",
    "tokenization",
    "The quick brown fox jumps over the lazy dog.",
    "12345678901234567890",
    '{"user_id": 42, "name": "Alice", "active": true}',
]

for s in samples:
    ids = enc.encode(s)
    print(f"{len(ids):>3} tokens | {s}")

Example output:

  4 tokens | Hello, world!
  2 tokens | tokenization
 10 tokens | The quick brown fox jumps over the lazy dog.
  ...       (long digit runs fragment into several tokens)
 ...        (the JSON spends tokens on braces, quotes, and keys)

For Claude, the tokenizer isn't published — use Anthropic's token-counting endpoint (POST /v1/messages/count_tokens), which accepts the same shape as a real request and returns the input token total. It's free to call. For Gemini, use Google's countTokens API. Don't cross-apply one model's count to another — they diverge.

Why the Same Text Costs Different Amounts on Different Models

Send Hello, world! to GPT and you might pay 4 tokens; send it to Claude and you might pay 3. Same text, different integers out. Why?

Because each provider trained its own tokenizer on its own data, with its own vocabulary size, merge tables, and rules for whitespace and non-Latin scripts. They share a family resemblance but diverge in the details:

GPT (OpenAI) — tiktoken, open source. cl100k_base (~100k vocab) on older models; o200k_base (~200k vocab) on newer ones. Exact local counts.
Claude (Anthropic) — proprietary BPE, not published. Use the count_tokens endpoint. Note: newer Claude models use a tokenizer that can produce ~30% more tokens for the same text than older ones — which matters when you compare sticker prices.
Gemini (Google) — SentencePiece-based, not published. Use countTokens.

Takeaway: a token count measured on one model does not transfer to another — not even between generations of the same model. Budget with the exact model you'll deploy.

What Makes Token Usage Worse (and Better)

Token count isn't just "length of text." Several factors push it up or down — and these are things you can actually control.

Inflates your token count 📈

Numbers. Digits tokenize unpredictably. 127 might be one token; 677 can split into two; long numbers fragment into several. (Also a big reason LLMs are shaky at arithmetic.)
Whitespace & indentation. Spaces, tabs, and newlines are tokens too. Deeply indented code spends real tokens on whitespace.
Rare / made-up words. Jargon, UUIDs, hashes, and base64 fragment into many small pieces.
Heavy formatting. JSON — with its braces, quotes, and repeated keys — is heavier than leaner formats. This is why YAML often tokenizes cheaper than JSON for the same data.
Template overhead. Every request is wrapped in role markers and template tokens. Same overhead in every language, so on short messages it's a proportionally bigger tax.

Reduces your token count 📉

Concise, plain prose over verbose phrasing.
Common words over rare synonyms.
Removing repeated context — cache it or reference it instead of re-sending.
Leaner data formats where structure allows.
Capping output length (see cost math below — output is where the money goes).

This doesn't mean mangle your prompts into unreadable shorthand. It means the obvious wins — don't re-send a giant system prompt on every call, don't pad with filler, cap max_tokens — are real money.

The Language Tax: Why Non-English Costs 2-15x More

The biggest and least-known factor. Tokenizers are trained on English-heavy text, so they learn big efficient tokens for English and few for everything else. This is measured as fertility — tokens per word. English sits ~1.2-1.4; other languages run far higher.

Language	English-relative cost	Why
English	~1.2-1.3×	Baseline (best case)
Spanish / French	~1.5-2×	Accents + morphology
Hindi	~1.6-2.7×	Non-Latin script
Chinese / Japanese / Korean	~2-3×+	CJK fragments heavily
Arabic	~3-4×	Script + morphology
Turkish / Finnish	~2-3×	Agglutinative words
Tamil / Telugu / Malayalam	up to ~12-16×	Worst-hit; some tokenizers explode these

(Figures are approximate and vary by tokenizer — measure your own.)

Two drivers: script/encoding (English is 1 UTF-8 byte per char thanks to ASCII; other scripts need 2-4 bytes) and word frequency (underrepresented languages never earned efficient tokens).

The consequences are structural — you can't prompt them away:

Cost: 3x the tokens ≈ 3x the bill for the same meaning.
Context window: the same 200k window holds far less non-English content.
Latency: more tokens = slower responses.

If your usage is global, estimate cost and context per language, not once in English. The English number is the best case, not the average.

Now the Money: How to Actually Calculate Cost

APIs price input and output tokens separately, and output is almost always far more expensive. The core formula:

cost = (input_tokens  / 1,000,000 × input_price)
     + (output_tokens / 1,000,000 × output_price)

Representative flagship prices (per 1M tokens)

⚠️ Prices change constantly and vary by exact model/tier — treat these as an illustrative August 2026 snapshot, and always confirm on the provider's current pricing page before budgeting. Some models also double rates past ~200k-token context, and newer Claude tokenizers emit ~30% more tokens for the same text.

Model (illustrative)	Input / 1M	Output / 1M
Claude Haiku (small)	~$1	~$5
Claude Sonnet (mid)	~$2-3	~$10-15
Claude Opus (flagship)	~$5	~$25
GPT-5-class (flagship)	~$1.75-5	~$14-30
GPT mini (small)	~$0.25	~$2
Gemini Pro (flagship)	~$2	~$12
Gemini Flash (small)	~$0.10-0.50	~$0.40-3

Worked example: one chatbot request

Say a single request has 1,500 input tokens (system prompt + history + user message) and 500 output tokens, on a mid-tier model at $3 / $15 per 1M:

input  = 1,500  / 1,000,000 × $3   = $0.0045
output =   500  / 1,000,000 × $15  = $0.0075
------------------------------------------------
total per request                  ≈ $0.012

Just over one cent per request. Feels trivial — until you scale.

Scaling it out (same request, 100,000/day)

per day   = 100,000 × $0.012        = $1,200
per month = $1,200 × 30             ≈ $36,000

That "one cent" is now a $36k/month line item. This is why the math matters before launch, not after the invoice.

The output tax, made concrete

Notice that in the example above, output cost more than input despite being one-third the tokens ($0.0075 vs $0.0045). If you let max_tokens default to a huge buffer and the model rambles, output balloons. Capping output length is often the single highest-leverage cost lever you have.

The language tax, in dollars

Take that same request, but the user writes in a language with 3x fertility. Input and output token counts roughly triple:

input  = 4,500  / 1,000,000 × $3   = $0.0135
output = 1,500  / 1,000,000 × $15  = $0.0225
------------------------------------------------
total per request                  ≈ $0.036   (3× the English cost)

Same feature, same user intent, triple the bill — purely from tokenization.

Blended rate: comparing providers honestly

Input and output prices differ, so don't compare "input price" in your head. Compute a blended rate using your real traffic mix. For an 80% input / 20% output workload:

blended = 0.8 × input_price + 0.2 × output_price

On a $3/$15 model: 0.8×3 + 0.2×15 = 2.4 + 3.0 = $5.40 per 1M blended. Run that for each candidate model with your ratio — it often reorders the "cheapest" ranking versus headline input prices.

Cost Levers That Actually Work

In rough order of impact:

Prompt caching. Cached input tokens often cost ~10% of the normal rate. If you re-send a big fixed system prompt every call, this can cut costs up to ~90% on that portion.
Route by difficulty. Send trivial requests (greetings, formatting, lookups) to a small/cheap model; reserve flagships for hard reasoning. A tiny classifier in front of your router commonly cuts mixed-workload cost 60-80%.
Cap max_tokens aggressively. The default output buffer is usually far bigger than you need, and output is the pricey side.
Batch API for non-realtime work. Typically ~50% off for jobs that don't need instant responses.
Trim and cache context. Don't re-send history or documents you can reference or summarize.
Budget per language. Weight your cost model by the token cost of each language you serve.

Practical Takeaways

You pay per token, not per word — and the exchange rate shifts with language, format, and content.
Count with the real tokenizer: tiktoken (OpenAI, exact/local/free), count_tokens (Claude), countTokens (Gemini). Never cross-apply.
Count the final request body you actually send — system prompt, tool schemas, history, all of it.
Output tokens usually dominate cost. Cap them.
Non-English can cost 2-15x more. Budget per language.
Compute a blended rate with your real input/output mix before picking a model.
Re-verify prices — they change often and can double past long-context thresholds.

The tokenizer isn't an implementation footnote. It's the layer where the economics of your app are quietly set — the interface between human language and the model's math, and the exact place your bill is decided. Understanding it turns a mysterious invoice into something you can reason about, forecast, and control.

Have you hit a surprising token bill or a weird tokenization bug in production? Drop the story in the comments — I collect these.

A Claude Code skill fixed my app's UI — here's what broke and how to use it yourself

James Anderson — Mon, 17 Aug 2026 12:03:47 +0000

My app worked. It just looked like three different apps stitched together.

BrandMeld (Flutter, AI brand generator) had grown screen by screen: some had plain AppBars, some had gradient heroes, settings was a gray ListTile dump, spacing was random, and the "create brand" form asked for 8 fields before you could do anything. Every screen was fine in isolation and inconsistent as a whole.

Instead of hand-fixing 14 screens, I installed a Claude Code skill built for mobile UI/UX and pointed it at them. This post is the concrete before/after — the actual problems it fixed — plus how you install and use it.

The problems it fixed (real, not vibes)

1. No shared header language. Every screen reinvented its top bar. The skill replaced them with one full-width gradient header pattern (back button + actions merged in), so the app reads as one product.

2. A settings screen that was a wall of gray rows. Became iOS-style grouped cards with tinted icon badges — scannable in a glance.

3. An 8-field create form. Reworked to name-only required, with the rest behind an "Add more details" expander. Then it added a smart default I didn't ask for: pick a preset industry → target audience + keywords auto-fill (without overwriting anything you typed). Cognitive load dropped from "fill a form" to "type a name."

4. A dead-end paywall. "Your trial ended." → a card that lists what you lose access to (names, palettes, logos, brand guides, saved brands) with lock icons. Loss aversion instead of a shrug.

5. Actual rendering bugs — caught from screenshots. This surprised me. I'd send a screenshot and it diagnosed things code review misses:

a selected card's border clipped at the top corners (border + Clip.antiAlias on the same container),
an avatar hidden behind a floating stats card (header too short, card overlapped it),
weird blur blobs behind the header (translucent circles rendering as hard-edged shapes).

The screenshot is the debugger.

Why a "skill" and not just "make it nicer"

"Redesign this" gives you noise. The skill encodes a rubric the model follows every time:

8-pt spacing grid
60/30/10 color system
named patterns (floating stat strips, grouped cards, gradient headers)

That's the difference between one-off prettiness and a consistent design system.

What you do with it

Install it

A skill is just a folder with a SKILL.md. Drop it in:

Project: .claude/skills/<name>/SKILL.md
Personal (all projects): ~/.claude/skills/<name>/SKILL.md

Clone the repo into that folder, restart Claude Code, done.

Use it

Point it at a screen:

"Redesign the settings screen. Apply the design principles."

Then iterate with screenshots:

"The avatar's hidden — fix it."

It keeps your logic/routing and rewrites the layout. You review UI, not re-test features.

Or build your own

The real lesson: you can package any discipline into a skill. A SKILL.md is tiny:


markdown
---
name: my-skill
description: "What it does AND when to use it. This is how Claude decides to load it."
---

# Instructions
The rules/steps Claude should follow when this runs.

Encode your conventions — API patterns, test style, commit format — once, and get consistent output forever.

Takeaway

The win isn't "AI writes UI." It's that a rubric-in-a-file tura cohesive one in an afternoon, and caught rendering bugs fromscreenshots along the way.

Skill (clone + drop into .claude/skills/): https://github.com/ceorkm/mobile-app-ui-design

If you try it, send me a before/after — those are the fun ones.

How PDF & Document Parsers Actually Work Under the Hood

James Anderson — Sun, 16 Aug 2026 10:52:46 +0000

To the average user, opening a PDF feels no different than viewing a webpage or reading a Word document. You see headers, paragraphs, multi-column articles, and neatly bordered tables. To a developer trying to extract structured data from that same PDF, it feels like staring into an abyss. That is because a PDF has no concept of a paragraph, a column, or a table. In fact, a PDF doesn't even store text in reading order. A PDF is essentially a set of vector drawing instructions for a printer canvas. Here is a look under the hood at how document parsers turn raw, chaotic drawing streams into structured, machine-readable data.
1. The Core Illusion: PDFs Are Canvas Commands
When a word processor exports a document to PDF, it discards semantic document structure (DOM) in favor of visual position.

Instead of saving <h2>Title</h2> or <table>, the PDF encoder generates low-level drawing operators:

BT                   % Begin Text object
/F1 12 Tf            % Set font F1 at size 12pt
72 712 Td            % Move cursor to coordinate (x: 72pt, y: 712pt)
(Hello ) Tj          % Draw character stream "Hello "
12.5 0 Td            % Move cursor right by 12.5pt
(World) Tj           % Draw character stream "World"
ET                   % End Text object

The Spatial Order Nightmare
Notice what is missing? No space characters, no paragraph breaks, and no reading order. Because glyphs are placed at hardcoded $(x, y)$ coordinate points on a Cartesian canvas, a PDF generator might draw the footer first, the header second, the right-hand sidebar third, and the main content paragraph last. If you naive-dump the raw text stream, your output becomes a scrambled mess of interleaved columns.
Phase 1: Decoding Binary Anatomy (pypdf)
This snippet inspects low-level PDF primitives, trailer metadata, decompressed byte streams, and font mappings.

reader = PdfReader(pdf_path)

What happens under the hood: PdfReader opens the binary file and instantly seeks to the very end of the file. It reads the last 1,024 bytes to locate the startxref keyword. From there, it parses the XRef (Cross-Reference) Table, which builds an internal index mapping Object IDs (e.g., 12 0 R) to byte offsets in the file.

trailer = reader.trailer
print(f"Root Catalog keys: {list(trailer['/Root'].keys())}")

reader.trailer: Returns the PDF Trailer Dictionary, which contains top-level pointers:

Points to the Document Catalog object (the root node of the PDF's structural tree).
Points to metadata (Author, CreationDate, Producer).
Total number of entries in the XRef table.

trailer['/Root'].keys(): Resolves the indirect object reference for the Root catalog and exposes top-level keys like /Pages (the root of the page tree), /Names, or /AcroForm (interactive form fields).

page = reader.pages[0]
contents = page.get_contents()

page.get_contents(): A page object delegates its visual drawing commands to a stream object (or an array of stream objects) specified under the /Contents key.

raw_stream_bytes = contents.get_data()
raw_operators = raw_stream_bytes.decode("latin1", errors="ignore")

contents.get_data(): In raw PDF files, stream data is compressed using the FlateDecode filter (standard zlib compression). get_data() inflates the compressed stream back into uncompressed PDF drawing operators in RAM.
decode("latin1"): PDF stream strings do not use UTF-8 by default; they use 8-bit string encodings (often WinAnsiEncoding or PDFDocEncoding). Using latin1 prevents Python decoding exceptions while preserving byte values.

resources = page["/Resources"]
fonts = resources["/Font"].get_object()

/Resources: Every page has a resource dictionary containing assets required to draw it, including /Font, /XObject (embedded images or sub-canvases), and /ColorSpace.
/Font Lookup: Maps font alias identifiers used inside the drawing stream (like /F1 or /F2) to indirect Font dictionary objects containing /BaseFont names and embedded /ToUnicode CMaps (Character Maps).

Phase 2: Spatial Geometry & Layout Analysis (PyMuPDF / fitz)
This snippet extracts coordinate-level word tokens and groups them spatially to solve multi-column reading order problems.

doc = fitz.open(pdf_path)
page = doc[0]
words = page.get_text("words")

fitz.open(): Uses MuPDF's C engine to parse the document structure into memory.
page.get_text("words"): Executes the page stream and returns a list of 8-element tuples for every detected word:
```
text = (x0,y0, x1, y1,"wurd_text", block_no,word_no,line_no)
```
(x0,y0 ) represents the Top-Left coordinate.
(x1, y1) represents the Bottom-Right coordinate.
Note: In PDF geometry, (0,0) is at the top-left corner of the page, measured in PDF points (1 pt= 1/72 inch)

COLUMN_THRESHOLD_PTS = 250
left_column_words = [w for w in words if w[0] < COLUMN_THRESHOLD_PTS]
right_column_words = [w for w in words if w[0] >= COLUMN_THRESHOLD_PTS]

Spatial Partitioning: Instead of relying on the raw PDF stream order, this splits words into two distinct groups based on their left coordinate ($x_0$). Words starting before 250 pt belong to Column 1; words starting at or after 250 pt belong to Column 2.

sorted_left = sorted(left_column_words, key=lambda w: (round(w[1], -1), w[0]))

w[1] is the vertical top position (y0).
round(w[1], -1) rounds the Y-coordinate to the nearest 10 points (e.g., 102.3 to 100). This creates vertical alignment buckets (lines). Without this rounding, tiny sub-pixel baseline variations (y0 = 100.1 vs y0 = 100.3) would cause words on the same line to be treated as different vertical rows.
w0 is the secondary tie-breaker. Once words are placed into the same line bucket, they are ordered strictly from left to right.

Phase 3: Table Extraction (pdfplumber)

This snippet demonstrates two contrasting programmatic techniques for extracting tabular data from PDFs: vector parsing vs. whitespace channel projection.

with pdfplumber.open(pdf_path) as pdf:
    page = pdf.pages[0]

pdfplumber Architecture: Built on top of pdfminer.six, pdfplumber builds a rich spatial engine by turning every character, line, rectangle, and path into a Python dictionary with explicit geometry.

Strategy 1: Lattice (Vector Path Analysis)

lattice_settings = {
    "vertical_strategy": "lines",
    "horizontal_strategy": "lines",
    "snap_tolerance": 3,
}

"vertical_strategy": "lines" & "horizontal_strategy": "lines": Directs pdfplumber to ignore text initially and look exclusively at vector graphics paths (m for moveto, l for lineto, re for rectangle) drawn on the page canvas.
snap_tolerance: 3: PDF line paths often do not connect perfectly at corners due to rendering rounding errors. A snap_tolerance of 3 points automatically snaps endpoints together if they are within 3 points of each other, forming closed grid cells.
Intersection Algorithm: The parser finds all points where horizontal vector lines intersect vertical vector lines. These intersection points form bounding boxes (cells). Text character coordinates are then mapped inside these boxes.

Strategy 2: Stream (Borderless Table Whitespace Projection)

stream_settings = {
    "vertical_strategy": "text",
    "horizontal_strategy": "text",
    "min_words_vertical": 3,
}

"vertical_strategy": "text": Used when there are no vector lines (borderless tables). The engine projects the x0 and x1 coordinates of all text elements onto a horizontal axis histogram. Continuous vertical "blank spaces" (channels without text) are inferred as column dividers.min_words_vertical: 3: Specifies that at least 3 vertically aligned text elements must share the same horizontal alignment to establish a valid column boundary, preventing single stray words from triggering false column divisions.

Phase 4: Scanned PDFs & OCR Pipeline (pytesseract + PyMuPDF)

This snippet bridges raster graphics rendering and optical character recognition for documents that lack text vectors entirely.

pix = page.get_pixmap(dpi=300)
img_bytes = pix.tobytes("png")
image = Image.open(io.BytesIO(img_bytes))

page.get_pixmap(dpi=300): Rasterizes the vector PDF page object into an uncompressed pixel matrix (bitmap) in memory.
Why 300 DPI matters: Standard screen resolution (72 DPI) causes character edges to become pixelated, severely degrading OCR accuracy. 300 DPI provides the ideal pixel density for OCR algorithms to perform edge detection.
io.BytesIO(img_bytes): Converts raw in-memory PNG byte buffers into a file-like stream object so Pillow (PIL) can open it without writing an intermediate file to disk.

ocr_data = pytesseract.image_to_data(image, output_type=pytesseract.Output.DICT)

image_to_data(): Rather than returning plain string output, this function calls Tesseract's C++ library to output granular spatial data for every detected text bounding box.

Returned Dictionary Structure:

ocr_data['text']: List of detected strings.
ocr_data['left'], ['top']: (x, y) pixel coordinates of the top-left bounding box corner.
ocr_data['width'], ['height']: Pixel dimensions of the text block.
ocr_data['conf']: Confidence score (0 to 100) returned by Tesseract's LSTM neural network model for each token prediction.

if conf > 60 and len(text) > 0:
    x, y, w, h = (
        ocr_data['left'][i],
        ocr_data['top'][i],
        ocr_data['width'][i],
        ocr_data['height'][i]
    )

Filtering and Bounding Box Normalization:

conf > 60: Filters out low-confidence OCR predictions, noise artifacts, scan smudges, or faint line marks that are falsely detected as text.
Converts the raw pixel coordinates into standard bounding box representation (x0, y0, x1, y1) where x1 = x + w and y1 = y + h.

How I Stopped Burning Out as a Developer

James Anderson — Mon, 08 Jun 2026 10:42:11 +0000

For a long time, I thought burnout was something that happened to other people. People who didn't love coding as much as I did. People who weren't committed enough to push through.
Then I became one of those people.
Not in a dramatic way. There was no breakdown, no quitting on the spot. It was slower and quieter than that — a gradual draining of the thing that used to make me excited to open my editor. Code that used to feel like play started to feel like lifting weights I never got to put down.
This is what I learned getting out of it, and the warning signs I wish I'd taken seriously sooner.
The Signs I Ignored
Burnout didn't announce itself. It crept in through small things:

I stopped wanting to build side projects, then stopped wanting to read about tech at all.
Small bugs that used to be fun puzzles started to feel infuriating.
I was "working" more hours but shipping less.
I felt tired after a full night's sleep.
I was irritable about things that didn't deserve it.
I kept telling myself I just needed to push through one more sprint.

That last one was the real trap. "Just push through" is great advice for a hard afternoon. It's terrible advice for a pattern that's lasted months.
What Actually Helped
Here's what made a real difference. None of it was a magic fix — it was a stack of small changes that added up.

I separated "tired" from "done" I used to treat finishing my work and stopping work as the same event. They're not. There's always more to do. The codebase is never finished. If I waited until everything was done to stop, I'd never stop. So I started defining the end of my workday by time and energy, not by an empty task list. When the day's planned work was done — or when my focus was clearly gone — I stopped. The remaining tasks would still be there tomorrow, and I'd handle them better with a working brain.
I made real boundaries, not vague intentions "I'll try to log off earlier" never worked. Specific rules did:

No code editor open after a set time in the evening.
Notifications off outside work hours — actually off, not "I'll just glance."
One full day a week with zero programming, including side projects.

The day off was the hardest and the most important. Rest isn't a reward you earn after finishing everything. It's part of how the work gets sustainable.

I stopped tying my identity entirely to my output This one was subtle but huge. When my entire sense of worth was "how much I shipped," every slow day felt like a personal failure. That's an exhausting way to live. I'm a developer, but I'm also other things. Building a life outside of code — hobbies, people, time that has nothing to do with a screen — gave me somewhere to stand when work was hard. It also, ironically, made me better at work.
I asked for help instead of grinding in silence A lot of my exhaustion came from carrying things alone: an overloaded sprint, an unrealistic deadline, a problem I was too stubborn to ask about. Saying "this scope isn't realistic" or "I'm stuck, can we pair on this?" felt like admitting weakness. It wasn't. It was the thing that lightened the load. If your workload is genuinely unsustainable, no personal productivity hack will fix that. Sometimes the honest conversation with a manager is the real solution.
I protected the parts of coding I actually loved Burnout had made all coding feel the same shade of gray. Part of recovery was deliberately reconnecting with the parts that drew me in originally — a tiny project with no deadline, a language I was curious about, building something silly just because. No pressure to ship, no audience, no metrics. Just the thing that made it fun in the first place. What I'd Tell Past Me If I could go back, I'd say this:

Burnout is information, not weakness. It's your capacity telling you something is off. Listen earlier.
Rest is part of the work, not a break from it. You don't do good engineering on an empty tank.
"Push through" has a shelf life. Useful for a day. Dangerous for a season.
The work will still be there tomorrow. It genuinely always is.

A Note on the Serious End of This
What I've described is the everyday, manageable kind of burnout — the kind better habits and boundaries can address. But burnout can also blur into something heavier: persistent hopelessness, depression, the sense that nothing will get better.
If you're somewhere in that territory, the advice in this post isn't enough, and that's not a personal failing. Talking to a doctor, a therapist, or someone you trust is a reasonable and worthwhile step. You don't have to debug that one alone.
Closing
I still love building software. Maybe more now than before, because I'm not running on fumes to do it. The difference wasn't working harder or caring more — it was building a way of working I could actually sustain.
If you're reading this while feeling some of the signs I described: you're not lazy, and you're not failing. You're a person with limits, like everyone else. The sooner you work with those limits instead of against them, the longer you get to keep doing the thing you love.

What's helped you avoid or recover from burnout? I'd love to hear what worked for you in the comments.

How AI Is Reshaping Software Development (and Where It's Heading)

James Anderson — Thu, 04 Jun 2026 10:31:29 +0000

How AI Is Reshaping Software Development (and Where It's Heading)

A few years ago, "AI in software development" mostly meant autocomplete that guessed your next variable name. Today it writes functions, reviews pull requests, generates tests, explains unfamiliar codebases, and occasionally argues with you about architecture. The shift has been fast, and it's still accelerating.

This post is a practical look at what's actually changing, what's hype, and where things are likely headed.

What's Actually Changing Right Now

1. Writing code is no longer the bottleneck

For most of the history of programming, the slow part was typing the right thing. AI assistants have quietly removed a lot of that friction. Boilerplate, glue code, config files, and one-off scripts now take seconds instead of minutes.

The interesting consequence: the bottleneck moves upstream. The hard part becomes knowing what to build and how to structure it, not the act of writing each line.

2. Reading and understanding code got easier

Onboarding to a new codebase used to mean days of confused scrolling. Now you can ask an AI to summarize a module, trace a function's call path, or explain why some gnarly regex exists. This is arguably more valuable than code generation, because developers spend far more time reading code than writing it.

3. Testing and review are becoming AI-assisted

Generating unit tests, catching obvious bugs, suggesting edge cases, and doing a first pass on pull requests are all things AI handles reasonably well. It doesn't replace a senior reviewer, but it removes the trivial back-and-forth so humans can focus on the judgment calls.

4. The barrier to entry dropped

People who couldn't previously ship software now can. A designer can build a working prototype. A data analyst can wire up a small tool. This expands who participates in building software, which is both exciting and messy.

What's Still Overhyped

It's worth being honest here, because the hype cycle is loud.

"AI will replace developers." It won't, at least not in any near-term sense. AI is great at local, well-specified tasks and unreliable at large, ambiguous, system-level decisions. Software engineering has always been mostly about managing complexity and ambiguity — exactly the part AI is weakest at.
"You don't need to learn fundamentals anymore." The opposite is closer to true. When AI generates plausible-but-wrong code, you need more understanding to catch it, not less.
"It just works." Anyone who has shipped AI-generated code to production knows the failure modes: subtle bugs, outdated patterns, confident hallucinations, and security holes that look fine at a glance.

How the Developer Role Is Shifting

The job isn't disappearing — it's changing shape. A few trends worth watching:

From writing to reviewing. More of your time goes into evaluating generated output rather than producing it from scratch.
From syntax to systems. Knowing how to express a loop matters less; knowing how components interact, scale, and fail matters more.
From solo craft to orchestration. Increasingly the skill is directing tools — describing intent clearly, breaking work into verifiable pieces, and validating results.
Taste becomes a differentiator. When generating code is cheap, knowing what good looks like becomes the scarce, valuable thing.

Where This Is Likely Heading

Predictions are risky, but here are some reasonable bets for the next few years.

Agents that do multi-step work

We're moving past single-suggestion autocomplete toward agents that can take a task, plan it, make changes across files, run tests, and iterate. These already exist in early form. They'll get more reliable, but "reliable enough to trust unsupervised" is a high and slow-moving bar.

Verification becomes the centerpiece

As generation gets cheaper, the value shifts to checking. Expect more tooling around automated testing, formal-ish verification, sandboxed execution, and ways to prove that generated code does what it claims.

Specs and intent become the source of truth

If code is increasingly generated, the durable artifact becomes the clear specification of intent — well-written requirements, types, contracts, and tests. The skill of expressing intent precisely will only grow in importance.

A widening gap based on judgment

Tools will be roughly equally available to everyone. The differentiator won't be access; it'll be the judgment to use them well — knowing when to trust output, when to throw it away, and how to architect something that survives contact with reality.

What This Means for You

If you're a developer, a few practical takeaways:

Use the tools, but stay skeptical. Treat AI output like code from a fast, confident junior dev: useful, but verify everything.
Double down on fundamentals. Systems design, debugging, data modeling, and reasoning about tradeoffs are getting more valuable, not less.
Get good at specifying intent. Clear thinking and clear writing are now directly productivity multipliers.
Learn to verify, not just to produce. Testing and review skills are quietly becoming core competencies.

Closing Thought

AI isn't making software development obsolete — it's removing the parts that were never the point. Typing was never the job. Thinking was. The developers who thrive will be the ones who lean into the judgment, design, and verification that machines still can't do well.

The tools will keep getting better. The question worth asking isn't "will AI take my job?" but "what's the most valuable thing I can do once the routine parts are handled?"

What's your experience been? Has AI changed your workflow more than you expected, or less? Drop a comment.

DEV Community: James Anderson

Inside the Tokenizer: Why the Same Prompt Costs Different Amounts on Every Model

What a Token Actually Is

How Tokenizers Are Built: BPE in Plain Terms

Count It Yourself: tiktoken

Why the Same Text Costs Different Amounts on Different Models

What Makes Token Usage Worse (and Better)

Inflates your token count 📈

Reduces your token count 📉

The Language Tax: Why Non-English Costs 2-15x More

Now the Money: How to Actually Calculate Cost

Representative flagship prices (per 1M tokens)

Worked example: one chatbot request

Scaling it out (same request, 100,000/day)

The output tax, made concrete

The language tax, in dollars

Blended rate: comparing providers honestly

Cost Levers That Actually Work

Practical Takeaways

A Claude Code skill fixed my app's UI — here's what broke and how to use it yourself

The problems it fixed (real, not vibes)

Why a "skill" and not just "make it nicer"

What you do with it

Install it

Use it

Or build your own

How PDF & Document Parsers Actually Work Under the Hood

How I Stopped Burning Out as a Developer

How AI Is Reshaping Software Development (and Where It's Heading)

How AI Is Reshaping Software Development (and Where It's Heading)

What's Actually Changing Right Now

1. Writing code is no longer the bottleneck

2. Reading and understanding code got easier

3. Testing and review are becoming AI-assisted

4. The barrier to entry dropped

What's Still Overhyped

How the Developer Role Is Shifting

Where This Is Likely Heading

Agents that do multi-step work

Verification becomes the centerpiece

Specs and intent become the source of truth

A widening gap based on judgment

What This Means for You

Closing Thought

Count It Yourself: `tiktoken`