DEV Community: Emma Schmidt

The Vulnerability That Took Down Your Build Wasn't in Your Code

Emma Schmidt — Wed, 19 Aug 2026 05:46:54 +0000

You didn't write a single insecure line. You reviewed every pull request. Your own code is clean. And yet a routine dependency update just pulled in a compromised package three layers deep in your dependency tree, and now there's malicious code running in your CI pipeline that nobody on your team ever actually looked at. This isn't a hypothetical, it's the exact pattern behind a wave of real software supply chain attacks that's made securing the dependency chain one of the most active DevSecOps priorities of the year, not a compliance checkbox anymore, but a genuine engineering discipline.

Here's what's actually driving this, what a real defense looks like in practice, and how to start building it into a pipeline that doesn't already have it.

Why This Became Urgent Rather Than Theoretical

For years, supply chain security lived in the "we should probably think about this someday" category for most teams. A few things changed that.

Attackers shifted their target. Compromising one popular open-source package gives an attacker access to every downstream project that depends on it, which is a dramatically better return on effort than attacking one company directly
Dependency trees have grown enormous and mostly invisible. A modern project with a handful of direct dependencies can easily pull in hundreds of transitive ones, most of which nobody on the team has ever actually reviewed
CI/CD pipelines became a genuine attack surface of their own. A compromised build step doesn't just affect one developer's machine, it can inject malicious code directly into what gets shipped to production
Regulatory pressure caught up. Software bill of materials requirements are increasingly showing up in vendor contracts and compliance frameworks, turning "do we know what's in our dependency tree" from a nice-to-have into a real audit question
AI-driven systems raised the stakes further. As more production pipelines now include model training, fine-tuning, and inference steps alongside traditional code, the same supply chain exposure now extends to model weights, training data sources, and the infrastructure feeding them

What a Real Supply Chain Attack Actually Looks Like

Understanding the shape of these attacks makes the defense make a lot more sense.

A maintainer's account on a popular package gets compromised, and a malicious update gets published under a trusted name
A typosquatted package with a name one character off from a popular library gets installed by mistake during a routine npm install or pip install
A legitimate package gets acquired or handed off to a new maintainer who quietly introduces malicious code in a later version
A build script inside a dependency runs arbitrary code during installation, before your own application code ever executes

None of these require your team to write a single insecure line. They exploit the trust baked into how modern dependency management works by default.

The Core Defenses Worth Actually Building

Software Bill of Materials (SBOM)
A structured, complete inventory of every dependency in your project, direct and transitive. Without this, you genuinely don't know what's actually running in production, which makes it impossible to respond quickly when a vulnerability in a specific package gets disclosed.

Dependency pinning and lockfiles
Pinning exact versions rather than accepting a range prevents a compromised update from silently flowing into your build the next time someone runs an install. Lockfiles should be committed and treated as part of your actual source of truth, not a generated afterthought.

Automated vulnerability scanning
Continuously checking your dependency tree against known vulnerability databases, ideally as part of CI, so a newly disclosed issue in something you already depend on gets flagged automatically rather than discovered by accident.

Provenance verification
Confirming that a package actually came from where it claims to have come from, using cryptographic signing where the ecosystem supports it, rather than trusting a package registry blindly.

Least-privilege CI pipelines
Build steps should only have access to exactly what they need. A compromised dependency running inside an overprivileged CI job can do far more damage than one running with tightly scoped permissions.

A Practical Starting Sequence

You don't need to solve every layer of this at once. A reasonable order to tackle it in:

Generate an SBOM for your existing project as a baseline, so you actually know what's in your dependency tree today
Enable automated vulnerability scanning in CI, so new issues surface automatically going forward
Audit and commit your lockfiles, closing the gap where version ranges could silently pull in something unexpected
Review CI pipeline permissions, scoping down anything with broader access than it actually needs
Set a recurring cadence for dependency review, not just reacting when something breaks or gets flagged

Common Mistakes Worth Avoiding

Treating an SBOM as a one-time compliance document instead of something that needs to stay current as dependencies change
Scanning for vulnerabilities but never actually acting on what gets flagged, which makes the scanning theater rather than defense
Assuming a popular, widely-used package is automatically safe, when popularity is exactly what makes a package an attractive target
Giving CI pipelines broad, unscoped access "to make things easier," which turns a single compromised dependency into a much bigger incident
Skipping this entirely for internal tools and side projects, when those often have the least oversight and the most direct access to sensitive systems

A Quick Self-Check

Do you actually know every dependency in your project right now, including the transitive ones, or would you have to go find out?
Is your CI pipeline scanning for known vulnerabilities automatically, or does that only happen when someone remembers to check?
If a popular package you depend on were compromised tomorrow, how quickly could your team confirm whether you're affected?
Do your CI build steps have more access than they actually need to do their job?

If more than one of those gave you pause, there's real, addressable exposure sitting in your pipeline right now.

Why This Deserves Real Investment, Not a Checkbox

Supply chain security tends to get deprioritized precisely because it's invisible until the day it isn't. Building this properly, real SBOM tooling, automated scanning wired into CI, scoped pipeline permissions, is genuinely cross-cutting work that touches DevOps, security, and every team shipping code, and it's easy to underestimate the ongoing maintenance this requires as dependencies constantly shift underneath a project. This is exactly where AI security and AI infrastructure work increasingly overlaps with traditional DevSecOps too, since teams running models in production now need the same rigor applied to ML CI/CD pipelines, model governance, and compliance around training data provenance that they'd apply to any other dependency chain. Getting this foundation right early, across both code and model infrastructure, is a lot cheaper than untangling it after an incident.

The Takeaway

The vulnerability that takes down your build was never guaranteed to be one your own team wrote. Most modern software is built on a foundation of trust in dependencies nobody on the team has actually reviewed, and that trust is exactly what attackers are exploiting at scale right now. Treating supply chain security as core infrastructure, not an afterthought, is quickly becoming table stakes rather than a nice-to-have.

Has your team actually built out SBOM tooling and automated dependency scanning yet, or is this still sitting on the someday list? Curious how far along everyone actually is with this.

7 AI Integration Mistakes I Keep Seeing Teams Make in 2026

Emma Schmidt — Tue, 18 Aug 2026 05:32:29 +0000

I don't write about AI integration failures because they're rare.

I write about them because I keep seeing the exact same ones, on different teams, in different industries, month after month.

A mistake earns a spot on this list when I've watched it happen more than once. Not a one-off bad decision. A pattern.

I've spent a lot of time this year looking closely at how companies actually bring AI into existing products, not the demo version, the real production version with real users and real data on the line.

These are the seven mistakes that showed up again and again.

1. Treating AI Integration Like a Feature Flag

Repository of thought: "we'll just bolt it on and see how it goes."

I get why this feels reasonable. Ship fast, learn fast, iterate. But AI features aren't like a new button on a page. They touch data flow, user trust, and cost in ways a typical feature toggle doesn't.

The teams that get burned here usually find out the hard way, weeks later, when the "quick add" needs a rewrite because nobody thought about:

Where the data actually comes from
What happens when the model is wrong
How much this costs at real usage volume

2. Skipping the "What Happens When It's Wrong" Conversation

Every demo works. Every demo is cherry-picked, even unintentionally.

The conversation that actually matters happens after the demo: what does the user see when the AI confidently gives a wrong answer? Is there a fallback? Is there a way to flag it? Does anyone even find out it happened?

I've watched teams skip this conversation entirely and only have it for the first time after a customer complaint. That's the wrong order.

3. No Evaluation Loop, Just Vibes

"It feels like it's working well" is not a metric.

Teams that are serious about this build some kind of evaluation loop, a way to systematically check output quality against known good examples, before shipping and continuously after. Teams that skip this find out their AI feature quietly got worse three months in, and nobody noticed until a user did.

4. Underestimating the Data Quality Problem

This one isn't glamorous, which is probably why it gets skipped so often.

AI integration is only as good as the data feeding it. I've seen teams spend weeks tuning a prompt to fix a problem that was actually a data quality issue the whole time. Messy, inconsistent, or outdated source data will sink an otherwise well-built integration every time.

5. One Model, No Fallback

Betting an entire feature on a single model provider with no fallback plan is a real, recurring pattern I keep seeing.

What happens when that provider has an outage. What happens when pricing changes overnight. What happens when a model update shifts behavior in a way that breaks your carefully tuned prompts.

Teams that plan for this from day one recover from disruptions in hours. Teams that don't scramble for days.

6. Governance as an Afterthought

This is the one that worries me most, because it's invisible until it's a real problem.

Who's responsible for reviewing what the AI feature actually does. Who signs off on it touching sensitive data. Who's accountable if it produces something harmful or wrong in a regulated context. If nobody can answer these clearly, that's the actual gap, not a hypothetical one.

7. Measuring the Wrong Thing

"We shipped an AI feature" isn't a success metric. Neither is "users are using it."

The teams doing this well are measuring whether the feature actually improved the outcome it was supposed to improve, resolution time, task completion, accuracy against a real baseline. Everything else is activity dressed up as progress.

A Quick Table, Because Patterns Are Easier to See This Way

Mistake	What It Actually Costs You
Bolting AI on like a feature flag	Rework once real data and edge cases show up
No plan for wrong answers	Trust erosion the first time a user hits one
No evaluation loop	Silent quality drift nobody catches in time
Ignoring data quality	Wasted weeks tuning prompts that were never the real problem
No fallback model	Full outages instead of graceful degradation
No governance ownership	A real incident with no clear accountability
Vanity metrics	Confidently shipping something that isn't actually working

What I'd Tell a Team Starting This Today

Slow down on the parts that feel boring. Data quality, evaluation, governance, none of it is exciting, all of it is where the real risk actually lives.

The AI part is usually not the hard part anymore. The hard part is everything around it, the parts that don't show up in a demo.

Final Thoughts

None of these seven mistakes are exotic. They're not edge cases. They're the ordinary, predictable ways AI integration goes sideways when a team moves fast without slowing down for the boring, unglamorous parts.

I don't think that's a knock on any team that's hit one of these. I've watched genuinely sharp engineers walk into every single one of them, because the pressure to ship fast is real and these mistakes are easy to make with the best of intentions.

If you're integrating AI into a real product right now, I'd rather you catch these on paper than in production.

Which of these have you actually run into? Curious how many of these seven are familiar to people reading this.

Top AI Coding Tools Every Developer Should Know in 2026

Emma Schmidt — Mon, 17 Aug 2026 12:56:58 +0000

A year ago, "AI coding assistant" mostly meant autocomplete on steroids. In 2026, that's no longer true. We've moved from inline suggestions to agentic AI tools that can plan a task, write across multiple files, run your test suite, and fix their own mistakes before you even open the diff.

If you're a developer trying to figure out which tools are actually worth wiring into your workflow (and which are just hype), this guide breaks it down: what's changed, how to evaluate these tools, a side-by-side comparison, and a practical framework for adopting them on a real engineering team.

Why This Shift Matters

The jump from "autocomplete" to "agent" isn't just a marketing term. It changes how you work:

Autocomplete-era tools predicted the next few lines of code based on context.
Chat-era tools let you ask questions and get code snippets back.
Agentic-era tools take a goal ("add pagination to this API and update the tests"), break it into steps, execute those steps across your codebase, and self-correct using tool feedback like compiler errors or failing tests.

That last category is what's driving real productivity gains right now but it also introduces new questions around code review, security, and trust that teams need a process for, not just a tool.

How to Evaluate an AI Coding Tool

Before comparing specific products, it helps to have a checklist. Here's what actually matters when you're picking a tool for a team (not just a solo side project):

Context window & codebase awareness: Can it reason across your whole repo, or just the open file?
Agentic capability: Can it execute multi-step tasks (run tests, fix errors, open a PR) or only suggest text?
IDE / CLI fit: Does it live where your team already works, or does it force a workflow change?
Security & data handling: Where does your code go? Is there an enterprise/on-prem option?
Cost model: Per-seat, usage-based, or bundled into an existing subscription?
Review friction: Does it produce diffs a human can review quickly, or large unreviewable rewrites?

Comparison: Popular AI Coding Tools in 2026

Tool	Best For	Works In	Agentic?	Notable Strength
GitHub Copilot	General-purpose autocomplete + chat	VS Code, JetBrains, GitHub.com	Partial (Copilot Workspace)	Deep GitHub/PR integration
Claude Code	Terminal-first, multi-file agentic coding	CLI, VS Code, JetBrains	Yes	Strong reasoning on large refactors
Cursor	AI-native IDE experience	Standalone editor (VS Code fork)	Yes	Tight inline editing + chat loop
Amazon Q Developer	AWS-centric teams	IDEs, AWS Console	Partial	Native AWS service knowledge
Windsurf	Agentic pair-programming	Standalone editor	Yes	"Flow" mode for long tasks
Tabnine	Privacy-conscious teams	Most major IDEs	No	Self-hosted / on-prem options

Note: this landscape moves fast pricing, features, and even product names shift every few months, so treat this as a snapshot rather than a permanent ranking. Always check the vendor's docs before committing a team to one.

A Practical Adoption Framework

Handing an entire engineering org an AI coding tool on day one usually backfires you get inconsistent usage, review bottlenecks, and skepticism from senior engineers who've been burned by low-quality AI output. A more reliable rollout looks like this:

Pilot with a small, willing team: on a real (but non-critical) project for 2–4 weeks.
Define review rules: e.g., AI-generated PRs still require the same review bar as human ones, no exceptions.
Track outcomes, not vibes: cycle time, bug rate, and PR size are better signals than "it feels faster."
Standardize the toolchain: so context (style guides, architecture docs) is shared across the team instead of living in one engineer's prompts.
Revisit quarterly:this space changes fast enough that a tool decision from six months ago may already be outdated.

This is close to how engineering teams evaluate any new dependency: agentic AI coding tools are powerful, but they still need the same rigor as any other addition to a production workflow. Teams that treat AI tooling as an accelerant for experienced engineers not a replacement for code review and architecture decisions tend to get the most durable results.

A Quick Example: Agentic Refactor in Practice

Here's a simplified illustration of the kind of task an agentic tool can now handle end-to-end, versus what you'd have done manually a year ago:

# Old way: you write the prompt, get a snippet, paste it, fix imports, run tests manually
$ ai "write a function to paginate this list"

# Agentic way: the tool plans, edits multiple files, runs tests, and reports back
$ agent run "add cursor-based pagination to /api/users, update the OpenAPI spec, and make the existing tests pass"

# Example output the agent might produce and then validate against your test suite
def paginate_users(users, cursor=None, limit=20):
    start = 0
    if cursor:
        start = next((i for i, u in enumerate(users) if u["id"] == cursor), 0) + 1
    page = users[start:start + limit]
    next_cursor = page[-1]["id"] if len(page) == limit else None
    return {"items": page, "next_cursor": next_cursor}

The difference isn't the code itself it's that the agent verified the change against your real test suite before handing it back, closing the loop that used to require a human every time.

Where This Is Heading

A few trends worth watching as 2026 progresses:

Agentic AI moving from "assistant" to "team member": tools that pick up tickets, open draft PRs, and tag humans only when they're stuck.
AI-driven workflow automation: spreading beyond coding into project management, QA, and DevOps pipelines not just writing code, but operating around it.
Low-code/no-code and pro-code converging: with AI acting as the translation layer between visual builders and hand-written code for teams that outgrow no-code tools.
Stronger emphasis on governance: code provenance, license compliance, and audit trails for AI-generated code, especially in regulated industries like fintech and healthcare.

Final Thoughts

AI coding tools in 2026 aren't a novelty anymore they're infrastructure. The teams getting the most value aren't the ones chasing every new tool, but the ones treating adoption like any other engineering decision: pilot it, measure it, standardize it, and keep your review bar exactly where it was before AI showed up.

If you're building out this kind of process for your own team, start small, measure real outcomes, and resist the urge to lower your review bar just because the code came from an agent instead of a human.

Your AI Chatbot Forgets Everything the Moment You Close the Tab. Here's the Fix Nobody's Talking About Yet.

Emma Schmidt — Wed, 05 Aug 2026 05:18:19 +0000

Ask a typical RAG-powered AI assistant a follow-up question that depends on something you mentioned three messages ago, and watch it quietly lose the thread. Standard retrieval-augmented generation is excellent at pulling relevant chunks of text based on similarity, and genuinely terrible at understanding how pieces of information actually relate to each other or persisting what it learned about you across a session. This exact gap is what's pushing serious AI application development toward GraphRAG and contextual memory, two of the fastest-moving concepts in AI architecture right now.

Here's what's actually broken with traditional RAG, what GraphRAG and contextual memory do differently, and how to think about building with them.

Why Standard RAG Hits a Wall

Traditional RAG works by chunking documents, embedding them, and retrieving the chunks most semantically similar to a user's query. It's simple, effective for straightforward lookup questions, and has become the default pattern for grounding AI responses in real data.

It also has real, well-known limitations that show up quickly in production.

Limitation	What It Looks Like in Practice
No relationship awareness	Retrieves individual chunks but misses how entities and facts connect across documents
Weak multi-hop reasoning	Struggles with questions requiring several linked facts, like "which vendor supplies the part used in the product that failed last quarter"
No persistent memory	Each query is treated in isolation, with no real understanding of what happened earlier in the conversation
Chunking artifacts	Splitting documents into fixed-size chunks can sever context that spans a chunk boundary, losing meaning
Redundant retrieval	Similar chunks from different documents often surface together, adding noise instead of new information

None of this makes standard RAG useless. It makes it a genuinely good starting point that runs out of runway the moment your application needs to reason across relationships or remember anything meaningfully across turns.

A Concrete Example of Where This Breaks Down

Picture an internal AI assistant built for a mid-sized company's operations team, trained on years of vendor contracts, compliance documents, and internal policy memos. Someone asks: "Which of our vendors would be affected if the new data residency policy applies to our EU contracts?"

A standard RAG setup retrieves chunks mentioning "vendors," separately retrieves chunks mentioning "data residency policy," and separately retrieves chunks mentioning "EU contracts." Nothing in that retrieval step actually confirms which vendors are tied to which contracts, or which contracts fall under EU jurisdiction. The model is left to guess at the connections from loosely related text, and in a compliance context, a confident guess dressed up as an answer is genuinely dangerous.

This is exactly the class of failure that pushed graph-based approaches from a research curiosity into a real production pattern this year, anywhere the cost of a wrong but confident-sounding answer is high.

What GraphRAG Actually Changes

GraphRAG restructures the underlying knowledge as a graph, entities as nodes, relationships as edges, instead of a flat collection of embedded text chunks. Instead of retrieving isolated pieces of text based purely on similarity, the system can traverse relationships to answer questions that require connecting multiple facts together.

Entity-relationship structure. Facts aren't just stored, they're connected: a product links to its supplier, which links to a region, which links to a regulation
Multi-hop query support. A question requiring three connected facts can be answered by traversing the graph, rather than hoping all three facts happen to land in the same retrieved chunk
Community-level summarization. Graph-based approaches can summarize entire clusters of related entities, giving a system a genuine overview rather than a pile of disconnected snippets
Better handling of sparse or niche domains. Where a document corpus is small or highly specialized, relationship structure often surfaces relevant context that pure similarity search misses entirely

How a Knowledge Graph Actually Gets Built

It's worth demystifying this a bit, since "build a knowledge graph" can sound more exotic than the actual process.

Entity extraction. An initial pass through the source documents identifies key entities, people, products, organizations, regulations, dates
Relationship extraction. A second pass identifies how those entities connect, which vendor supplies which product, which contract falls under which jurisdiction
Graph construction. Entities become nodes and relationships become edges, forming a structured map of the underlying data instead of a flat pile of text chunks
Community detection. Related clusters of nodes get grouped, which enables the community-level summarization that flat retrieval simply can't produce
Ongoing maintenance. As source documents update, the graph needs re-processing to reflect new entities and relationships, which is the step most teams underestimate when planning a build

Contextual Memory: The Other Half of the Problem

GraphRAG solves the relationship problem. Contextual memory solves the persistence problem, the part where a system actually remembers relevant information across a conversation or across sessions, instead of starting from zero every single time.

Short-term working memory tracks what's been discussed in the current session, so a follow-up question doesn't need to restate context the user already gave
Long-term memory persists meaningful facts about a user or task across sessions, a returning customer's preferences, a project's ongoing status, a prior decision that shouldn't need repeating
Selective memory, not total recall matters just as much as remembering. A well-designed system decides what's actually worth retaining rather than storing every message verbatim, which keeps retrieval fast and relevant instead of bloated

Different Approaches to Implementing Memory

Not all contextual memory systems work the same way, and picking the right approach matters.

Approach	How It Works	Best Suited For
Sliding window	Keeps the last N messages in full	Short interactions where recent context is all that matters
Summarization-based	Periodically compresses older turns into a running summary	Longer conversations where full detail on early turns matters less over time
Fact extraction	Pulls out specific durable facts and stores them separately from the raw conversation	Cross-session persistence, like remembering a user's stated preferences
Hybrid	Combines a short-term window with extracted long-term facts	Most production systems, balancing responsiveness with genuine persistence

Most serious production systems end up using some version of the hybrid approach, since a sliding window alone can't persist across sessions, and pure summarization alone tends to lose specific, useful details that fact extraction preserves.

How These Two Pieces Fit Together

GraphRAG and contextual memory solve different problems, and the strongest AI applications right now are combining both rather than picking one.

GraphRAG handles reasoning across your knowledge base, connecting facts that live in different documents or records
Contextual memory handles reasoning across the conversation itself, remembering what the user already told the system
Together, they let an AI application answer a question like "does this apply to the client I mentioned earlier, given the update we discussed last week" by pulling both the relationship structure from the knowledge graph and the recalled context from memory

A Simplified Illustration of the Difference

Standard RAG on a multi-hop question:

"Which of our suppliers are affected by the new import regulation?"
Retrieves chunks mentioning "suppliers" and chunks mentioning "import regulation" separately, with no guarantee the system connects which supplier is actually affected by which regulation.

GraphRAG on the same question:

Traverses supplier nodes connected to region nodes connected to regulation nodes, returning the specific suppliers actually linked to the affected region, because that relationship is explicitly modeled rather than inferred from text similarity.

Add contextual memory:

If the user previously said "we're focused on our European suppliers this quarter," the system recalls that context and narrows the answer accordingly, without the user needing to repeat it.

Where Teams Get This Wrong

Jumping to GraphRAG for every use case, when a simple, well-chunked standard RAG setup is genuinely sufficient for straightforward lookup tasks and adds unnecessary complexity for no real benefit
Building memory that stores everything indiscriminately, which bloats retrieval and actually makes responses slower and noisier rather than smarter
Treating graph construction as a one-time task, when knowledge graphs need ongoing maintenance as entities and relationships change over time
Ignoring evaluation entirely, shipping a GraphRAG or memory system without a way to systematically test whether it's actually improving answer quality over the simpler baseline it replaced
Conflating memory with logging, storing raw conversation history and calling it memory, without any real process for deciding what's actually worth retaining long term

A Quick Framework for Deciding What You Need

Does your use case genuinely require connecting facts across multiple documents or records, or are most queries simple, single-fact lookups?
Does your application need to remember information across turns or sessions, or is each interaction naturally self-contained?
Do you have the ongoing resources to maintain a knowledge graph as your underlying data evolves?
How costly is a wrong but confident-sounding answer in your specific use case? The higher that cost, the stronger the case for the added structure GraphRAG provides

Common Questions Teams Ask When Evaluating This

Does GraphRAG replace standard RAG entirely?
No, most production systems use both together. Simple lookups can still go through standard retrieval, while multi-hop reasoning questions route through the graph.

How much does this increase latency?
Graph traversal adds some overhead compared to a single similarity search, though well-indexed graphs with efficient traversal patterns keep this manageable for most interactive use cases. Benchmarking on your actual data is essential rather than assuming.

Is this only worth it for large enterprises?
No. Smaller, highly specialized domains often benefit even more, since sparse document corpora are exactly where relationship structure tends to add the most value over pure similarity search.

Why This Is Worth Building Properly

Getting graph construction, retrieval traversal, and memory persistence right together is genuinely more involved than standing up a basic RAG pipeline, and it's easy to underestimate the ongoing maintenance a knowledge graph requires as your data changes. This is exactly the kind of RAG development and AI application architecture work that separates a genuinely smarter AI app from one that just looks smart in a demo, and it's usually worth getting a second opinion on before committing to a full build.

The Takeaway

Standard RAG got a lot of AI applications off the ground, and it's still the right choice for plenty of use cases. But the moment an application needs to reason across relationships or remember anything meaningfully across a conversation, that flat, isolated retrieval model runs out of road fast. GraphRAG and contextual memory are exactly the pair of ideas closing that gap this year, and understanding when you actually need them, not just how to build them, is quickly becoming a core skill for anyone building serious AI products.

Has your team started experimenting with GraphRAG or persistent memory yet, or is standard RAG still handling everything you're throwing at it? Curious how far along everyone actually is with this.

Your App Has AI Features. That's Not the Same Thing as Being an AI Company Anymore.

Emma Schmidt — Fri, 31 Jul 2026 10:48:07 +0000

There's a quiet but important distinction reshaping how serious software gets built in 2026, and most teams haven't caught up to it yet. Bolting a chatbot onto an existing app and designing an application where AI is the actual architecture are two completely different engineering disciplines, and the gap between them is becoming one of the clearest competitive differentiators in software today. This is exactly the shift driving demand for real custom AI development, not a plugin added after the fact, but AI woven into how a system makes decisions, processes data, and adapts from the very first architectural diagram.

Here's what actually separates AI-native software from AI-enhanced software, why the distinction matters more than it sounds, and how to think about it if you're planning a build.

AI-Enhanced vs AI-Native: The Real Difference

Most existing "AI-powered" products today are AI-enhanced. A traditional application, built the same way it always would have been, with a recommendation widget, a search-with-AI feature, or a support chatbot added on top. The underlying system, its data flow, its decision logic, its architecture, remains essentially unchanged.

An AI-native application is architected differently from day one. Intelligence isn't a feature sitting on top, it's part of how the system actually functions.

Dimension	AI-Enhanced	AI-Native
Where AI sits	Bolted on as a feature or module	Built into the core architecture
Data flow	Static, mostly predefined	Continuous, feeding real-time model decisions
Adaptability	Fixed behavior between releases	Learns and adjusts from ongoing data
Typical origin	Retrofit of an existing product	Designed around AI from the first architecture decision
Failure mode when done poorly	AI feature feels bolted on, inconsistent with the rest of the app	Poor governance and evaluation gaps compound across the whole system

Neither approach is inherently wrong. A lot of genuinely good products are AI-enhanced and don't need to be anything else. But for products where intelligence is meant to be the actual value proposition, retrofitting an existing architecture tends to be far more expensive and painful than building AI-native from the start.

Why This Shift Is Happening Now

A few forces are converging to push serious teams toward AI-native thinking rather than incremental AI feature additions.

Retrofitting is proving genuinely costly. Teams that added AI as an afterthought are running into architectural walls, data pipelines that weren't built for continuous model input, decision logic that was never designed to be adaptive, and it's turning into expensive rework rather than a quick feature addition
Evaluation and observability are now considered table stakes, not extras. Serious AI development in 2026 increasingly means having real evaluation pipelines that systematically test outputs against known-good examples, not just shipping a prompt and hoping it behaves
Security expectations have caught up with adoption. Prompt injection, data exfiltration through AI interfaces, and output manipulation are now treated as real attack vectors, which means the AI layer needs the same security discipline as any other system boundary, not a bolted-on afterthought
Multi-model orchestration is becoming a standard architectural component. Rather than routing everything through one model, AI-native systems increasingly route different tasks to different models based on cost, latency, and capability, which only works cleanly if the architecture was designed with that flexibility in mind

What AI-Native Architecture Actually Requires

Building this way touches more of the stack than a typical feature addition would.

Data pipelines designed for continuous flow, not periodic batch updates, since adaptive systems need fresh data to actually adapt
Evaluation infrastructure built in from the start, tracking output quality, catching model regressions, and validating behavior against real, known-good examples
Clear governance around AI decision-making, especially for anything touching regulated data or high-stakes outcomes, where explainability and audit trails matter
Architecture flexible enough for multi-model routing, so the system isn't locked into a single model provider or a single point of failure
Security treated as a first-class concern for the AI layer specifically, not assumed to be covered by the same protections as the rest of the application

Real-World Scenarios Where This Distinction Actually Matters

To make this concrete, here's where the AI-native versus AI-enhanced choice genuinely changes the outcome.

A support platform adding a chatbot to an existing ticketing system is AI-enhanced. A support platform where routing, prioritization, and resolution suggestions are all driven continuously by live data from the start is AI-native, and it behaves fundamentally differently as ticket volume and complexity grow
A financial analytics tool with an AI summary feature bolted onto existing dashboards is AI-enhanced. One where risk scoring and forecasting are core to how the system processes every transaction is AI-native
An internal knowledge tool with search-with-AI added to an existing wiki is AI-enhanced. One built around continuously updated embeddings and adaptive retrieval from the ground up is AI-native
A logistics platform with a recommendation widget for routing suggestions is AI-enhanced. One where routing decisions are made continuously by an adaptive model integrated into the core dispatch logic is AI-native

Where Teams Get This Wrong

Assuming AI-native means "use more AI features." It's an architectural distinction, not a feature count. A product can have fewer visible AI features and still be genuinely AI-native if intelligence is core to how it functions
Skipping the evaluation and governance layer to move faster. This is the corner that gets cut most often, and it's the one that causes the most expensive problems later, once the system is handling real production decisions
Treating security as something to add after the AI features work. By the time an AI-native system is live, retrofitting proper security around prompt handling and data access is far harder than designing it in from the start
Underestimating the data infrastructure work. Teams often plan for the model and the interface, but underestimate how much of the actual effort lives in the data pipelines that feed continuous, adaptive behavior

A Quick Framework for Deciding What You Actually Need

Before committing to a full AI-native rebuild, it's worth being honest about which category your product actually falls into.

Is intelligence the core value proposition, or a helpful addition to a product that works fine without it?
Does your system need to adapt continuously based on new data, or would periodic updates genuinely be sufficient?
Do you have, or are you willing to build, the evaluation and governance infrastructure this approach requires?
Is your team prepared for the security implications of treating the AI layer as a first-class system boundary?

If the honest answer to the first two questions is "yes, genuinely," AI-native architecture is worth the upfront investment. If AI is a nice complement to an otherwise solid, deterministic product, a well-built AI-enhanced approach is often the more practical, lower-risk choice.

Why This Is Genuinely Hard to Get Right Alone

This isn't a criticism of any team, it's just an honest reflection of how new and fast-moving this discipline still is. The specific technical profile this requires, deep experience across data pipeline design, model evaluation, multi-model orchestration, and AI-specific security, remains genuinely scarce in most hiring markets right now. That's exactly why a lot of serious AI software development work today involves custom AI development support and AI readiness assessment brought in from outside, specifically to architect the foundation correctly before a team commits to a direction that's expensive to unwind later. Getting an experienced second opinion on whether a product genuinely needs full AI-native architecture, or whether a well-built AI-enhanced approach would serve it just as well, is often the highest-leverage conversation to have before writing a single line of the core architecture.

The Future Outlook

The gap between AI-enhanced and AI-native products is likely to keep widening over the next year or two, not narrow. As evaluation tooling matures and multi-model orchestration becomes more standardized, AI-native architecture is expected to shift from a differentiator into a baseline expectation, particularly for products where intelligence genuinely is the value proposition rather than a nice-to-have layered on top.

The Takeaway

Having AI features and being architected as an AI company are not the same thing, and that distinction is only going to matter more as the products competing for the same users diverge further along these two paths. The teams thinking carefully now about whether they genuinely need AI-native architecture, versus a solid, well-built AI-enhanced product, are the ones avoiding the expensive retrofit conversation a year or two from now.

Is your product built AI-native from the ground up, or is it AI-enhanced with intelligence layered on top? Curious where most teams here actually land on that distinction once they think it through honestly.

Build Your First MCP Server: A Practical Guide to Connecting AI Agents to Real Tools

Emma Schmidt — Thu, 30 Jul 2026 06:43:25 +0000

Learn how to build a working MCP server from scratch and connect it to an AI client, step by step.

Introduction

If you've ever needed an AI agent to check a database, call an internal API, or search a knowledge base, you've probably hit the same wall: every tool integration ends up as custom, one-off code that doesn't transfer to your next project. MCP server development and MCP integration solve this by giving you one standardized way to expose any tool or data source to any compatible AI client, instead of writing bespoke glue code for every combination.

This tutorial walks through building an actual working MCP server, connecting it to a client, and testing that it works end to end. By the end, you'll have a functioning server exposing a real tool, and you'll understand exactly what's happening at each layer instead of just copying a snippet.

Prerequisites

Comfortable reading and writing basic Python
Python 3.10 or later installed
A terminal and a code editor
Roughly 30-45 minutes

What We're Building

We're building a small MCP server that exposes one practical tool: looking up the weather for a given city. It's simple enough to fully understand in one sitting, but it demonstrates every core piece of setting up MCP tool integration.

The finished server will be able to:

Advertise its available tools to any connecting MCP client
Accept a tool call with a specific input (a city name)
Return a structured response back to the calling AI agent
Run locally so you can test it directly from your terminal

Step 1: Set Up Your Environment

Create a project folder and install the MCP SDK.

mkdir mcp-weather-server
cd mcp-weather-server
python -m venv venv
source venv/bin/activate  # on Windows use venv\Scripts\activate
pip install mcp requests

This gives you the core MCP library plus requests, which we'll use to call a public weather API inside our tool.

Step 2: Define the Server and Register Your Tool

Create a file called server.py. This is illustrative, simplified code meant for learning, not a full production implementation.

# server.py
from mcp.server import Server
from mcp.types import Tool, TextContent

app = Server("weather-server")

@app.list_tools()
async def list_tools():
    return [
        Tool(
            name="get_weather",
            description="Get the current weather for a given city",
            inputSchema={
                "type": "object",
                "properties": {
                    "city": {"type": "string", "description": "Name of the city"}
                },
                "required": ["city"]
            }
        )
    ]

This list_tools function is how a client discovers what your server can do. When an AI agent connects, it asks "what tools do you have," and this is the answer it gets back, in a structured format it can reason about.

Step 3: Implement What the Tool Actually Does

Now add the logic that runs when the tool is actually called.

# server.py (continued)
import requests

@app.call_tool()
async def call_tool(name: str, arguments: dict):
    if name == "get_weather":
        city = arguments["city"]
        # Simplified example call, replace with a real weather API and key
        response = requests.get(f"https://example-weather-api.test/v1/current?city={city}")
        data = response.json()
        summary = f"The weather in {city} is {data.get('condition', 'unknown')} at {data.get('temp', '?')}°"
        return [TextContent(type="text", text=summary)]

This is the execution layer. The AI model never talks to the weather API directly, it calls your tool through the protocol, and your code decides exactly what happens and what comes back.

Step 4: Run the Server Locally

Add a simple entry point so the server can actually start.

# server.py (continued)
from mcp.server.stdio import stdio_server
import asyncio

async def main():
    async with stdio_server() as (read_stream, write_stream):
        await app.run(read_stream, write_stream, app.create_initialization_options())

if __name__ == "__main__":
    asyncio.run(main())

Run it with:

python server.py

At this point, the server is running and waiting for a client to connect over standard input and output, which is the simplest transport for local development and testing.

Step 5: Connect a Client and Test It

To actually test this, you need an MCP-compatible client configured to launch your server. Most MCP client configurations use a simple JSON entry pointing to your script.

{
  "mcpServers": {
    "weather": {
      "command": "python",
      "args": ["/full/path/to/server.py"]
    }
  }
}

Once your client is configured and connected, ask it something like "what's the weather in Lisbon." If everything is wired correctly, the client will discover your get_weather tool through the list_tools call, invoke it with the city argument, and return the result from your call_tool function back through the model's response.

Testing / Verifying It Works

A few checks to confirm your MCP server integration is actually functioning correctly:

Confirm the client's tool list shows get_weather as an available tool after connecting
Manually call the tool with a test city and check that the response text looks correct
Try an invalid or missing argument and confirm your server doesn't crash silently
Restart the client and reconnect to make sure the server initializes cleanly every time, not just on the first run

Common Pitfalls and How to Avoid Them

Forgetting to validate input. A missing or malformed city argument will crash a naive implementation. Add basic validation before making the external call.
Blocking calls inside async functions. Using a purely synchronous HTTP library inside an async tool handler can freeze your server under load. Consider an async HTTP client for anything beyond a simple learning project.
Not handling API failures gracefully. If the weather API is down, your tool should return a clear error message, not throw an unhandled exception that breaks the whole session.
Overloading one server with too many unrelated tools. Keeping a server focused on one coherent set of capabilities makes it easier for both you and the connecting AI agent to reason about.

Best Practices for Production Use

Add proper authentication and scoped permissions before exposing any server beyond local testing
Log every tool call, including inputs and outcomes, so failures are debuggable after the fact
Set reasonable timeouts on any external API calls inside your tools
Validate all inputs against your declared schema before executing any logic
Version your tool definitions so you can evolve them without silently breaking existing clients

What to Explore Next

Add a second, related tool to the same server, like a multi-day forecast, and see how tool discovery scales
Swap the stdio transport for Streamable HTTP to run your server as a proper remote service
Explore adding authentication middleware so only verified callers can invoke your tools
Look into structured output schemas so responses can be validated programmatically instead of just returned as plain text

Conclusion

Building your first MCP server strips away a lot of the mystery around MCP integration once you've actually wired up discovery, execution, and a real client connection yourself. The pattern you just built scales directly to far more complex tools, databases, internal APIs, search systems, without changing the underlying shape of what you learned here. Once this clicks, adding your next tool is a lot faster than the first one was.

There's a New Universal Plug for AI, and Most Developers Haven't Wired It In Yet

Emma Schmidt — Wed, 29 Jul 2026 12:26:39 +0000

Picture this. You've built an AI feature that needs to check a customer's order status, pull data from your CRM, and search internal documentation, three separate systems, three separate custom integrations, each with its own auth, its own data format, its own maintenance burden. Now imagine adding a fourth tool next quarter. And a fifth. This exact integration sprawl is precisely the problem the Model Context Protocol was built to solve, and it's become one of the fastest-adopted standards in recent memory across LLM & GenAI development. Understanding how to actually build with it, not just read about it, is quickly becoming a core skill for anyone shipping AI features that need to touch real data and real tools.

Here's what MCP actually is, why it's spreading this fast, and how to build your first working MCP server step by step.

Why This Protocol Is Spreading Faster Than Almost Anything in AI Tooling

The adoption numbers here are genuinely unusual, even for an industry used to fast-moving trends.

Metric	Figure
Monthly SDK downloads (March 2026)	97 million, up from roughly 100,000 at launch
Time to reach 100M+ monthly downloads	About 16 months
Comparable milestone for React's npm package	Roughly 3 years
Public MCP servers in the official registry	Around 9,600 as of a May 2026 snapshot
Fortune 500 companies with MCP deployed	Roughly 28% within 18 months of launch
Major platforms with shipped MCP support	OpenAI, Google, Microsoft, and others within about 13 months of launch

That last row is the one worth sitting with. A protocol originally released by Anthropic in November 2024 has since been adopted broadly enough across the industry that it now functions as shared infrastructure rather than one company's proprietary approach, which is exactly the kind of cross-platform momentum that turns a promising idea into a genuine standard.

What MCP Actually Solves

Before MCP, connecting an AI application to external tools meant building a custom integration for every single combination of model and tool. Ten AI applications and a hundred tools meant a potential one thousand separate integrations, each one bespoke, each one a maintenance burden of its own.

MCP replaces that with one standardized interface. Think of it the way USB-C replaced a drawer full of proprietary charging cables. Any MCP-compatible client can talk to any MCP-compatible server through the same protocol, regardless of which model or which tool is on either end.

The Three Core Pieces You Need to Understand

MCP Host. The AI application itself, the thing the end user actually interacts with
MCP Client. Lives inside the host and manages the connection to one or more MCP servers
MCP Server. Exposes specific tools, data sources, or capabilities in the standardized format the protocol expects

Most developers building on top of MCP will spend their time on the server side, exposing their own systems and tools so any compatible AI application can use them.

Building Your First MCP Server, Step by Step

Let's build a simple MCP server that exposes a single tool, checking an order status, using the official Python SDK.

Step one: install the SDK

pip install mcp

Step two: define the server and register a tool

from mcp.server import Server
from mcp.types import Tool, TextContent

server = Server("order-status-server")

@server.list_tools()
async def list_tools():
    return [
        Tool(
            name="get_order_status",
            description="Look up the current status of a customer order",
            inputSchema={
                "type": "object",
                "properties": {
                    "order_id": {"type": "string"}
                },
                "required": ["order_id"]
            }
        )
    ]

This is the discovery step. Any MCP client connecting to this server can now ask what tools are available and get back a structured description it can reason about.

Step three: implement what the tool actually does

@server.call_tool()
async def call_tool(name: str, arguments: dict):
    if name == "get_order_status":
        order_id = arguments["order_id"]
        status = lookup_order_in_database(order_id)
        return [TextContent(type="text", text=f"Order {order_id} status: {status}")]

This is the actual execution layer. The AI model never touches your database directly, it calls the tool through the standardized protocol, and your server controls exactly what happens and what gets returned.

Step four: run the server with the appropriate transport

from mcp.server.stdio import stdio_server

async def main():
    async with stdio_server() as (read_stream, write_stream):
        await server.run(read_stream, write_stream, server.create_initialization_options())

Local development typically uses stdio transport. Production deployments increasingly use Streamable HTTP, which lets the server run as a proper remote service rather than a local process, and it's the transport most enterprise MCP rollouts are standardizing around this year.

Step five: secure it before it goes anywhere near production

from mcp.server.auth import RequireAuth

@server.call_tool()
@RequireAuth(scopes=["orders:read"])
async def call_tool(name: str, arguments: dict, context):
    # verified caller identity is available in context
    ...

Every tool call should carry proper identity and scope verification. A recent industry analysis noted that in the MCP era, trust isn't established once at login, it's re-earned with every single tool call and data access an agent makes, which is a meaningfully different security posture than traditional API authentication.

What's Changing Right Now, as of This Week

This is genuinely current, not stale reporting. The next major MCP specification, dated 2026-07-28, has its release candidate locked and is shipping as the final spec this week. It introduces a stateless protocol core, an Extensions framework for adding capabilities without breaking existing implementations, a formal deprecation policy, and hardened authorization. If you're starting a new MCP implementation right now, building against this version rather than the November 2025 spec is the right call, since SDK maintainers are expected to support it within a defined validation window.

Common Mistakes Teams Are Making With Early MCP Adoption

Tool overexposure. Registering far more tools than a given workflow actually needs, which bloats the context an AI model has to reason over and increases the chance of it calling the wrong tool
Treating authentication as an afterthought. Standing up a functional MCP server quickly and only addressing proper scoped auth after it's already been connected to something sensitive
Ignoring observability. Deploying MCP servers into production workflows without tracking tool call success rates, latency, or error patterns, which makes debugging a failed agent workflow nearly impossible after the fact
Skipping the governance conversation. Rolling out MCP servers across a team or organization without agreeing on security, compliance, and identity controls up front, then retrofitting governance after adoption has already sprawled

Why This Matters Beyond the Hype Cycle

MCP exhibits genuine network effects. Every new MCP server makes every existing MCP client more capable, and every new client makes building an MCP server more worthwhile. That's a structurally different growth pattern than most developer tools, and it's part of why the ecosystem has grown as fast as it has.

For teams evaluating whether to invest in this now, the practical answer is usually yes, but scoped carefully. Building custom MCP connectors and servers for your own internal tools and data sources, wired into RAG development and existing LLM and GenAI workflows, is exactly the kind of foundational integration work that compounds in value as your AI features grow, rather than something that needs rebuilding from scratch every time a new tool gets added to the stack.

The Takeaway

The integration sprawl that used to make connecting AI to real business tools a slow, custom, one-off project each time is exactly what MCP was built to eliminate. With cross-vendor support, a maturing specification, and genuine network effects driving adoption, this has moved well past experimental territory into shared infrastructure that's shaping how serious AI applications get built this year.

Has your team started building or adopting MCP servers yet, or are you still wiring up custom integrations one tool at a time? Curious how far along everyone actually is with this.

Tags: #ai #llm #genai #tutorial

Your AI Feature Isn't Slow. Your Architecture Is Making It Feel Slow.

Emma Schmidt — Tue, 28 Jul 2026 10:30:54 +0000

You've seen it a hundred times. A user types a question into an AI feature, hits submit, and then just stares at a blank screen for eight seconds before a wall of text dumps onto the page all at once. That gap between "I asked something" and "something happened" is exactly where users lose patience, and it's rarely the model's fault. It's almost always a streaming problem, not a speed problem. This is one of the core patterns behind modern AI services and AI web application development, and getting it right in Next.js is simpler than most developers expect, especially now that the framework's App Router and Server Actions map almost perfectly onto how real-time AI responses actually need to flow.

Here's how to build a genuinely responsive, streaming AI feature in Next.js from the ground up, understanding what's actually happening under the hood instead of just wiring up a library and hoping.

Why Streaming Changed What "Fast" Means for AI Features

A non-streaming AI feature waits for the entire response to generate before showing anything. For short answers that's tolerable. For anything longer, a paragraph, a summary, a generated document, it means staring at a static loading spinner for several seconds with zero feedback that anything is actually happening.

Streaming flips this. Tokens appear as the model generates them, the same way ChatGPT-style interfaces work. The perceived speed difference is enormous even when the total generation time is identical, because the user sees progress immediately instead of waiting in silence.

The Real-World Difference This Makes

Picture a support tool generating a written summary of a long customer thread. The generation itself takes about six seconds either way.

Without streaming: the user submits, sees a static spinner, and waits the full six seconds staring at nothing. To someone who isn't sure the request even registered, six seconds of silence feels much longer than it actually is, and a meaningful number of users will refresh or click again out of doubt, sometimes triggering a duplicate request.

With streaming: the user sees the first words appear within a few hundred milliseconds, and text keeps flowing steadily until the summary is complete. The total time is identical, but it reads as fast because there's constant, visible progress instead of dead air.

Same backend, same model, same six seconds of actual compute. The only difference is architecture, and it's the difference between a feature that feels broken and one that feels instant.

Choosing Your Transport Mechanism

Before writing code, it's worth understanding the three real options for getting streaming data from server to client, since picking the wrong one for your use case creates real problems later.

Server-Sent Events (SSE)

One-directional, server to client only
Works over standard HTTP, no special infrastructure required
The natural fit for AI text generation, since the client only needs to receive tokens, not send data mid-stream

WebSockets

Bidirectional, full duplex communication
Better suited for genuinely interactive real-time features like collaborative editing or live multiplayer state
Overkill for most AI chat features, and adds real infrastructure complexity you don't need for one-way token streaming

Web Streams API (ReadableStream)

The lowest-level option, giving direct control over how chunks are read and processed
What the AI SDK and most Route Handler implementations are actually built on under the hood
Best when you need fine-grained control over parsing structured data mid-stream, like tool calls or JSON fragments

For the vast majority of AI chat and generation features, SSE built on top of the Web Streams API is the right choice. Reach for WebSockets only when you genuinely need bidirectional communication during generation, which is rarer than most implementations assume.

Step-by-Step: Building a Streaming AI Route Handler

Step one: set up the project

npx create-next-app@latest ai-streaming-app
cd ai-streaming-app

Select the App Router when prompted. This pattern depends on Route Handlers, which live specifically in the App Router architecture.

Step two: create the streaming Route Handler

// app/api/chat/route.ts
export async function POST(req: Request) {
  const { message } = await req.json();

  const stream = new ReadableStream({
    async start(controller) {
      const response = await callModelProvider(message);

      for await (const chunk of response) {
        controller.enqueue(new TextEncoder().encode(chunk));
      }
      controller.close();
    },
  });

  return new Response(stream, {
    headers: { "Content-Type": "text/event-stream" },
  });
}

The key detail here: the API key and the call to the model provider stay entirely server-side. Nothing sensitive ever ships to the client bundle, which matters both for security and for keeping your provider costs from being exposed.

Step three: consume the stream on the client

// app/components/ChatInput.tsx
'use client';
import { useState } from 'react';

export default function ChatInput() {
  const [response, setResponse] = useState('');

  async function handleSubmit(message: string) {
    setResponse('');
    const res = await fetch('/api/chat', {
      method: 'POST',
      body: JSON.stringify({ message }),
    });

    const reader = res.body?.getReader();
    const decoder = new TextDecoder();

    while (true) {
      const { done, value } = await reader!.read();
      if (done) break;
      setResponse((prev) => prev + decoder.decode(value));
    }
  }

  return <div>{response}</div>;
}

This is the piece that actually creates the typing effect. Each chunk arrives and appends to state as it comes in, instead of waiting for one giant response.

Step four: add proper error and cancellation handling

const controller = new AbortController();

async function handleSubmit(message: string) {
  try {
    const res = await fetch('/api/chat', {
      method: 'POST',
      body: JSON.stringify({ message }),
      signal: controller.signal,
    });
    // stream reading logic here
  } catch (err) {
    if (err.name === 'AbortError') {
      console.log('Generation cancelled by user');
    } else {
      setError('Something went wrong, please try again');
    }
  }
}

function handleCancel() {
  controller.abort();
}

A stop button that actually cancels an in-flight generation is a small detail users notice immediately when it's missing, and it costs very little to implement properly.

Step five: validate structured output when you need more than plain text

import { z } from 'zod';

const responseSchema = z.object({
  summary: z.string(),
  confidence: z.number(),
});

function validateResponse(raw: string) {
  const parsed = JSON.parse(raw);
  return responseSchema.parse(parsed);
}

Instead of hoping a model returns well-formed JSON, defining a schema and validating against it catches malformed output before it ever reaches your UI, which matters a lot once an AI feature is doing more than just displaying conversational text.

Step six: add rate limiting before this ever reaches production

// app/api/chat/route.ts
const requestCounts = new Map<string, number>();

export async function POST(req: Request) {
  const userId = getUserIdFromSession(req);
  const currentCount = requestCounts.get(userId) ?? 0;

  if (currentCount >= 20) {
    return new Response("Rate limit exceeded", { status: 429 });
  }

  requestCounts.set(userId, currentCount + 1);
  // proceed with streaming logic
}

This is a simplified example, a real implementation would use a proper store like Redis rather than an in-memory map, but the principle holds regardless of scale. Without rate limiting, a single user or a scripted abuse pattern can run your model provider costs up fast, and this is one of the most commonly skipped steps in early AI feature builds.

Step seven: track cost per feature, not just in aggregate

function logUsage(feature: string, tokenCount: number, model: string) {
  const cost = tokenCount * getCostPerToken(model);
  analytics.track("ai_usage", { feature, tokenCount, cost, model });
}

Breaking cost down by feature rather than watching one aggregate monthly number lets you catch a specific feature quietly ballooning in usage long before it shows up as a surprise on the bill.

Common Mistakes Worth Avoiding

Forgetting to handle stream errors mid-response, leaving the UI stuck in a permanent loading state if the connection drops partway through
Skipping the cancellation handling entirely, which leaves users stuck watching a response they no longer want finish generating
Calling the model provider directly from client-side code, which exposes API keys and lets costs run up uncontrolled
Not validating structured output, and discovering malformed data only when it breaks something downstream in production
Shipping without rate limiting and finding out the hard way when a single user or bot script drives up provider costs overnight
Choosing WebSockets by default out of habit when SSE would be simpler, cheaper to run, and perfectly sufficient for one-way token streaming

A Quick Self-Check Before You Ship

Does your UI show visible progress within the first second, or does it sit on a static spinner while waiting for the full response?
Can a user actually cancel a generation mid-stream, or does clicking away just leave the request running in the background?
Is there a rate limit in place, or is the endpoint currently wide open to unlimited requests per user?
Are you tracking cost per feature, or would a sudden spike in usage only show up once the monthly bill arrives?

Why This Is Worth Building Properly, Not Just Quickly

Getting a demo chat working in an afternoon is genuinely easy now. Getting it production-ready, proper error handling, rate limiting, cost monitoring, structured output validation, and a UI that gracefully degrades when something goes wrong, is where most teams underestimate the actual scope of the work. Custom AI development and API integration work focused specifically on shipping these features at production scale tends to close that gap far faster than assembling it piece by piece under a deadline, especially for teams integrating multiple providers or building on top of an existing enterprise codebase.

The Takeaway

The gap between an AI feature that feels instant and one that feels sluggish usually isn't the model's raw speed, it's whether the response streams token by token or dumps all at once after a long, silent wait. Next.js's architecture, particularly Server Actions and Route Handlers, maps naturally onto this pattern, which is exactly why it's become the default framework for AI-powered web applications this year.

Has your team already shipped a streaming AI feature in production, or are you still working through the plumbing on this one? Curious what's tripped people up most.

Your ML Model Doesn't Need to Be Bigger. It Needs to Be Smaller and Smarter.

Emma Schmidt — Mon, 27 Jul 2026 06:37:54 +0000

For the last few years, the assumption in machine learning was simple: bigger models mean better results, so scale up and worry about the bill later. That assumption is quietly breaking down in 2026. The real shift happening across serious ML teams right now isn't about building bigger models, it's about deployment efficiency, running smaller, specialized models that cost a fraction as much to run while matching or beating general-purpose performance on the specific task they're actually built for. This is exactly where a lot of ML services work is heading this year, helping teams figure out which tasks genuinely need a massive general model and which ones a smaller, purpose-tuned one handles just as well for a fraction of the cost.

This shift has a name, small language models, and understanding how to actually build and deploy one is quickly becoming a core ML skill rather than a niche specialty.

Why Bigger Stopped Automatically Meaning Better

For a specific, well-defined task, a massive general-purpose model is often carrying enormous amounts of capability you never actually use. Ask a giant model to classify customer support tickets into five categories, and it's spending compute capacity it could use to write poetry or debug Python, none of which the task needs.

A few forces are pushing the industry toward smaller, specialized models instead:

Inference cost adds up fast at real scale. A model that's cheap to query once becomes expensive very quickly across millions of daily requests, and a smaller model tuned for the specific task can cut that cost dramatically
Latency matters more as AI moves into real-time features. A smaller model responds faster, which matters enormously for anything user-facing where a slow response actively hurts the experience
On-device and edge deployment is only realistic with smaller models. A model that needs to run on a phone, a browser, or an embedded device simply can't be enormous
Specialized accuracy often beats general capability. A model trained specifically on your domain's data and terminology frequently outperforms a general model on that exact task, even at a fraction of the parameter count

The Two Core Techniques Worth Actually Understanding

Two techniques make this shift practical, and both are worth understanding at a working level even if you're not building the tooling yourself.

Quantization reduces the precision of a model's internal numbers, typically from 32-bit floating point down to 8-bit or even 4-bit representations. This shrinks the model's memory footprint and speeds up inference, often with only a small, carefully managed loss in accuracy.

Distillation trains a smaller "student" model to mimic the behavior of a larger "teacher" model. Instead of learning from raw data alone, the student learns from the teacher's outputs, effectively compressing what the larger model learned into a much smaller package.

Together, these two techniques are why a well-tuned small model can now handle tasks that would have required a massive general model just a couple of years ago.

A Practical Walkthrough: Quantizing a Model for Deployment

Let's walk through a basic quantization workflow using a common open-source library, the kind of step that turns a model too large for practical deployment into one that runs efficiently in production.

Step one: load your baseline model

from transformers import AutoModelForSequenceClassification, AutoTokenizer

model_name = "your-base-model"
model = AutoModelForSequenceClassification.from_pretrained(model_name)
tokenizer = AutoTokenizer.from_pretrained(model_name)

Step two: apply post-training quantization

import torch

quantized_model = torch.quantization.quantize_dynamic(
    model, {torch.nn.Linear}, dtype=torch.qint8
)

This converts the model's linear layers to 8-bit integer precision, typically shrinking the model size significantly and speeding up inference on CPU, often with minimal accuracy loss for well-suited tasks.

Step three: benchmark before trusting the result

import time

def benchmark(model, input_ids, runs=50):
    start = time.time()
    for _ in range(runs):
        with torch.no_grad():
            model(input_ids)
    return (time.time() - start) / runs

original_latency = benchmark(model, sample_input)
quantized_latency = benchmark(quantized_model, sample_input)
print(f"Original: {original_latency:.4f}s, Quantized: {quantized_latency:.4f}s")

Never skip this step. The whole point of quantization is a measurable efficiency gain, and you need real numbers on your actual hardware and actual data, not an assumption borrowed from a blog post.

Step four: validate accuracy didn't quietly degrade

def evaluate_accuracy(model, test_dataset):
    correct = 0
    for inputs, label in test_dataset:
        with torch.no_grad():
            prediction = model(inputs).logits.argmax()
        correct += (prediction == label)
    return correct / len(test_dataset)

print(f"Original accuracy: {evaluate_accuracy(model, test_set):.3f}")
print(f"Quantized accuracy: {evaluate_accuracy(quantized_model, test_set):.3f}")

A quantized model that's faster but meaningfully less accurate isn't actually a win. This comparison step is what turns quantization from a guess into a real, measured engineering decision.

Where Small Language Models Are Already Winning

This isn't theoretical. Real deployments are already showing the pattern clearly.

Customer support ticket routing using a small, domain-tuned classifier instead of a general-purpose model call for every incoming ticket, cutting both latency and per-request cost significantly
On-device features in mobile apps, where a compact model handles tasks like text summarization or intent detection locally, without a round trip to a server at all
Manufacturing and IoT anomaly detection, where small, efficient models run directly on edge hardware near the sensors generating the data, since sending everything to the cloud for a giant model to process isn't fast enough for real-time decisions
High-volume, narrow classification tasks like spam detection or content moderation, where a small specialized model handles the bulk of routine cases and only escalates genuinely ambiguous ones to a larger model

A Quick Framework for Deciding When to Go Small

Before defaulting to the largest available model, ask these questions honestly.

Is this task narrow and well-defined, or does it genuinely require broad, general reasoning across many domains?
Do we have enough quality training or distillation data specific to this task to teach a smaller model well?
Does latency or on-device deployment actually matter for this specific feature?
What's the real cost difference at our expected production volume, not just at demo scale?

If a task is narrow, well-defined, and running at real volume, a small, specialized model is very often the better engineering choice, not a compromise.

Common Mistakes Worth Avoiding

Quantizing a model and shipping it without actually benchmarking the accuracy tradeoff on real evaluation data
Assuming distillation is a one-time step rather than an ongoing process that needs revisiting as the underlying task or data shifts over time
Defaulting to the largest general model out of convenience for tasks that a much smaller, cheaper model would handle just as well
Skipping proper monitoring after deployment, since a smaller model can drift or degrade in production just like any other model, and needs the same kind of ongoing evaluation

Why This Is Worth Doing Properly, Not Just Quickly

Getting the quantization, distillation, and evaluation pipeline right takes real ML engineering discipline, not just running a library function once and hoping for the best. For teams without a dedicated MLOps function already in place, building this properly alongside shipping product features is a genuinely heavy lift. Custom model development, deployment pipeline setup, and ongoing MLOps support built around this exact efficiency-first approach tend to get a team to a well-benchmarked, production-ready small model far faster and more reliably than assembling it piecemeal under deadline pressure.

The Takeaway

The biggest available model was never automatically the right choice, it just looked that way while nobody was measuring the actual cost and latency tradeoffs closely. As ML moves deeper into real-time, on-device, and high-volume production use cases, the teams pulling ahead are the ones treating model size as a deliberate engineering decision, matched carefully to the task, rather than a default setting left on maximum.

Is your team still defaulting to the largest available model for every task, or have you already started matching model size to the actual job? Curious how far along different teams are with this shift.

Your Chatbot Answered Perfectly Yesterday. Today It's Confidently Making Things Up. Nobody Noticed.

Emma Schmidt — Fri, 24 Jul 2026 06:54:44 +0000

Your uptime dashboard is green. Response times look normal. Error rates are near zero. And yet, somewhere in production right now, your AI feature might be giving out subtly wrong answers, hallucinated policy details, a drifted tone, an answer that's technically valid but wrong for your actual use case, and none of your existing monitoring would ever catch it. This is exactly the blind spot pushing AI services from a bolted-on feature toward something that needs the same rigor as any other production infrastructure, with dedicated observability, evaluation, and monitoring built in from the start rather than added after something goes wrong.

This gap has a name now: LLM observability, and it's becoming one of the fastest-growing corners of AI infrastructure this year for a very simple reason. Traditional monitoring was built for deterministic software. AI models aren't deterministic, and the old playbook doesn't catch the failures that actually matter.

Why Your Existing Monitoring Can't See This

Traditional application monitoring answers one question well: did the request succeed, and how fast. That's genuinely useful, and it's also completely blind to the specific way AI systems fail.

A response can be fast, formatted correctly, and still be wrong. A hallucinated fact, a policy detail that's outdated, or an answer that sounds confident but references the wrong context all return a perfectly healthy 200 status code
The same prompt can produce different quality on different days. A model that answered a question correctly last week can drift into a subtly wrong pattern this week with no code change on your end at all
Multi-step AI workflows hide their failures in the middle. When a system retrieves documents, reasons over them, and calls tools before producing a final answer, a failure in any middle step can still produce an output that looks fine on the surface
Cost and token usage can balloon silently. A workflow that used to cost a few cents per request can quietly triple in cost as prompts grow or retrieval pulls in more context than intended, with nothing in a standard dashboard flagging it

None of these show up as an outage. They show up as a slow, invisible erosion of quality that nobody notices until a customer complains, or worse, until nobody notices at all.

The Four Things Real AI Observability Actually Tracks

Getting real visibility into an AI system means watching more than the usual infrastructure signals.

System health. The traditional layer, latency, error rates, uptime, still matters and still needs watching
Model performance and quality. Accuracy against known good answers, hallucination rates, and semantic drift over time, not just whether a response was returned
Cost tracking. Token usage and per-request cost broken down by feature, so a quiet cost creep gets caught before it shows up as a surprise on the monthly bill
Behavioral tracing. Full visibility into multi-step workflows, what was retrieved, what reasoning path was taken, what tools were called, so a wrong answer can actually be debugged instead of shrugged at

A Practical Way to Start Building This

You don't need a massive platform overhaul to start closing this gap. Here's a reasonable starting sequence.

Step one: log the full trace, not just the final output

def log_llm_call(prompt, retrieved_docs, response, metadata):
    trace = {
        "timestamp": datetime.utcnow(),
        "prompt": prompt,
        "retrieved_context": retrieved_docs,
        "response": response,
        "token_count": metadata["tokens"],
        "latency_ms": metadata["latency"],
        "model_version": metadata["model"],
    }
    trace_store.save(trace)

Without the full trace, a bad output down the line is nearly impossible to debug. Was it a bad retrieval, a bad prompt, or a bad model response? You can't tell from the final answer alone.

Step two: run automated quality scoring on a sample of live traffic

def score_response(prompt, response, retrieved_docs):
    return {
        "faithfulness": check_grounded_in_context(response, retrieved_docs),
        "relevance": check_relevance(prompt, response),
        "hallucination_risk": detect_unsupported_claims(response, retrieved_docs),
    }

Scoring every single production response can be expensive, but sampling a meaningful percentage catches drift long before it becomes a visible pattern to users.

Step three: set alerts on quality signals, not just infrastructure signals

if quality_scores["faithfulness"] < 0.7:
    alert_team(f"Faithfulness score dropped: {quality_scores}")

A latency alert tells you something broke. A faithfulness or drift alert tells you something is quietly getting worse, which is the failure mode that actually damages user trust over time.

Step four: track cost per feature, not just cost in aggregate

cost_by_feature = defaultdict(float)
cost_by_feature[feature_name] += token_count * cost_per_token

A single feature quietly ballooning in token usage is easy to miss in an aggregate monthly bill and very easy to catch when broken down by feature and tracked over time.

Step five: close the loop by feeding failures back into evaluation

Every flagged failure should become part of an ongoing evaluation dataset, not a one-off fire drill. This is what actually improves the system over time instead of just reacting to the same category of failure repeatedly.

Real Numbers Worth Knowing

The scale of this gap is bigger than most teams assume. A large share of enterprises now require AI monitoring in production, yet a majority cite a lack of adequate observability tooling as a major barrier to actually having it. That gap between what teams know they need and what they've actually built is exactly where the invisible failures described above tend to live.

Evaluation overhead itself is a real cost too. Adding a full evaluation pass to a live request can meaningfully increase response latency, which is part of why the emerging best practice for 2026 is combining lightweight in-line scoring on every request with deeper evaluation sampling run asynchronously, rather than trying to fully evaluate every single response in the live path.

Common Mistakes Teams Make Getting Started

Treating LLM observability as identical to traditional APM and only tracking latency and error rates, missing quality entirely
Scoring 100 percent of production traffic with an expensive evaluation model, which quietly doubles response latency and cost
Building alerts only for hard failures and missing the slow, gradual drift that's actually the more common failure pattern
Never closing the loop between a caught failure and an updated evaluation dataset, so the same category of mistake keeps recurring

Why This Is Worth Building Properly

Getting this right touches more than one layer, evaluation design, tracing infrastructure, alerting thresholds, and cost monitoring all need to work together rather than existing as disconnected tools. For teams without dedicated MLOps or AI infrastructure specialists on staff already, building this from scratch while also shipping product features is a genuinely heavy lift. Custom AI integration and observability setup, done by people who've built this exact monitoring layer before, tends to get a team from flying blind to genuinely production-grade visibility far faster than assembling it piecemeal under deadline pressure.

The Takeaway

A green dashboard doesn't mean your AI feature is actually working well, it means the parts of it that traditional monitoring knows how to measure are working. The parts that actually matter to users, whether the answer was true, relevant, and useful, live in a layer most teams still aren't watching. That gap is exactly what LLM observability exists to close, and closing it early is a lot cheaper than discovering the failure the way most teams still do, from a customer complaint.

Is your team actually tracking AI output quality in production, or is your monitoring stack still only watching the infrastructure layer? Curious how many of us have actually closed this gap yet.

Your App Is Fast Enough. It's Also Quietly Burning More Energy Than It Needs To.

Emma Schmidt — Thu, 23 Jul 2026 12:12:35 +0000

Tags: #sustainability #cloudcomputing #webdev #discuss
Somewhere in your cloud bill this month, there's a number nobody on your team has ever really looked at: how much energy your application actually consumes to do its job. Not cost, energy. Every unnecessary database query, every bloated Docker image, every inefficient loop running at scale is drawing real power from a real data center, and most engineering teams have never once measured it. This gets even more pressing once AI services enter the picture, since AI inference is one of the most energy-hungry workloads a modern application can run, and it's exactly where a lot of this invisible waste tends to hide first.

That's starting to change fast, and not because of idealism. Green coding, writing and architecting software specifically to minimize energy consumption, has moved from a nice-to-have sustainability talking point to a genuine engineering discipline businesses are being pushed toward from multiple directions at once.

Why This Is Suddenly Everyone's Problem

A few forces are converging here, and none of them are going away.

Cloud costs and energy consumption are now directly linked in a way they weren't a few years ago, as providers increasingly price compute in ways that reflect actual power draw, not just raw usage time
Regulatory pressure is rising in multiple regions requiring companies to report on and reduce their digital carbon footprint, and software energy use is a growing line item in those reports
AI workloads specifically have made this urgent. Training and running AI models is extraordinarily energy intensive, and companies bolting AI features onto every product are watching their infrastructure energy costs climb in ways that are hard to ignore
Customers and investors are starting to ask. Sustainability reporting increasingly includes technology stack efficiency, not just the usual categories like travel and office energy use
Data center capacity itself is becoming a real constraint, with some regions seeing power grid strain directly tied to the growth of compute-heavy workloads, which puts pressure back on the software layer to use less of it

None of this requires anyone to care personally about the planet to take it seriously. It's becoming a real line item, a real compliance risk, and a real cost center.

What Inefficient Code Actually Looks Like at Scale

Here's the part that surprises people. None of this requires exotic new practices. It's mostly ordinary engineering habits that were never a priority because energy cost was invisible.

N+1 queries running millions of times a day across a large user base, each one drawing real compute and real power for work that could've been a single efficient join
Polling instead of event-driven updates, where a client checks for new data every few seconds indefinitely, burning compute around the clock for data that changes rarely
Oversized container images that take longer to build, deploy, and run than they need to, multiplied across every deployment and every autoscaling event
Unoptimized AI inference calls, sending far more tokens or running a larger model than the task actually requires, when a smaller, purpose-built model would do the job with a fraction of the energy draw
Idle infrastructure left running in staging and development environments around the clock, consuming power for workloads nobody's actually using outside business hours
Unbounded background jobs that retry endlessly or run on overly aggressive schedules, quietly consuming compute for work that could run a fraction as often

Individually, each of these looks like a minor inefficiency. At the scale most production systems run at, they add up to a genuinely significant amount of wasted energy and wasted money, quietly, every single day.

A Real-World Example of How This Adds Up

Take a mid-sized ecommerce platform serving a few million requests a day. Picture a single product page that fires an N+1 query pattern loading reviews, one that polls every three seconds for inventory updates that actually change a few times an hour, and a recommendation feature calling a large general-purpose AI model when a lightweight, purpose-built model would return the same quality of result.

None of those three issues, on their own, would ever show up as an incident. No outage, no page, no angry customer email. But multiplied across millions of page loads a day, that's a meaningful chunk of unnecessary compute running continuously, month after month, for work that could've been a fraction of the size with no visible change to the user experience at all. That's the actual shape of this problem in most real production systems: not one dramatic failure, but a thousand small, invisible ones running constantly.

A Practical Starting Point

You don't need a sustainability team to start improving this. A few concrete places to look first.

Audit your database query patterns
Look specifically for N+1 patterns and queries running far more often than the data actually changes. This is usually the single highest-impact place to start, because database load scales directly with both cost and energy draw.

Replace polling with event-driven architecture where you can
A webhook or a push notification that fires only when something actually changes uses a fraction of the compute of a client checking in every few seconds forever.

Right-size your AI model usage
Not every task needs the largest, most capable model available. Matching model size to task complexity is one of the fastest-growing cost and energy savings levers teams are finding this year, precisely because so many teams default to the biggest model out of convenience rather than necessity.

Shut down idle environments automatically
Staging and dev environments running 24/7 for workloads only used during business hours are pure waste. Automated shutdown schedules are a low-effort, high-impact fix.

Batch and schedule background work intelligently
Jobs that run every minute out of habit rather than necessity are an easy, low-risk place to cut unnecessary compute cycles without touching anything user-facing.

Measure before you optimize
Most teams have never actually measured their application's energy footprint because the tooling to do it wasn't mainstream until recently. You can't meaningfully improve what you've never measured.

A Quick Self-Check

Before deciding this doesn't apply to your team, answer these honestly.

Do you know which endpoints in your application generate the most database load relative to how often that data actually changes?
Are any of your staging or dev environments running continuously, with nobody checking whether they need to be?
Is your team defaulting to the largest available AI model for every task, regardless of how simple the task actually is?
Has anyone ever pulled a report on your cloud provider's energy or carbon metrics, or does that dashboard exist unopened?

If more than one of these gave you pause, there's likely real, low-risk savings sitting in your stack right now, unmeasured and unaddressed.

Common Questions Teams Ask When Starting This

Does this require rewriting our whole application?
Almost never. Most of the highest-impact fixes, like resolving N+1 queries or shutting down idle environments, are targeted, low-risk changes rather than architectural rewrites.

Will this slow down feature development?
An initial audit takes focused time, but the fixes that come out of it are usually incremental and can be folded into normal sprint work rather than requiring a dedicated multi-month initiative.

Is this only relevant for very large-scale applications?
The savings scale with traffic, but the underlying inefficiencies exist at any scale. Catching them early is actually easier and cheaper than untangling them after years of accumulated scale.

Why This Matters Beyond the Obvious

There's a business case here independent of any environmental motivation. Energy-efficient code tends to correlate strongly with genuinely well-architected code. The same discipline that reduces unnecessary compute cycles also reduces latency, reduces cloud spend, and reduces the blast radius of scaling problems under real traffic. Teams chasing energy efficiency usually end up with a faster, cheaper, more resilient system as a side effect, not a tradeoff.

This is also exactly the kind of work that benefits from a proper audit rather than scattered fixes. Untangling inefficient query patterns, right-sizing infrastructure, and rearchitecting around event-driven patterns across an existing production system touches database design, cloud architecture, and AI services integration all at once, which is a lot to take on internally while also shipping new features on the usual roadmap. A dedicated cloud architecture and performance audit, done by people who've mapped this exact kind of waste before, tends to surface savings that are easy to miss when you're deep inside your own codebase every day.

The Takeaway

The energy your code consumes was always real, it just wasn't visible on a dashboard anyone was watching. As regulation tightens, AI workloads grow, and cloud providers price compute closer to its real power draw, that invisibility is disappearing fast. The teams getting ahead of this now aren't doing it out of idealism, they're doing it because efficient code was always going to be cheaper, faster, and more resilient too, the environmental benefit just happens to come along for free.

Has your team ever actually measured your application's energy footprint, or is it still an invisible number buried somewhere in a cloud bill nobody's dug into? Curious how many of us are actually tracking this yet.

Prompt Engineering Is Dead. Context Engineering Is What Actually Fixes Your AI's Wrong Answers.

Emma Schmidt — Wed, 22 Jul 2026 10:54:05 +0000

You've tuned the prompt a dozen times. Added examples, adjusted the tone, tried three different phrasings of the same instruction. The AI still hallucinates a fact, misses context from earlier in the conversation, or confidently answers a question using the wrong data. If this sounds familiar, the problem was never the prompt. It's what's called context engineering, and it's quietly become one of the most important skills in AI development this year. Teams that get this right often lean on structured AI development and integration services specifically because getting the architecture right the first time avoids months of trial-and-error prompt tweaking that never actually fixes the root cause.

Here's what context engineering actually is, why it matters more than clever prompting, and how to build it into a real application step by step.

Why Prompt Engineering Alone Stops Working

Prompt engineering treats the model like a black box you talk to more cleverly. Context engineering treats the entire system around the model, what data it sees, in what order, and how it's structured, as the thing you actually design.

Here's the distinction that matters:

Prompt engineering optimizes the instruction you give the model in a single message
Context engineering optimizes everything the model has access to when it generates that response: retrieved documents, conversation history, system instructions, tool outputs, and how all of that is assembled and prioritized

A perfectly worded prompt pointed at the wrong or poorly organized context still produces a wrong answer. This is why teams that only tune prompts eventually hit a wall, no phrasing fixes a model that's missing the right information entirely.

The Core Problem You're Actually Solving

Every AI application built on top of a language model faces the same limitation: a fixed context window and no persistent memory between calls. Context engineering is the discipline of deciding what goes into that limited space and how it's organized so the model can actually use it well.

Get this wrong and you'll see familiar symptoms:

The model confidently answers using outdated or irrelevant retrieved data
Earlier parts of a long conversation get silently dropped or ignored
Retrieved documents get stuffed in with no clear structure, and the model can't tell what's actually relevant
Adding more context makes answers worse, not better, because signal gets buried in noise

Building a Real Context Pipeline, Step by Step

Let's walk through building this properly, using a support chatbot pulling from a knowledge base as the example.

Step one: separate your context into clear categories

context = {
    "system_instructions": "You are a support assistant for Acme SaaS...",
    "retrieved_docs": [],       # from your knowledge base
    "conversation_history": [], # prior turns in this session
    "user_profile": {},         # account tier, past tickets, etc
    "current_query": ""
}

Treating these as separate, labeled components instead of one giant blob of text is the single biggest shift from prompt engineering to context engineering. It lets you control, debug, and prioritize each piece independently.

Step two: retrieve only what's actually relevant

def retrieve_context(query, top_k=3):
    query_embedding = embed(query)
    results = vector_db.search(query_embedding, top_k=top_k)
    return [r for r in results if r.score > 0.75]

Notice the relevance threshold. Pulling in five loosely related documents to "give the model more to work with" usually makes answers worse, not better. Fewer, highly relevant documents consistently outperform a large pile of mediocre ones.

Step three: manage conversation history deliberately, not by just appending everything

def build_history(messages, max_turns=6):
    recent = messages[-max_turns:]
    if len(messages) > max_turns:
        summary = summarize(messages[:-max_turns])
        return [{"role": "system", "content": f"Earlier context: {summary}"}] + recent
    return recent

Long conversations without this kind of management silently push early, important context out of the window entirely. Summarizing older turns instead of dropping them keeps continuity without blowing the token budget.

Step four: assemble everything with clear structure, not a wall of text

def build_prompt(context):
    return f"""
System: {context['system_instructions']}

Relevant documentation:
{format_docs(context['retrieved_docs'])}

Conversation so far:
{format_history(context['conversation_history'])}

User profile: {context['user_profile']}

Current question: {context['current_query']}
"""

Clear section labels matter more than they seem like they should. Models handle structured, clearly delineated context noticeably better than the same information dumped in as one undifferentiated paragraph.

Step five: test with adversarial and edge-case queries, not just the happy path

test_cases = [
    "question referencing something from 10 turns ago",
    "question where retrieved docs are slightly outdated",
    "question with no relevant docs in the knowledge base at all",
]

This is the step most teams skip. A context pipeline that works great on clean, simple test queries often falls apart the moment a real user asks something messy, which is exactly when it matters most.

Common Mistakes Worth Knowing Before You Build This

Context window stuffing. Cramming in every possibly-relevant document instead of the few genuinely relevant ones, which drowns out the signal
No relevance filtering on retrieval. Returning the top results regardless of how weak the match actually is
Flat, unstructured context. Pasting everything into one block of text with no labeled sections for the model to anchor on
Ignoring token budget until it breaks. Not tracking how much of the context window is consumed by each component until responses start silently truncating
No evaluation loop. Shipping the pipeline without a way to systematically test whether context changes actually improved or hurt answer quality

The Takeaway

If your AI feature is giving inconsistent or wrong answers and you've already spent real time tuning the prompt, the prompt was probably never the actual problem. Look at what the model can see when it answers, how it's retrieved, how it's structured, and how much of it is genuinely relevant. That's where the real fix usually lives.

Have you run into this wall yet, prompt tuning stops helping and the real fix turns out to be the context pipeline underneath it? Curious what broke it open for you.