DEV Community: PriyanshuValiya

Bigger Context Windows Aren't Making Your AI Smarter (And Here's Why)

PriyanshuValiya — Tue, 01 Sep 2026 05:10:51 +0000

Every few months, another model ships with a bigger context window.

1M tokens. 2M. Someone's already at 10M.

The pitch is always the same:

Feed it more, get better answers.

I ran into the opposite while building a retrieval system designed to enforce coding guidelines before code generation:

More context in, worse output out.

Turns out, that's not a fluke.

It's a consequence of how transformers work.

The Problem: Context Rot

Dump a 300-page PDF into a prompt and you'd expect the model to have "read the whole thing."

What actually happens is degradation.

Studies from Chroma and Anthropic show a similar pattern: as the surrounding context grows, reasoning accuracy on the actual task can drop.

Not because the model literally forgot the tokens. They're technically still in the context.

The problem is that the model is reasoning over a much noisier signal.

More context doesn't automatically mean more useful context.

Lost in the Middle

There's a well-documented effect where models are better at retrieving information from the beginning or end of a long prompt than from the middle.

If the fact you need is buried somewhere in the middle of a large document, retrieval and reasoning accuracy can drop significantly.

And that's not a rounding error.

It's the difference between:

a demo that impresses a stakeholder, and
a production system that quietly gives the wrong answer.

The model might technically have access to the information.

That doesn't mean it can reliably use it.

Needle in a Haystack Is the Wrong Benchmark

A lot of the "look how big our context window is" marketing leans heavily on needle-in-a-haystack tests.

The setup is simple:

Take a huge document.
Hide one specific fact somewhere inside it.
Ask the model to find that fact.

That's useful for measuring retrieval.

But it doesn't tell us much about reasoning across a large context.

Real-world tasks are usually more like:

Find three relevant pieces of information from three different sections, understand how they relate, and use them to make a decision.

A model can ace a needle-in-a-haystack benchmark and still fall apart when it needs to synthesize multiple pieces of information scattered across the context.

Finding one needle isn't the same as understanding the haystack.

Why This Happens: It's Architectural, Not a Bug

This comes down to the attention mechanism itself.

At a high level, tokens attend to other tokens to determine what's relevant.

As the amount of input grows, there are more relationships to reason over.

The model now has to distinguish the useful signal from an increasingly large amount of surrounding information.

There's no simple patch coming for this.

It's not necessarily a broken mechanism.

It's the mechanism doing what it was designed to do, just at a scale where the signal-to-noise ratio starts working against you.

And that's an important distinction.

The problem isn't simply:

"The model isn't smart enough."

The problem can be:

"We're giving the model too much information to reason over reliably."

What Actually Helps (and Where It Breaks)

RAG is the obvious mitigation.

Instead of dumping everything into the prompt, filter the data before it reaches the model.

Retrieve only the information that's likely to matter.

It works.

But RAG has its own failure mode.

Fetch the wrong chunk, or fetch a pile of "almost relevant" chunks, and you've recreated the same dilution problem you were trying to avoid.

I hit this directly while building a guideline-retrieval system.

A surprising amount of the engineering work wasn't the retrieval itself.

It was ranking.

The system needed to distinguish:

highly relevant chunks
partially relevant chunks
superficially similar chunks
completely irrelevant chunks

Because if you return ten "almost relevant" chunks alongside the one chunk that actually matters, you've made the model's job harder again.

Retrieval quality isn't just about finding relevant information.

It's also about not returning information that looks relevant but isn't.

Context Engineering and Recursive Models

Newer approaches like context engineering and recursive language models are trying to address the problem from a different angle.

Instead of asking:

"How do we fit more tokens into the context window?"

the question becomes:

"How should the model ingest, organize, compress, and reason over information?"

That's a much more interesting direction.

These approaches are still relatively early, but they're worth watching because they challenge the assumption that the solution to context limitations is simply more context.

The Takeaway

Sam Altman has talked about AI eventually having perfect context over your entire life.

Maybe that happens.

But there's a real gap right now between that promise and what a transformer can reliably reason over, even with tens of thousands of tokens.

If you're building anything RAG-dependent or long-context-dependent, that gap is where a lot of your bugs are going to come from.

Not necessarily because the model is "not smart enough."

But because the architecture is doing exactly what it's built to do at a scale where more information can actually make the answer worse.

The question isn't:

How much context can the model handle?

It's:

How much context can the model reliably use?

Those are very different numbers.

If you're fighting this in production, what has actually moved the needle for you?

Better chunking? Re-ranking? Context compression? Something else?

Your E-Commerce Backend Doesn't Need a New UI. It Needs an MCP Server.

PriyanshuValiya — Thu, 13 Aug 2026 12:13:30 +0000

Every ecommerce admin panel I've worked with has the same shape: a sidebar of menus, a table with filters, a "search" box that only matches exact strings. If the owner wants to know "which products are low on stock and haven't sold in 30 days," they either learn the filter UI cold or they ping a developer to write a query.

That's the gap MCP closes. Not by replacing the admin panel by giving an LLM a typed, permissioned door into the same business logic your REST API already exposes.

The setup most people get wrong

The instinct is to let the model touch the database directly, or worse, generate SQL on the fly. Don't. Your business logic already lives in a service layer for a reason validation, ownership checks, side effects like inventory recalculation. MCP tools should call that layer, not go around it.

If you already have a layered backend (controllers → services → repositories), you're not rebuilding anything. You're adding a third entry point next to your REST API:

 Frontend  ─┐
            ├─> Business Services ─> Repository ─> DB
MCP Server ─┘

Each MCP tool is a thin wrapper: getLowStockProducts, getOrdersByDateRange, updateProductPrice. The tool description is doing real work here it's the only documentation the model has, so vague names and descriptions produce vague tool calls. I've found being almost annoyingly explicit ("returns products where stock < threshold AND no sale in the last N days, sorted ascending by stock") gets far more reliable results than a clever one-liner.

Why voice/sentence queries actually work here

This isn't a chatbot bolted onto a search bar. When a business owner says "which products are low on stock and haven't sold this month," Claude Desktop (or any MCP client) doesn't guess an answer it decides which tool(s) to call, calls them with structured params, and reasons over the actual returned data. The natural language is just the interface; the tool call is still a real, typed function invocation hitting your real backend.

That distinction matters for trust. You're not asking the owner to believe an LLM's memory. You're asking it to translate intent into the same API call a developer would write by hand.

Auth is not optional, and it's not free

The part that'll bite you if you skip it: MCP tools inherit whatever access the calling session has, so you need the same ownership/authorization checks your API already enforces not a parallel, weaker set for "the AI path." I've been working through exactly this on a Jira-style project management backend right now, and the honest answer is auth design took longer than writing the tools themselves. Worth it. An MCP server with no scoping is just a very polite SQL injection vector.

What I'd actually start with

Pick 3–5 read-only queries first (stock levels, recent orders, revenue by category). Ship those before touching any write operations.
Log every tool call with the same audit trail you'd want for a human admin action.
Write tool descriptions like you're documenting for someone who's never seen your schema because that's exactly the situation the model is in. The interesting part isn't "AI can query your database now." It's that non-technical operators get to ask business questions in the language they already think in, against data that was previously locked behind a UI someone else designed.

If you're exposing an existing backend this way, I'd genuinely like to hear what tripped you up auth scoping, tool granularity, or something else entirely.

Running Claude Code on AWS Bedrock Instead of the Anthropic API (a weekend rabbit hole)

PriyanshuValiya — Tue, 04 Aug 2026 04:42:25 +0000

I've been learning Claude Code for agentic development lately, but I didn't want to burn through paid API credits while I was still figuring out how the tool actually works. Every wrong prompt, every abandoned session, every "let me just try this real quick" costs money when you're hitting the Anthropic API directly.

So I spent a weekend building a small LiteLLM proxy that lets the Claude Code CLI talk to Amazon Nova on AWS Bedrock instead. Same agent loop, same tool calling, same workflow you'd get with Claude Code normally just pointed at a different backend, running on AWS credits instead of API billing.

The problem I was actually trying to solve

I looked at the usual options for cheap/free agentic coding practice and none of them held up for real sessions:

Ollama locally - needs a beefy machine with real RAM and GPU headroom. Most people don't have that sitting around, and I didn't want to buy it just to practice.
Free-tier providers like OpenRouter - you hit token limits within a handful of basic prompts. Not nearly enough runway for long-running agentic workflows where the model is calling tools back and forth for a while. I wanted something that let me learn tool use, agent loops, and long context handling without either of those walls showing up mid-session.

What I built

A LiteLLM proxy sitting between Claude Code's CLI and AWS Bedrock, translating requests so Claude Code thinks it's talking to Anthropic's API while it's actually routing to Amazon Nova.

A few things I locked down on purpose:

No artificial session caps. It's self-hosted, so there's no vendor imposed limit on how long or how often I can run sessions.
Everything runs on AWS credits, not API billing.
Roughly 20–35% cheaper per token than Claude Haiku pricing, based on current Bedrock rates.
IAM scoped to a single model. I locked the role down so the proxy physically can't invoke anything more expensive, even by accident.
AWS Budget alerts wired in, so I'm not finding out about a runaway bill after the fact. The Docker image and config are public if you want to poke at it yourself: https://github.com/PriyanshuValiya/Claude-Code/blob/main/README.md

Where this actually helps (and where it doesn't)

This is genuinely one of the more practical setups I've found for running long agentic sessions without either buying hardware or hitting a free-tier wall every ten minutes. If you're trying to internalize how these coding agents plan, call tools, and recover from errors, having room to run long sessions matters more than having the "best" model underneath.

But I want to be straight about what this is not: it's not free Claude. It's a translation layer that lets Claude Code's client talk to a different model entirely. Amazon Nova is not Claude, and the outputs, reasoning quality, and tool-calling behavior won't match what you'd get from Anthropic's actual models. If you're evaluating Claude Code itself, or building something you plan to ship, this setup will give you a skewed picture.

What it's good for is exactly what I built it for: learning the mechanics of agentic coding tools - the loop, the tool calls, the context management without worrying about a credit card every time a session runs long.

Takeaway

If you're in the same spot I was, wanting to learn Claude Code without either buying a GPU rig or getting cut off after three prompts, a Bedrock proxy is a reasonable middle ground. Just don't mistake it for the real thing when it's time to judge the model itself, that comparison only makes sense against Anthropic's actual API.