DEV Community: Katie

Does AI get worse with more documents? Open-source vs paid models, tested.

Katie — Mon, 12 Oct 2026 04:23:20 +0000

This is a submission for the Kaggle Benchmarking Challenge

What I Benchmarked

Lots of companies now hand an AI model a stack of their own documents and ask it questions: "How many days of leave do I get?" or "Who signs off on this purchase?" Developers call this RAG, but the idea is simple. The AI reads what you give it and answers from that.

There is a common worry about this setup: the more documents you hand over, the worse the answers get. The right page gets buried, an old copy gets picked by mistake, or the AI makes something up.

I wanted to see whether that is true, so I built a test.

The setup. I invented a company and generated 50 short internal documents for it: leave policies, travel rules, supplier contracts, product sheets. Because the company is made up, no AI model could have seen these documents before. Then I asked each model 90 questions. For each question the model got a pile of documents, anywhere from 2 to 50, and had to do two things: give the answer, and say which document it came from. If the answer was not in the pile, it had to say "not found".

I changed four things from question to question:

What I changed	The options	What I wanted to learn
Size of the pile	2, 5, 10, 25 or 50 documents	Do answers get worse as the pile grows?
Where the right document sits	Top, middle or bottom of the pile	Does the AI skim over the middle?
Kind of question	One-step, two-step, or no answer exists	Can it look things up, connect two documents, and admit when it doesn't know?
The other documents in the pile	Unrelated, or very similar to the right one	Do look-alike documents confuse it?

Two of these need a little more explanation.

Out-of-date copies. Ten documents have an older version with the same wording and different numbers. In the harder questions, the old version is in the pile next to the current one. This is what a real shared drive looks like.

Two-step questions. Every document ends with a line like this:

Unresolved questions about this document are escalated to whoever approved the Annual Leave Policy for the Marrowby studio.

The question asks: who is that person? To answer, the model has to go to a second document, the Annual Leave Policy, and find the name of the person who approved it. One document points to the other, and the answer is only in the second one.

Models Tested

I tested seven models. I picked them to cover the choices a team actually faces: the big expensive model or the small cheap one, and a paid service or a free model you run yourself.

Model	What it is
Claude Sonnet 5.5	Anthropic's main model
Claude Haiku 4.5	Anthropic's smaller, cheaper model
GPT-5.5	OpenAI's main model
GPT-5.4 mini	OpenAI's smaller, cheaper model
Gemini 3.8 Flash	Google's fast model
Gemma 4 (31B)	A free model from Google that you can run on your own machines
gpt-oss-20b	A small free model from OpenAI that you can run on your own machines

The comparison I cared about most was big against small from the same company, and whether the free models could keep up.

Findings

A bigger pile did not make the answers worse

This was the thing I set out to measure, and the answer was no. Here is how often each model got the right answer as the pile grew. Each box is about 12 questions.

Model	2 docs	5 docs	10 docs	25 docs	50 docs
Claude Sonnet 5.5	100%	100%	100%	100%	100%
Gemini 3.8 Flash	100%	100%	100%	100%	100%
Gemma 4	100%	100%	100%	100%	100%
gpt-oss-20b	100%	100%	100%	92%	100%
GPT-5.5	100%	100%	100%	100% (5 of 5)	no data
Claude Haiku 4.5	67%	58%	50%	75%	50%
GPT-5.4 mini	67%	58%	50%	42%	50%

Four models scored the same with 50 documents as with 2. The two at the bottom were already struggling with only 2 documents, so the size of the pile is not what hurt them. Their numbers bounce around because each box is only 12 questions, and one wrong answer moves the score by 8 points.

It also made no difference whether the right document was at the top, in the middle or at the bottom.

One caution: my documents are short, less than a page each. Fifty of them come to about 16,500 words, roughly 35 pages. I can say the pile size did not matter up to there. I cannot say what happens with a pile ten times bigger.

Two-step questions are where the small models failed

When I split the results by kind of question, a very different picture appeared:

Model	One-step questions	Two-step questions
Claude Sonnet 5.5	30 of 30	30 of 30
Gemini 3.8 Flash	29 of 29	30 of 30
Gemma 4	30 of 30	30 of 30
GPT-5.5	17 of 17	18 of 18
gpt-oss-20b	30 of 30	29 of 30
Claude Haiku 4.5	30 of 30	6 of 30
GPT-5.4 mini	29 of 30	3 of 30

Every model could find a single fact in a pile of 50 documents. But two of them could not follow a pointer from one document to another, even when the pile was just those two documents.

They failed in different ways.

Claude Haiku 4.5 gave up. In 21 of its 24 wrong answers it said "not found". Often it had found the pointer and stopped there:

ANSWER: NOT_FOUND
SOURCE: NONE

The Business Travel Policy for the Quillhaven lab (DOC-08) states that "Unresolved
questions about this document are escalated to whoever approved the Annual Leave
Policy for the Marrowby studio," but it does not provide the actual name of that person.

The name it wanted was in another document in the same pile. It did not go and look.

GPT-5.4 mini answered without finishing the job. In 20 of its 27 wrong answers it repeated the pointer as if it were the answer: "whoever approved the Equipment Maintenance Schedule for the Aldersgate office". In the other 7 it gave a real person's name, but the wrong person, and pointed to a real document as its source.

These two failures are not equally bad. If an AI says "I couldn't find it", you know to check yourself. If it gives you the wrong name and cites a real document, you will probably believe it.

The free models beat the small paid ones

This surprised me most. Gemma 4, which is free to run yourself, got all 60 answerable questions right and named the correct source every time. gpt-oss-20b, a small free model, got 59 of 60. The two small paid models got 36 of 60 (Claude Haiku 4.5) and 32 of 60 (GPT-5.4 mini).

The one weak spot: gpt-oss-20b usually had the right answer but sometimes named only one of the two documents it needed. It listed every source on 52 of 60 answers.

Model	Type	Answerable questions right
Gemma 4	Free, open	60 of 60
gpt-oss-20b	Free, open	59 of 60
Claude Haiku 4.5	Small paid	36 of 60
GPT-5.4 mini	Small paid	32 of 60

The paid models' failures may be partly about how they follow instructions. Haiku mostly said "not found" instead of guessing, which is cautious behaviour, not an inability to read. A differently worded question might change its score a lot.
"Free" isn't free to run. You don't pay per question, but you need your own hardware, and Gemma 4 at that size needs a serious machine.

No model made things up

A third of my questions had no answer anywhere in the documents. Every model said "not found" to every one of them, at every pile size. I expected at least a few invented answers with 50 documents and got none.

The out-of-date copies also fooled almost no one. Each model saw about 27 questions where an old version was sitting in the pile. Only one answer in the whole test used the old number.

What these results cannot tell you

GPT-5.5 did not finish. I ran out of usage allowance partway through, and it never got to answer 35 of its 90 questions, including all of the 50-document ones. It was perfect on the 55 it did answer. I left the missing ones out of my tables. Kaggle counts them as wrong, so its score looks lower there.
Each model ran once. I did not repeat the test to see how much the scores move between runs.
The groups are small. A difference of one or two questions means nothing. The gap on two-step questions (30 of 30 against 3 of 30) does.
The documents are invented. They are tidier than a real company's files, and five of the seven models scored at or near perfect, so this test is too easy to tell those five apart.

What I would test next

My test turned out to be easy on the thing I built it for and revealing on something else. So next I would:

Make the pile much bigger than 50 documents, and make the documents longer, to find where the scores finally drop.
Add three-step and four-step questions. If two steps separate the small models from the rest, more steps may separate the rest from each other.
Run GPT-5.5 again with enough allowance to finish.
Run every model several times.

The practical lesson for now: in this test, handing the AI more documents was not the risk. The risk was asking a small model a question whose answer is spread across two documents.

My Benchmark

Task: rag-doc-scaling on Kaggle Benchmarks

Dataset (50 documents, 90 cases): katienguyen27/rag-doc-scaling

From Vanilla JS to React: React Makes More Sense When the Problem Comes First

Katie — Sat, 28 Mar 2026 08:10:19 +0000

I had an e-commerce project — plain HTML/JavaScript frontend, Node.js/Express backend. I took on a more interesting challenge: migrating the frontend from vanilla HTML/JS to React.

Migrating code you already wrote and understand is one of the better ways to go deeper with React. You're not learning the syntax and the concepts at the same time — you already know what the code does, so you can focus entirely on why each React pattern exists. This is what that looked like.

The original version

Each page was a separate HTML file loaded its own JS modules. State lived in the DOM. Navigation caused full page reloads. The patterns were familiar:

updateCartIcon() fetches the cart total from the backend and updates the cart icon in the banner — it had to be manually called in every place that could affect the cart count
showHideMenuItems() checks if the user is logged in and toggles the visibility of login, signup, and logout buttons by directly setting element.style.display
Any change to the shared header or footer meant touching every HTML file

None of these are bugs, but they're all symptoms of the same root problem: state is scattered and has to be manually synchronized.

Pages → components

The mapping is direct. Each HTML file becomes a page component, and React Router replaces file-based navigation. window.location.href = '/cart.html' becomes navigate('/cart'). <a href="/login.html"> becomes <Link to="/login">.

The more meaningful difference is that only the page component swaps out on navigation — the rest of the component tree stays mounted, so state persists across the session without any extra work.

Extracting shared UI

Every page shared the same banner, header, and footer. In the old app this was solved with copy-paste. In React, a Layout component wraps all routes via React Router's <Outlet />:

Layout
├── Banner
├── Header
├── <Outlet />   ← current page renders here
└── Footer

The Banner itself had three distinct responsibilities — a mobile menu toggle, auth-aware navigation links, and a cart icon — so it was broken into Banner, Navigation, and CartIcon components.

The toggle was the first real state decision: the button lives in Banner, and Navigation receives isOpen as a prop. The state lives in the closest common ancestor, which is the standard lift-state-up pattern. Straightforward.

The custom hook trap

The cart count was where things got more interesting.

CartIcon in the banner needs to display the total items the user added to the cart. ProductList needs to refresh that total after adding an item. Two components, one shared value — the obvious move is a custom hook that fetches the total from the backend and exposes a refresh function to re-fetch on demand:

function useCartCount() {
  const [cartCount, setCartCount] = useState(0);
  const refresh = async () => {
    /* fetch /api/cart/cart-count */
  };
  useEffect(() => {
    refresh();
  }, []);
  return [cartCount, refresh];
}

If you call this hook in both components, it looks clean. It doesn't work.

The issue is obvious in hindsight: each hook call is an independent instance. CartIcon and ProductList each had their own isolated cartCount state. When ProductList called refresh(), it updated its own copy. CartIcon never knew anything changed.

Extracting stateful logic into a custom hook does not automatically make that state shared.

Context as the fix

The fix was to move ownership of the cart count into a provider above both components:

CartContextProvider
├── Banner
│   └── CartIcon  ← reads cartCount
└── Home
    └── ProductList  ← calls refresh()

There are two useful hook roles here:

A private hook can manage the provider’s state and fetching logic.
A public hook can read the values shared through context.

The custom hook still exists — but now it runs inside CartContextProvider, whenever the provider renders, but its state persists across those renders. Everything downstream reads from that one instance via useContext. When ProductList calls refresh(), it's the same function CartIcon is subscribed to.

I could also keep the state and fetching logic directly inside CartProvider. Extracting that implementation into a private hook is optional.

The same pattern applied to auth. Navigation needs isLoggedIn to render the right links. CartIcon needs it to decide whether to render at all. Pulling both into an AuthContext replaced the old showHideMenuItems() with declarative rendering — components just read state and React handles the rest.

Where login logic lives — and why it matters

The first version had the login fetch inside the Login component. It sent the request, and on success, called navigate('/').

The problem: the Login component had no connection to AuthContext. After a successful login, isLoggedIn in the context hadn't changed. The banner still showed "Log in". The app looked broken until you refreshed.

The fix is to move login() and logout() out of the component and into AuthContext — so they have direct access to setUser and can update the shared state immediately on success.

This also revealed a cleaner separation of concerns. useAuth should only do one thing: check the session on mount and expose the user state. The actions — login, logout — belong in the context that owns that state:

useAuth — runs checkAuth on mount, returns { user, setUser }
AuthContext — defines login() and logout(), uses setUser from the hook to update state, provides everything to the tree
Login - handles the form, error messages, button disabled state, loading indicators, and navigation after success

This settled into a useful pattern for the app: the provider manages shared auth, the hook provides access, the component owns UI — error messages, button disabled state, loading indicators. The Login component now just calls login() through useAuth() and reacts to the response.

Session cookies vs localStorage

There are two common ways to handle auth persistence on the frontend.

JWT + localStorage — on login, the backend returns a token. The frontend stores it in localStorage and manually attaches it to every request as an Authorization header. It persists across refreshes, but the token is exposed to JavaScript, making it vulnerable to XSS attacks.

Cookie-based auth — on login, the backend sets a cookie containing a session identifier or token. The browser stores it and automatically attaches it to requests that match the cookie's scope and browser policies. With HttpOnly, JavaScript can't directly read the cookie. HttpOnly prevents direct cookie theft through JavaScript, but it doesn't stop an XSS attack from making authenticated requests.Cookie-based auth also needs appropriate CSRF protection and cookie settings such as Secure and SameSite.

JWTs can also be stored in cookies. The token format and the storage mechanism are separate choices.

For same-origin requests, fetch includes cookies by default. If the frontend and backend are on different origins, the frontend needs credentials: 'include', and the backend needs the appropriate CORS configuration. This option tells the browser to include them, so the backend can read the session and know who you are.

This app uses an HttpOnly cookie for its session. The consequence for React state is that isLoggedIn doesn't survive a page refresh — it's just JavaScript in memory. On every mount, useAuth calls /api/auth/me. If the session cookie is still valid, the backend confirms the user and state is restored. If not, the user is logged out.
While that check is pending, auth is still loading. A failed network request is also different from the backend confirming that the session is invalid.

React state reflects auth. The backend validates it. The cookie carries the credential. Keeping those roles separate is the right mental model.

The actual takeaway

The migration made one thing very concrete: in my old app, UI updates were driven by manual DOM changes; in React, the UI was rendered from state..

In the old app, showHideMenuItems() existed because the DOM didn't know about login state — you had to reach in and update it manually every time something changed. In React, isLoggedIn changes once and everything that depends on it re-renders. The DOM reflects state, rather than being the thing you manage directly.

Vanilla JS can use explicit state as its source of truth too. React gave this app a consistent way to organize that relationship.

That shift — from imperative DOM manipulation to declarative state — is what every React pattern ultimately comes back to. Lifting state up, custom hooks, Context — each one is just a different answer to the same question: how far does this state need to reach?