Telling a cheap model to "think a little" made it slower and sometimes silent

Cuepreem — Thu, 15 Oct 2026 03:49:54 +0000

I told a model to think less, and it started thinking.

That's the whole bug. But how I got there is worth writing down, because anyone putting one endpoint in front of several model families will probably hit some version of it.

The setup: I build a desktop app that can talk to a model three ways. A local model through Ollama, the user's own API key, or cloud credits that go through a small gateway I run (a Cloudflare Worker) in front of OpenRouter. In that last mode the app doesn't know which real model it's using. It asks for a virtual label like cuepreem-cloud-fast, and the gateway decides what that means today. So anything model-specific has to happen in the gateway, because it's the only place that knows the real model name.

One model-specific thing it does is cap reasoning. Some models think before they answer by default, and that thinking comes out of the same max_tokens budget as the answer. Give one of them a tight budget and it can spend all of it thinking and hand back an empty string, with the tokens still billed. So the gateway checks the model name and, for models that reason, adds a cap. Trimmed down, it looks like this:

function buildUpstreamBody(body, spec, model) {
  return {
    ...body,
    model,
    // The client only sends a virtual label, so only the gateway can decide this.
    ...(isReasoningModel(model) ? { reasoning: { effort: 'low' } } : {}),
  };
}

And isReasoningModel was a regex over model names:

// before (trimmed)
return /gpt-oss|deepseek-r1|qwq|magistral|gemini-2\.5/i.test(model);

The Gemini 2.5 models I had in mind think by default, so gemini-2\.5 looked right.

Then I moved the fast lane to Gemini 2.5 Flash-Lite. It's cheap, it's quick, and it doesn't think unless you ask it to. Which is exactly what was happening. google/gemini-2.5-flash-lite contains gemini-2.5, so the gateway sent it reasoning: { effort: "low" }.

Here's the part I had wrong in my head. I'd been treating effort as a dial that only turns down: "low" means "less than you'd normally do". For a model that thinks by default, that's roughly true. For a model that doesn't, there's nothing to turn down. Any effort value is a request to start. "Low" meant "some", and some is more than none.

To confirm it, I made a small test call with max_tokens set to 16. The response reported 12 reasoning tokens and an empty reply. Most of the budget went on thinking, and the user-visible answer was blank.

With a roomier budget the answer can still fit after the thinking, so the failure is quieter: the reply just arrives later than it should, from the model you picked because it was fast. With a tight budget, which is what you set when you want a short answer quickly, the thinking can take all of it.

That's what made it slippery. Nothing errored, and the upstream call came back 200. The symptoms were "slower than it should be" and, sometimes, nothing at all.

The fix was one negative lookahead:

// after (trimmed)
return /gpt-oss|deepseek-r1|qwq|magistral|gemini-2\.5(?!-flash-lite)/i.test(model);

plus a test that pins it, next to one that makes sure a non-reasoning model doesn't get the field either:

it('never sends an effort to Gemini 2.5 Flash-Lite — it would switch thinking ON', () => {
  const body = buildUpstreamBody({ messages: [] }, spec, 'google/gemini-2.5-flash-lite');
  expect(body.reasoning).toBeUndefined();
});

it('leaves a non-reasoning model alone', () => {
  const body = buildUpstreamBody({ messages: [] }, spec, 'meta-llama/llama-3.3-70b-instruct');
  expect(body.reasoning).toBeUndefined();
});

Separately, the gateway no longer takes a 200 at its word. If the visible reply is empty once any <think> block is stripped, that's a failed call, not a delivered answer.

The lookahead is the smallest fix, not the best one, and I know it. The real lesson is that I was asking the wrong question. It isn't "does this model support reasoning?" Flash-Lite does support it, and that's the problem. The question is "does this model think when nobody tells it anything?" Model families don't answer that consistently. A -lite or -mini variant can have the opposite default to its siblings, and a family-name regex will quietly sweep it in the day you switch to it.

The better rule is to key the cap on exact model IDs you've checked, not on names that look like ones you've checked. And the cheap test that would have caught this in one call: before a model goes into a latency-sensitive lane, call it with a tiny max_tokens, read the reasoning-token count, and assert the visible reply isn't empty.

I ran into this building CuePreem, an interview-prep desktop app (disclosure: it's mine, and the cloud gateway is the part this post is about): https://cuepreem.com/byok-interview-copilot?utm_source=devto&utm_medium=article&utm_campaign=build-in-public

If you run more than one model family behind one endpoint, how are you tracking which ones think by default? A hand-kept regex is all I've got, and after this I don't trust it.

DEV Community: Cuepreem

Telling a cheap model to "think a little" made it slower and sometimes silent