A few weeks ago, I added a small AI voice assistant called Jarvis to my portfolio website.
The setup was intentionally simple: Groq handled text generation, while the browser’s Web Speech API handled speech-to-text and text-to-speech. Visitors could ask Jarvis about my work, projects, experience, or services, and it could also help them book a call.
It worked.
There was just one problem.
After about five minutes of fairly light testing, I had already burned through the Groq free-tier limit.
My first thought was probably the obvious one:
Do I need another provider or a paid API plan?
Before changing anything, though, I decided to figure out where the tokens were actually going.
That turned out to be the right decision.
3,113 Prompt Tokens Just to Say “Hello”
The first thing I wanted was an actual measurement.
Instead of relying only on a token-counting estimate, I made a real Groq request with a minimal output limit and inspected the token usage returned by the API.
Then I sent:
hello
The result surprised me:
3,113 prompt tokens.
Not for a complicated question.
Not for a request involving several projects.
Just “hello.”
At that point, the problem became pretty obvious.
My model wasn't necessarily expensive.
My request architecture was.
I found two major problems.
- I was sending my entire portfolio every time
My system prompt contained almost everything about me:
services
project descriptions
testimonials
FAQs
statistics
tech stack
experience
behavioral instructions
And all of it was being sent on every request, regardless of what the visitor asked.
Someone saying “Hi” was effectively sending thousands of tokens of portfolio context to the model.
Someone asking about pricing received the same context.
Someone asking about one project also received information about every other project.
It worked, but it was incredibly wasteful.
- I had duplicated part of my knowledge base
While auditing the knowledge object, I found something even simpler.
My tech stack existed twice under two different fields.
Two parts of the system had been built separately, and both included the same information without me noticing.
So I wasn't just sending too much context.
I was literally paying in tokens to send some of it twice.
The Bigger Realization: Not Every Message Needs AI
This ended up being the most important change.
I had originally treated Jarvis like this:
User message → Groq → response
Every message went through the model.
But why should an LLM generate an answer to:
Hello
Or:
Who are you?
Or:
Do you sign NDAs?
Those answers are already known.
So I added a lightweight local intent layer before Groq.
Now the flow looks more like:
User message → local intent check → Groq only when necessary
Simple regex and keyword matching handles predictable requests locally.
For example:
Greetings → predefined greeting
“Who are you?” → fixed identity response
Pricing questions → predefined pricing-policy response
Known FAQs → existing FAQ answer
Requests for private information → fixed refusal
Obvious prompt-injection attempts → fixed security response
If the intent matches one of these cases, Groq is never called.
API usage for that interaction: zero.
This also made the assistant feel faster because there is no reason to wait for model inference when the application already knows the answer.
Then I Shrunk the System Prompt
The next target was the prompt itself.
Instead of shipping my entire portfolio with every request, I separated the prompt into two layers.
Layer 1: A small base prompt
The base prompt contains only information Jarvis always needs:
identity
behavior
privacy rules
pricing policy
important boundaries
response style
After cleaning it up, the base context dropped to roughly 652 tokens, compared with the 3,113-token request I measured earlier.
Layer 2: Retrieve Only Relevant Portfolio Knowledge
The rest of my portfolio became selectively retrieved context.
I divided the knowledge into sections such as:
services
projects
testimonials
faqs
contact
stats
Before calling Groq, the application checks the user's message and determines which sections are actually relevant.
If someone asks:
“What AI automation work have you done?”
Jarvis might receive relevant services and AI project information.
It doesn't need every testimonial, contact detail, FAQ, and unrelated project.
Usually only 2–4 relevant sections are added to the prompt.
It isn't a complicated vector database or a full RAG pipeline.
For a portfolio this size, simple retrieval works perfectly well.
And more importantly, it's cheap.
I Also Found a Retry Problem
There was another source of unnecessary requests that wasn't immediately obvious: retries.
SDKs often retry certain failed requests automatically, including rate-limit errors.
That's normally helpful.
But when you're already hitting a quota limit, automatic retries can make the situation worse.
A request fails with a 429.
The client retries.
It fails again.
Another retry happens.
From the application's point of view, the visitor sent one message.
From the API's point of view, multiple attempts may have occurred.
So I tightened the retry behavior and added request-level protection around the voice agent.
The final flow includes:
request deduplication
cooldown protection
aborting superseded requests
controlled retries
request tracking
The goal was simple:
One visitor message should result in at most one intentional Groq generation request.
The Result
After the changes, the difference was significant.
Before
Measured “hello” request:
3,113 prompt tokens
After
Compact base context:
~652 tokens
For requests that still need Groq, selective knowledge retrieval reduced prompt-token usage by roughly:
70–72% in my testing.
And several common interactions now use zero Groq requests:
greetings
identity questions
pricing-policy questions
known FAQs
security refusals
obvious prompt-injection attempts
duplicate submissions
The interesting part is that I didn't switch models.
I didn't move to another AI provider.
And I didn't solve the problem by simply paying for a larger quota.
I changed how my application used the model.
What I Learned
When an AI application starts burning through API quota, it's easy to assume the model or provider is the problem.
Sometimes it is.
But before switching providers, I think it's worth looking at the request path itself.
Ask:
Does this request actually need an LLM?
Am I sending context the model doesn't need?
Am I sending the same information repeatedly?
Can part of this response be deterministic?
Can I retrieve only the knowledge relevant to this question?
Can one user action accidentally trigger multiple API requests?
In my case, those questions mattered far more than changing the model.
The biggest optimization wasn't finding a cheaper LLM.
It was calling the LLM less often and sending it less unnecessary information when I did.
I'm Asad Ibrahim, a full-stack developer and AI integration engineer. I build AI-powered web applications, automation systems, voice assistants, and production integrations for businesses.
I also document experiments like this from projects I'm actually building.
You can see more of my work, AI projects, and engineering case studies at asadibrahim.com.
Top comments (0)