An AI feature can answer correctly and still cost too much to run. Before swapping models, look at what you are asking the API to do: how many calls, how much input, how much output, and how often the same work repeats.
These five changes are worth testing. They are not a promise of a particular saving. The useful number is cost per successful task, with your quality requirements held steady. A cheaper answer that needs a second attempt might not be cheaper work.
1. Remove calls that do not need a model
OpenAI's cost-optimization guide starts with reducing requests, minimizing tokens, and choosing an appropriately sized model. Start with the request count because it is easy to overlook while comparing token prices.
As a design exercise, trace one user action through your application. Does it trigger a separate call to classify the request, another to rewrite it, and another to answer it? Each call may be justified. The point is to check, not to combine everything blindly.
A fixed lookup or an ordinary validation rule might not need a model. That is an engineering judgment, not a claim that every classification task belongs in an if statement. Test the simpler path against the cases your feature actually handles.
2. Send less irrelevant input
A model cannot usefully answer from information you removed if that information was necessary. So the target is irrelevant input, not the shortest possible prompt.
For a support feature, consider whether each request needs the entire product manual or only the sections relevant to the question. For document extraction, decide which pages belong to the task before sending an archive of unrelated material.
Those are illustrative designs. Their quality depends on how reliably you select the right context. Keep a test case where the answer depends on a detail near the end of a document. A trimming strategy that loses that detail has failed even if the invoice falls.
3. Ask for the output you will actually use
If your application needs a category and a short explanation, a page of prose is unnecessary output. Define the required fields and expected length before tuning anything else.
OpenAI's guide recommends optimizing for shorter outputs. My suggestion is to make that concrete in your test: compare the answer your interface needs with the answer you currently request. Remove explanations that nobody sees or uses.
Do not simply force a tiny output limit on every task. Truncated JSON or an incomplete answer can create retries and extra handling. Verify that your output stays complete, valid, and useful across difficult cases, not only the happy path.
Photo by Daniil Komov, free to use under the Unsplash License.
4. Give a cheaper model a fair test
A lower-priced model is a candidate, not an automatic replacement. OpenAI's guidance is to balance lower cost with maintained accuracy.
Build a small evaluation set from tasks you are allowed to use. Include routine cases, ambiguous requests, missing information, and the failures you cannot accept. Decide what passing means before seeing the results.
Then compare the existing model and the candidate on the same tasks. Record successful completions, retries, latency, and cost. If you introduce routing between models, test the routing decision too. Sending a difficult case down the cheap path is part of the system's performance, not a separate excuse.
5. Match repeated and non-urgent work to the right feature
Two options deserve separate checks: prompt caching and batch processing. They solve different problems.
Prompt caching reuses processing for an eligible matching prompt prefix. Put stable, reusable material before changing request-specific content where the model's documented behavior supports it. Watch actual cache usage rather than assuming that a repeated conversation guarantees a hit.
Read the selected model's current terms. Minimum lengths, cache lifetimes, and write/read rates matter. A cache write can cost more than ordinary input processing; reuse is what can make the trade worthwhile. Do not add filler just to reach a cache threshold.
The Batch API is for asynchronous groups of requests that do not need an immediate response. Think offline evaluations or a backlog of documents, not a customer waiting in live chat. Check model support, completion status, and the current processing window before choosing it. A submitted batch is not proof every request succeeded.
Keep the change only if the task still works
Start with a baseline. Change one lever, run the same evaluation, and compare cost per successful task. Keep the change when it meets your quality and latency requirements; otherwise revert it or narrow where it applies.
That is the whole discipline: less unnecessary work, not a cheaper-looking request that quietly moves the cost into retries or human cleanup.
Cover photo by Kelly Sikkema, free to use under the Unsplash License.
I write about startups and AI on StartupWiki.
Top comments (0)