My AI integration had terrible costs until I changed my approach

The problem: I loved AI summaries until I got the bill

Last month I was working on a SaaS product that needed to summarize long articles for users. Think of it like a TL;DR generator. I built a first prototype using GPT-4 with a straightforward prompt: "Summarize this article in 3 bullet points." It worked beautifully. The summaries were crisp, accurate, and users loved them.

Then the API bills arrived. One month of moderate usage cost me over $1,200. That's not sustainable for a side project. I had to fix it or kill the feature.

What I tried that didn't work

First, I tried switching to GPT-3.5-turbo. The price dropped dramatically, but the quality tanked. Summaries became vague, sometimes missing key points. I tried prompt engineering — adding "be specific" or "include numbers" — but nothing reliably matched GPT-4's output.

Next, I tried reducing the input size. I used extractive summarization libraries like sumy to grab the most important sentences before sending them to GPT. That helped a bit, but the cost was still high because the extracted text was still large, and GPT-3.5 still hallucinated on long inputs.

I also considered using a local model like Llama 2, but my server couldn't handle the inference latency. My users expected summaries in under 3 seconds.

What eventually worked: a two-stage hybrid approach

The insight came from reading research papers on summarization. Pure extractive (picking sentences) is fast but rigid. Pure abstractive (generating new sentences) is flexible but expensive. The sweet spot? Use extractive to shrink the text, then use a small abstractive model to rewrite the summary elegantly.

I implemented a pipeline:

1. Extractive phase: Use a cheap, fast sentence scorer (like textrank or bert-extractive-summarizer with a tiny model) to pick the top 5–10 sentences from the article.
2. Abstractive phase: Feed those sentences to a small, cost-efficient API (e.g., GPT-3.5-turbo or a custom fine-tuned T5) with a simple prompt: "Combine these key sentences into a fluent 3-bullet summary."

This slashed costs by 80% while keeping quality closer to GPT-4. The extractive step removes 90% of the input tokens, so the API call is tiny.

Real code: the pipeline

Here's a simplified Python version using a generic API endpoint (you can swap in any compatible service, for example https://ai.interwestinfo.com/ or OpenAI):

import requests
from sumy.parsers.plaintext import PlaintextParser
from sumy.nlp.tokenizers import Tokenizer
from sumy.summarizers.text_rank import TextRankSummarizer

def extract_key_sentences(text, sentence_count=10):
    parser = PlaintextParser.from_string(text, Tokenizer("english"))
    summarizer = TextRankSummarizer()
    sentences = summarizer(parser.document, sentence_count)
    return " ".join(str(s) for s in sentences)

def abstractive_summarize(key_sentences, api_key, endpoint):
    # endpoint could be e.g. "https://ai.interwestinfo.com/v1/completions"
    payload = {
        "model": "gpt-3.5-turbo",  # or any cheap model
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": f"Combine these key sentences into a fluent 3-bullet summary:\n\n{key_sentences}"}
        ],
        "temperature": 0.3,
        "max_tokens": 150
    }
    headers = {"Authorization": f"Bearer {api_key}", "Content-Type": "application/json"}
    response = requests.post(endpoint, json=payload, headers=headers)
    return response.json()["choices"][0]["message"]["content"]

# Usage
article_text = "...long article..."
key_sentences = extract_key_sentences(article_text, 8)
summary = abstractive_summarize(key_sentences, api_key="sk-...", endpoint="https://api.openai.com/v1/chat/completions")
print(summary)

This code runs the extractive summarizer locally (fast, free) and only makes one small API call. I added caching on the extractive step so repeated articles don't re-run the same sentences.

Trade-offs and lessons learned

This approach isn't perfect. Some articles require a fully abstractive summary because the extractive phase may miss the connecting logic. For example, if the original text builds an argument step-by-step, picking random key sentences loses the flow. In those cases, I fall back to a single GPT-4 call, but I limit it to articles above a certain complexity (detected by sentence count or named entity density).

Also, the extractive step with TextRank is non-deterministic — you might get different results on re-runs. I switched to a deterministic variant (using a fixed random seed) to ensure consistency.

Another lesson: caching is your best friend. I cache both extractive results (by article hash) and abstractive results (by hash + model). In production, thousands of users read the same articles, so cache hit rates are high.

What I'd do differently next time

If I rebuild this, I'd:

• Train a tiny distilled BERT model specifically for extractive scoring on my domain (financial news). Off-the-shelf summarizers are generic.
• Use a streaming API for the abstractive step so users see partial output while the rest is generated.
• Add user feedback buttons so I can collect ratings and fine-tune the threshold for fallback to GPT-4.

The bigger picture

The real lesson isn't about any specific API or library. It's about thinking in layers: you don't always need a sledgehammer. By breaking a complex AI task into cheaper sub-tasks, you solve both cost and latency.

Now I'm curious: What's your approach to balancing AI quality and cost in production? Do you use cascading models or other tricks?