DEV Community

Cover image for How to Build a Cost-Saving Semantic Cache AI Assistant
Nagesh Gunji
Nagesh Gunji

Posted on Edited on

How to Build a Cost-Saving Semantic Cache AI Assistant

#ai #webdev #programming #productivity

Building AI assistants that call LLM APIs directly on every single user prompt gets expensive fast. Duplicate questions, repeated queries, and baseline RAG retrieval can quickly drain your API budget during testing and early production.

To solve this, I built an architecture that combines in-memory semantic query caching with Voyage AI embeddings, local Ollama runtimes ($0 token costs), and Prisma 7 pgvector on Next.js 16.

Here is a breakdown of how the cost-saving stack is structured.

🛠️ Core Tech Stack

  • Framework: Next.js 16 (App Router + Server Actions with React 19 support)
  • Embeddings & AI: Pre-configured for Voyage AI embeddings (high-accuracy production vector search) and local Ollama runtimes ($0 token testing offline)
  • Database & Vector Engine: PostgreSQL + Prisma 7 with native 1024-dimension pgvector support
  • Performance Layer: Sub-millisecond in-memory semantic query cache to skip redundant LLM processing
  • Monetization Layer: Pre-configured Paddle billing modal

⚡ How the Cost-Saving Cache Works

Instead of sending every query straight to an expensive cloud LLM, the system checks vector similarity first:

  1. Generate Vector: Convert incoming prompt to an embedding using Voyage AI (or local Ollama).
  2. Similarity Lookup: Perform a vector similarity search in PostgreSQL via Prisma 7 pgvector.
  3. Return or Fetch: If a high-confidence match exists in the semantic cache, return the cached answer in sub-milliseconds. If not, route to the LLM and store the new result. 
// lib/embeddings.ts
import { VoyageAIClient } from "voyageai";

const voyage = new VoyageAIClient({ apiKey: process.env.VOYAGE_API_KEY });

export async function generateVoyageEmbedding(text: string): Promise<number[]> {
  const response = await voyage.embed({
    input: text,
    model: "voyage-3",
  });

  return response.data?.[0]?.embedding ?? [];
}
Enter fullscreen mode Exit fullscreen mode

📐 Database & Schema Setup

Using Prisma 7's native pgvector extension allows storing and querying vector embeddings directly inside PostgreSQL without paying for third-party vector SaaS providers:

// schema.prisma snippet
datasource db {
  provider   = "postgresql"
  url        = env("DATABASE_URL")
  extensions = [pgvector(map: "vector")]
}

generator client {
  provider        = "prisma-client-js"
  previewFeatures = ["postgresqlExtensions"]
}

model DocumentEmbedding {
  id        String                 @id @default(cuid())
  content   String
  embedding Unsupported("vector(1024)")
  createdAt DateTime               @default(now())
}
Enter fullscreen mode Exit fullscreen mode

🎬 6.5-Minute Video Walkthrough

I recorded a full technical walkthrough demonstrating the code structure, terminal build (npm run build), schema migrations, Voyage AI integration, and live RAG retrieval speed:

🚀 Get the Full Production Codebase (.zip)

If you're building an AI micro-SaaS and want to skip 40+ hours of baseline setup, you can download the complete Aether OS Pro codebase.

It includes full setup instructions (README.md), pre-configured .env.example, zero-error terminal builds, and instant local deployment.

👉 Download Aether OS Pro (.zip) via Instant UPI Checkout

🏷️ Dev.to Tags to Add:

#nextjs #ai #prisma #webdev

Top comments (0)