Dead Model Walking

The terminal stared back at me with a 422 Unprocessable Entity — the third time in an hour — and somewhere in the Groq docs, buried under a deprecation notice, was the reason our entire Campus AI had gone silent.

1. The Crisis: When Your Model Gets Decommissioned Mid-Hackathon

Hackathons move fast. You commit to a stack, write your integration, test it locally, and then ship. What nobody prepares you for is the platform pulling the rug out from under you with a model deprecation notice dropped on a Friday evening.
Our team was deep into building the Smart Campus AI Assistant — a FastAPI-backed chatbot that answered student queries about dining hours, class schedules, campus events, and shuttle timings. We were running on Groq’s ultra-fast inference API and had chosen llama3-8b-8192 as our backbone model for its speed and efficiency.
Then the 422s started.
A 422 Unprocessable Entity from Groq’s API is particularly confusing because your JSON is syntactically valid — it’s semantically wrong. The server understood our request; it just couldn’t process it. After three rounds of validation and head-scratching, we dug into the Groq model documentation and found the culprit: llama3-8b-8192 had been officially decommissioned. Any request referencing it returned a 422, regardless of how perfectly formed the body was.

The Pivot: Choosing llama-3.3-70b-versatile
The replacement decision had to be fast and informed. We needed a model that was:
• Still available and actively supported on Groq
• Capable enough for nuanced student Q&A across multiple domains
• Not going to tank our inference latency targets
• A drop-in replacement with no prompt reformatting required

llama-3.3-70b-versatile checked every box. Despite the larger parameter count, Groq’s inference hardware kept latency low — and the “versatile” in the name wasn’t marketing fluff.

2. The FastAPI /ask Endpoint: Architecture and the CORS Requirement

Once we confirmed the model pivot resolved our 422s and we were getting clean 200 OK responses, we turned our attention to the broader integration architecture. The heart of our backend is a single FastAPI endpoint:

from fastapi import FastAPI
from pydantic import BaseModel
from groq import Groq

app = FastAPI()
client = Groq(api_key=os.getenv("GROQ_API_KEY"))

class Query(BaseModel):
question: str

@app.post("/ask")
async def ask_assistant(query: Query):
chat_completion = client.chat.completions.create(
messages=[{"role": "user", "content": query.question}],
model="llama-3.3-70b-versatile",
)
return {"answer": chat_completion.choices[0].message.content}
This endpoint is intentionally minimal. Pydantic’s BaseModel gives us automatic request validation, so malformed bodies are rejected at the framework level before they ever hit Groq. The async handler ensures our FastAPI server doesn’t block while waiting on inference.

Why CORS Was Not Optional

Our frontend runs on a different origin from our FastAPI backend. Without CORS headers, the browser’s same-origin policy would silently block every single API call.

from fastapi.middleware.cors import CORSMiddleware

app.add_middleware(
CORSMiddleware,
allow_origins=[""],
allow_methods=[""],
allow_headers=["*"],
)

The add_middleware call must happen before your route definitions. We had a duplicate middleware block at the bottom of our file that was effectively dead code, cleaned up once we understood FastAPI’s middleware registration order.

3. Agent Memory with Hindsight: Giving the Bot a Brain

A campus AI that can’t remember anything is just a fancy search bar. What makes our assistant genuinely useful is that it builds a model of each student’s preferences and habits over time. For this, we integrated Hindsight — an agent memory layer purpose-built for LLM applications.
Key resources used during integration:
• Hindsight GitHub Repository — source code, setup, and contribution guides
• Hindsight Official Documentation — full API reference and integration tutorials
• Vectorize Agent Memory Overview — the conceptual foundation behind persistent agent memory

How Hindsight Fits Into the Stack

Hindsight sits between our FastAPI layer and the Groq inference call. When a student submits a query, Hindsight retrieves relevant memory fragments — past queries, stated preferences, dormitory location, dietary restrictions — and injects them into the system context before the request reaches the LLM.
Practically, this means:
• A student who asked about vegan dining options last week doesn’t have to re-specify their dietary preferences today.
• A student in a specific dormitory gets shuttle schedules relevant to their building without being asked again.
• Frequently queried topics get surfaced proactively as the model learns usage patterns over time.

The memory architecture Hindsight uses is grounded in vector retrieval, which means preferences are stored semantically — not as rigid key-value entries. This handles the messy reality of natural language: “I’m vegetarian” and “no meat for me” map to the same underlying preference without requiring exact-match lookup logic.

4. What We Learned — and What the Bot Remembers Now

The decommissioned model crisis was a genuine setback. Two hours of debugging, a model deprecation we hadn’t anticipated, and a cascade of 422 errors that initially looked like our own serialization bugs. The pivot to llama-3.3-70b-versatile was the right call — not just as a fix, but as an upgrade.
But the bigger lesson was about architecture. A stateless LLM endpoint, however fast, plateaus in usefulness. Students repeat themselves, context is lost between sessions, and the bot treats every query as if it’s meeting the user for the first time. Hindsight eliminates that cold-start problem entirely.
A campus AI with memory isn’t just a convenience upgrade — it’s the difference between a tool students tolerate and one they actually return to. When the assistant knows you, it serves you better. And when it keeps serving you better over time, it earns the kind of trust that no single clever prompt can manufacture from scratch.
The model is live. The memory is persistent. The bot knows your name — and your coffee order.

Resources & Links
• Hindsight on GitHub
• Hindsight Documentation
• Vectorize Agent Memory