Mosaic AI is Databricks' answer to running GenAI features on top of governed lakehouse data rather than a separate ML stack bolted alongside it: Model Serving, Vector Search, Feature Serving, an Agent Framework, and MLflow's evaluation and tracing tooling, all sitting on Unity Catalog. The part teams get right quickly is standing up a serving endpoint — that's genuinely close to a few clicks or a short API call. The part that determines whether the feature is any good is everything that has to exist before the endpoint matters at all.
Serving is the easy part
A Mosaic AI Model Serving endpoint, whether it's fronting a Foundation Model API (Databricks-hosted models like Llama or DBRX, billed pay-per-token), a provisioned-throughput deployment for predictable high volume, or your own fine-tuned model, is genuinely fast to stand up. That speed is exactly what makes it tempting to treat as the whole project. It isn't. An endpoint that returns fast, well-formatted, confidently wrong answers is not a shipped feature — it's a liability with good uptime.
What has to exist first
Governed, retrievable data. If the feature is grounded in your own data (support docs, product catalog, internal knowledge base), Mosaic AI Vector Search needs a Delta table with Change Data Feed enabled as its source, so the index stays automatically synced as the underlying table changes. This means the retrieval foundation is a Unity Catalog governance problem before it's an AI problem — the same table hygiene, access controls, and column descriptions that make any lakehouse data trustworthy are what determine whether retrieval returns the right documents.
A real evaluation set, not vibes. MLflow's evaluation tooling (and Mosaic AI's Agent Evaluation on top of it) exists because a probabilistic system needs graded test cases the way deterministic code needs unit tests. Before any endpoint goes near real users, there should be a set of representative questions with known-good answers, and every prompt or retrieval change should be run against it. The Databricks review app — where domain experts label real outputs as correct or not — is the mechanism that turns "we think it's good" into a number you can track over time. Skipping this step doesn't make the project faster; it moves the discovery of quality problems from a controlled eval run to a Slack complaint from a user.
A gateway in front of the endpoint, not just the endpoint. Mosaic AI Gateway sits in front of serving endpoints and handles rate limiting, usage tracking, and safety guardrails (PII detection, content filtering) centrally, rather than each application team reimplementing its own version. For a first feature this is easy to defer and easy to regret deferring — retrofitting guardrails after a feature is live and something has already gone wrong is a much worse conversation than building them in from the start.
A cost model, before volume, not after. Foundation Model API pricing is per-token, and a naive retrieval-augmented prompt that stuffs in ten full documents per query has a very different cost profile than one with a tuned chunk size and a relevance threshold. Provisioned throughput trades that per-token variability for a fixed capacity cost, which is the right call once volume is predictable and wrong when it isn't. Knowing which regime you're in belongs in the design, not discovered on the first real invoice.
The mistake that shows up every time
Teams build the serving endpoint first because it's the most demoable piece, then work backward into retrieval quality and evaluation once the first round of user complaints comes in. It's cheaper, by a wide margin, to build in the order the dependencies actually require: governed data and a defined retrieval strategy first, an eval set before broad rollout, a gateway with guardrails in front of anything user-facing, and only then treat the serving endpoint as the finished piece it looks like on day one.
This is the same discipline our data analytics and AI team applies regardless of which specific platform sits underneath — Mosaic AI just makes more of the plumbing native to a workspace you may already run. Zephico is a Databricks Consulting Partner; if you're scoping a first GenAI feature on Databricks and want a straight read on what has to exist before the endpoint does, talk to us.
Originally published on the Zephico blog.
Top comments (0)