4 Systems I Built: Architecture Decisions That Held Up

Every architecture decision looks smart in a slide deck. The real test is what happens six months in, when the bill arrives, a model deprecates, or a client asks you to add a feature nobody planned for. I want to walk through four systems I built, the choices I made at the start, and which of those choices I would make again today.

I'll skip the tutorial framing. This is the honest version: what I picked, why, what broke, and what the numbers looked like once the thing was running in production.

System 1: ContentStudio, a multi-agent content and SEO machine

ContentStudio is the autonomous engine behind BizFlowAI. It researches topics, drafts, edits, optimizes for search and AEO, and publishes across multiple sites without me in the loop. It has been running for months and publishes on schedule.

Core decisions:

Decision	Choice	Why
Orchestration	Custom Node/TypeScript loop	LangGraph felt heavy for a linear pipeline with conditional edges
LLM	Claude API (Sonnet + Opus mix)	Best long-form writing quality; cost model works at my volume
Storage	Supabase (Postgres) + pgvector	One database for content, embeddings, and run history
Scheduling	Node worker on a small VPS + cron	Cheaper and simpler than serverless for a steady, predictable workload
Guardrails	Eval agent + hard checks (word count, links, factuality prompts)	The system will happily publish garbage without a gatekeeper

The interesting call was custom orchestration over a framework. I looked at LangGraph, CrewAI, and a few others. For a pipeline where I know the stages (research, outline, draft, edit, SEO pass, publish), a framework mostly adds abstraction I have to debug. A 400-line TypeScript state machine with typed transitions was clearer and cheaper to maintain. When I need branching, I add a conditional. When something breaks, the stack trace points at my code.

The choice I would revisit: I used Claude Sonnet for drafting and Opus for the final polish pass. That two-model split saved roughly 40% on inference versus running Opus end-to-end, with output quality my human review couldn't distinguish. With Opus 5 pricing where it is now, I would run more of the pipeline on the top model and drop a stage.

System 2: Sovereign AI POC with local LLMs and RAG

A different flavor entirely. This one is a proof of concept for scenarios where data cannot leave a private environment: regulated industries, government-adjacent workloads, anyone with a "no US-cloud LLM" rule. Everything runs local.

Stack:

• Ollama serving Llama and Qwen variants on a workstation with a single consumer GPU
• Postgres with pgvector for embeddings
• Full-text search (Postgres FTS) alongside vector search
• Reciprocal Rank Fusion to combine both retrievers
• A thin FastAPI layer for the app

The hard decision was hybrid search over pure vector. Pure semantic retrieval looks magical in demos and then fails on the queries that matter: exact identifiers, product codes, names with unusual spellings, acronyms. FTS catches those. Vector catches paraphrased intent. RRF (a simple rank-based fusion, no tuning weights) combines them in a way that consistently beat either retriever alone on my test set.

Rough numbers on a 12k-document corpus:

• Vector only: ~68% top-5 recall on my eval set
• FTS only: ~61%
• RRF hybrid: ~84%

That gap is the difference between "cute demo" and "the user trusts it."

The trade-off with local LLMs is honest: a 70B model on consumer hardware is slower than an API call to Claude, and the quality gap on hard reasoning is real. For extraction, summarization, and grounded Q&A over retrieved chunks, it is entirely workable. For open-ended reasoning, it isn't. I tell clients this upfront. A sovereign deployment is a constraint, not a feature.

System 3: Serverless AWS + Zendesk integration for SLA compliance

This one is older but the architecture lessons still apply. The problem: a ticketing system that was missing SLAs because the routing and escalation logic lived in a manual process. The solution: an event-driven serverless pipeline that watched Zendesk events, applied business rules, and triggered escalations before SLAs breached.

Architecture:

Zendesk webhook -> API Gateway -> Lambda (validate + normalize)
                                      |
                                      v
                                EventBridge (rules per event type)
                                      |
                    +-----------------+-----------------+
                    v                 v                 v
              Lambda (route)    Lambda (escalate)  Lambda (audit log)
                    |                 |                 |
                    v                 v                 v
                DynamoDB          Zendesk API        S3 + Athena

Why serverless and not a container on ECS or a small EC2:

1. Traffic is bursty and unpredictable. Ticket volume spikes when something breaks upstream. Lambda scales to that automatically. A container would be either overprovisioned or under.
2. The compute per event is tiny. Milliseconds of logic. Paying for an idle container 23 hours a day is silly.
3. EventBridge as the backbone meant I could add new consumers (a new escalation channel, a new dashboard feed) without touching existing code. Publish an event, subscribe a new Lambda. That decoupling saved me multiple times when requirements shifted.

The result was first-ever SLA compliance for that workflow. The lesson I keep coming back to: event-driven serverless is a genuine unlock for integration work, where the code is thin, the events are external, and the load is spiky. It is a bad fit when you have long-running processes, heavy dependencies, or per-request latency budgets that cold starts blow through.

Cold start was the one thing I fought with. For the routing Lambda that had to respond within a webhook timeout, I moved to provisioned concurrency for the two most-hit functions. That is essentially a paid warm pool. Adds cost, kills cold starts. Worth it for the hot path.

System 4: Analytics migration that cut annual cost

This wasn't a "build an AI thing" project, but the architecture reasoning is the same and it is the one that put the biggest number on the board: $30 to $60k a year in savings, depending on how you count.

The old system: a proprietary analytics platform with per-seat and per-event pricing that grew linearly with usage. The new system: a straightforward pipeline into a warehouse the client already owned, with dashboards built on top.

Decisions that mattered:

• Own the warehouse, rent the compute. Storage in the client's warehouse was already paid for. What we replaced was the marked-up ingestion and query layer.
• Batch where possible, stream where necessary. Most analytics questions do not need sub-minute freshness. Batching hourly cut compute cost by roughly an order of magnitude versus streaming everything.
• Pick a query engine that matches the shape of the questions. For this workload, a columnar warehouse with materialized views for the hot dashboards was faster and cheaper than the general-purpose engine we replaced.

The generalizable lesson: the biggest wins in cloud architecture usually come from removing a per-usage tax, not from writing better code. If your unit economics get worse as you scale, no amount of clever caching fixes it. Change the pricing surface.

The pattern behind the four choices

Looking across all four systems, the decisions that held up were the ones I made against a specific constraint, not against a generic best practice.

Local LLM vs cloud API

Cloud API wins by default. Use a local LLM when one of these is true:

• Data cannot leave a controlled environment (regulatory, contractual, sovereignty)
• You need predictable per-token cost at very high volume and you have the ops capacity
• Latency to a specific region matters more than model quality

Otherwise, Claude or OpenAI, every time. The engineering effort you save is worth more than the inference bill for almost every project under a certain scale.

Serverless vs containers

Serverless wins when:

• Traffic is spiky or unpredictable
• Per-request compute is small
• You are integrating with external event sources
• You want to add consumers without redeploying producers

Containers win when:

• The workload is steady and known
• You have long-running processes, heavy dependencies, or GPU needs
• Cold start latency is unacceptable and provisioned concurrency doesn't cover it
• Your team already runs Kubernetes well and adding serverless is cognitive overhead

For ContentStudio I picked a plain worker on a VPS. The workload runs on a predictable schedule, the container is warm, and I don't pay for cold starts I don't need. Serverless would have been the wrong answer even though I know AWS well.

Framework vs custom orchestration

I default to custom for anything I can hold in my head. LangGraph and similar frameworks earn their keep when you have genuinely complex graphs, dynamic agent spawning, or a team that needs a shared vocabulary. For a linear pipeline with a handful of conditionals, a typed state machine in code you wrote is easier to debug at 2 a.m. than a framework you learned from a README.

Vector vs hybrid search

Always hybrid. I have not built a production RAG system where pure vector search beat RRF hybrid on a real eval set. If you are shipping vector-only, you are leaving recall on the table for queries with exact terms, and those are often the queries that matter most.

What I'd do differently today

Three things:

1. Invest in evals earlier. On ContentStudio I built the eval harness after the first bad publish. It should have existed on day one. For any agentic system, the eval is the product's spine. Without it you cannot tell whether a prompt change made things better or worse, you only have vibes.

2. Pick the pricing surface before the tech. The analytics migration taught me that architecture decisions are often really pricing decisions in disguise. Before I write a line of code now, I ask: what is the unit cost, and how does it scale with success? If the answer is bad, no amount of engineering will save it.

3. Default to boring infrastructure. Postgres for state and vectors. A worker on a VPS or a Lambda behind API Gateway. A queue if I need one. Every time I have reached for something more exotic (a specialized vector DB, an orchestration framework, a bespoke event bus) I have paid for it in operational surface area. Boring wins.

The four systems above are not glamorous. They work, they run unattended, and they solve problems that had a dollar figure attached. That is the standard I hold my architecture decisions against.

If you are benchmarking your own stack choices or thinking through a build like one of these, I am always happy to talk shop. You can reach me at lazar-milicevic.com/#contact or dig through more of the blog for the deeper writeups on each system.