From self-serve AI to real architecture — what a PM learns by building the pipeline himself

Two voices run through this piece: the PM intrigued by what AI changes for a DevSecOps agile team, and the tinkerer who built the whole thing himself just to understand it. Neither one is trying to convince you of anything — just to tell the story.

This isn't meant to be academic or professional. It's the story of a personal exploration: how a plain, everyday need — stop wasting hours comparing prices by hand — dragged me from a single chat prompt all the way to a small multi-agent app running on a server. At every step, two voices talk to each other in my head. I'm keeping them separate here, on purpose.

Act 1 — The simple chat, and the wall

The tinkerer: it always starts the same way. A chat, a prompt, "find me the best price for X." It works, once, for one product. It breaks the moment you ask the same thing ten times across ten different retailers: same prompt, different response formats, a retailer missing every other time, no way to check what was just claimed. The wall shows up fast: a chat is built for conversation, not for faithfully repeating a task a hundred times in a row.

The PM: what grabs me at this point isn't the technical limit — it's how fast a single person, with a single chat window, can prototype something that would have needed a small team two years ago. That radically changes what a product manager can validate alone, before even talking budget or headcount with a DevSecOps team.

Act 2 — Why one agent isn't enough

The tinkerer: the obvious answer is to split things up. One AI to qualify the need. Another to read a product page and pull out facts. A third whose only job is to find an exact address when the first one fails. Sounds simple on paper. In practice, the first version of this split worked perfectly on two retailers — and failed silently on a third, no error, no message, just... nothing. I had to learn to never trust silence: an AI that finds nothing doesn't always say it found nothing.

The PM: I recognize this pattern. It's exactly what a DevSecOps agile team already does — separate responsibilities, keep each piece small and independently testable, never let one task carry too much logic at once. The difference here is that part of the "team" is made of AI agents instead of people. That doesn't change the principles of team governance — it just makes them more urgent, because a silently failing agent won't raise its hand in standup.

Act 3 — Getting out of the sandbox

The tinkerer: as long as everything runs in a terminal, on my machine, I can ignore a lot of problems. The moment it has to run without me — scheduled, accessible, reliable — all of those problems come back at once: where do the access keys live if it's no longer my own computer typing them, what happens when two calls fire at the same time and the provider rejects one of them, how do I even know it crashed overnight if I wasn't there to see it. Moving to serverless wasn't a choice of technical elegance — it was the only way to make those blind spots disappear.

The PM: this is the classic MVP-to-production moment, same as any project, even with a single user. The same gates open up: a minimal security review (no exposed secrets), cost control, resilience without continuous human supervision. A personal project doesn't get a pass on these gates just because it has one user — it just forces you to clear them alone, with no team to share the load.

Act 4 — Benchmarking instead of trusting

The tinkerer: two different AI search engines ended up side by side in the pipeline — one for broad discovery, the other for a targeted re-read when the first one failed. Each one offers several effort levels (low, medium, high): the higher the effort, the more thorough the answer, and the more it costs in consumed tokens. The only way to choose was to test all three levels on the same real queries and see where the cost/quality curve actually bends.

The measured orders of magnitude (approximate, not audited):

Engine Effort level Role in the pipeline Relative cost (approx.)
Perplexity Low Targeted re-read — exact address already known 1× (baseline)
Perplexity Medium Initial broad sweep across all retailers ~2 to 3× the low level
Perplexity High Tested, dropped in production (marginal gain) ~2 to 3× the medium level
Tavily Standard search Finding the exact address of a missed retailer Marginal, nearly negligible

Result: medium effort for the first broad sweep (it covers most cases without paying the high-tier price), low effort for the targeted re-read (the extra precision of the high tier barely changed anything once the exact address was already known). A validation probe across ten real products cost on the order of $0.20 in real API calls — low enough to re-measure often, real enough not to be free every time I had a doubt.

The PM: this is where profitability becomes tangible instead of an act of faith. Every full search comes out to a few cents — tracked, not estimated — against two hours of manual comparison. Even generously counting the time spent building and tuning the pipeline, the break-even point arrives after a handful of real uses. It's the same logic you'd apply to any tool in a company — cost per unit of value delivered — just scaled down to a single user.

Act 5 — Hands on the wheel

Both voices, together: no matter how many layers you stack, the pipeline never really runs on its own. A timeout that fails a call without warning. A URL that answered yesterday and is dead today. A model that invents a number with the exact same confidence as a real one — AI never hesitates before hallucinating, it does it for free and without warning. A pattern that looked reliable across ten tries and breaks on the eleventh, for no visible reason.

None of these are architecture failures. They're the normal cost of letting a machine act without continuous supervision. You can move very fast with AI — much faster than by hand — but without framing and verification at every step, the rocket drifts off orbit before anyone notices, until it's too late: wasted time, wasted money, for a claim that was never true. Keeping your hands on the wheel, concretely, means: never publish an AI answer without a confidence indicator attached, and accept that a system that sometimes fails silently should always be assumed silent until proven otherwise.

Search levels and their cost, in numbers

Operation Observed cost (approx.) Frequency
Validation probe (10 products, 5 categories) ~$0.20 Every time a hypothesis needs checking
One full search (3 candidate models, up to ~10 retailers) A few cents Every real use
Unit tests (CI) $0 Every code push
Real benchmark runs (CI, sensitive code only) A few tens of cents Only when search logic changes

These numbers are order-of-magnitude estimates gathered along the way, not a financial audit — accurate enough for an architecture decision, not for an accounting report.

The architecture, from simple chat to full kit

Stage What runs Where it lives Why this stage appeared

1. Simple chat One prompt, one answer A chat window Enough for a one-off question
2. Specialized agents Qualify / read / find an address, split into roles Still client-side Isolate failures — one role, one agent
3. Isolated business logic Calculations (total cost, confidence, ranking) in pure, tested code A dedicated file, tested with no network call Check a rule in milliseconds, not in money
4. Server Search engine calls move behind an API Serverless functions Hide the keys, manage rate limits, run unattended
5. Continuous integration Tests + real probes, automated CI pipelines Prove a fix holds, not just hope it does
6. The full kit, today Everything above, together See the stack below —

The tech stack assembled to get there:

• Language model: Claude (Anthropic) — client-side for conversation, server-side to arbitrate search results
• AI search engines: Perplexity (low/medium effort depending on the stage) + Tavily (address discovery)
• Hosting: Vercel, serverless functions
• Runtime: Node.js, plain JavaScript on the front end (no framework)
• Tests: Node's native test runner, no external dependency
• Continuous integration: GitHub Actions — one pipeline for unit tests on every push, a separate pipeline triggered only on sensitive code for real benchmark runs
• Benchmark reports: published as JSON on dedicated Git branches, readable without re-running the call
• Secrets: only as environment variables on the host and CI, never in the code repository

A short methodology note

A short format, borrowed from research, to stay honest about what these numbers are actually worth:

Sample — ten real products, five categories, one retailer dug into in depth on the reading side.

Method — a dedicated probe calls the search APIs directly (outside the production pipeline) and measures two distinct rates: discovery (is the retailer found?) and retention (is it kept as the best offer?).

Limitations — no control group, no repetition over several weeks, one project, one single pair of tools tested.

What I found looking for existing comparisons — not much. Very few public write-ups detail this exact pairing — a broad discovery engine and a targeted reading engine, benchmarked together on a real pricing use case, with real measured costs rather than estimated ones. That's not a claim that nothing exists — just that the search didn't turn up anything precise enough to cite here. The format that still carries over, even with no literature to cite: probe, measure, document the decision.

The budget, in plain sight

The whole project — probes, benchmark runs, real usage — cost a few dollars total in API calls. Not a few thousand. A few dollars.

For that price, I haven't found anywhere else a training this dense on multi-AI orchestration, search-model benchmarking, and actually shipping an agentic pipeline to production — secrets, rate limits, CI, all of it. At that cost-to-learning ratio, it's an outsized investment.

Two pictures to close on

The first one is what 90% of people would see if they opened the app: a chat window, a question asked in plain language, an answer arriving a few minutes later as a clean table. Nothing impressive to look at. That's the point.

The second is what's behind that screen: several AI agents with distinct roles, two search engines benchmarked and tuned to different effort levels depending on the step, cost tracked on every call, a server infrastructure that isolates the access keys, and someone — me — who stays the last line of verification before trusting what's on screen. The gap between the two pictures is exactly the lesson: you can do an enormous amount with AI, and move very fast, but only if someone keeps a hand on the wheel while it's running.

In short

This isn't an academic piece, and it isn't trying to prove anything to a team or a market. It's the story of a personal, almost initiatory exploration — a plain need that dragged a curious PM and a stubborn tinkerer into the inner workings of AI, with its twists, its benchmarks, its real costs, and hands that stayed on the wheel from start to finish. Nothing more than a share of experience.