DEV Community: Twio_AI

From Monolith Prompt to Event-Driven Agent — twio's Architecture Story

Twio_AI — Mon, 22 Jun 2026 09:43:42 +0000

TL;DR — Our goal was a free-form agent—like Cursor or Claude Code—where users start anywhere, ask anything, and never march through a fixed pipeline. Getting there meant progressively moving responsibility off the prompt and onto the harness: first sequencing and state, then lifecycle and triage. The conversational freedom at the top was only safe because we kept adding structural guardrails underneath.

We wanted twio to feel like Cursor, but for mortgage brokers: a free-form agent you simply talk to. Jump into the middle of a refix, ask "when does their rate expire?", paste a customer's email, and let the agent carry on. No fixed pipelines, no "step 1 of 4"—just work the way it actually arrives.

Our first version was the exact opposite of this. It could complete a refix end-to-end, but only if the broker followed the script's exact sequence. Start in the middle, ask a side question, or reply three days later, and the system would break.

This is the story of how we chased that conversational freedom across three architectures. On the surface, each rebuild looked like a structural change. Underneath, it was a steady offloading of responsibilities—stripping away what the LLM was bad at holding, and handing it to the harness.

(To understand the evolution, just know that a **refix—renewing a mortgage rate—sounds like a 4-step linear process, but in reality, it's fragmented, non-linear, and spans days. Keep that in mind.)

Architecture 1: The Monolith Script (and the Illusion of Control)

When we started, we had no prior experience with LLM harnesses. Freedom was the goal, but first we had to answer a simpler question: Can an LLM handle a real mortgage workflow end-to-end at all?

Our first architecture was one giant prompt driving the entire refix. It looked roughly like this:

You are a mortgage assistant. To do a refix:
1. Look up the customer by name; if several match, ask which one...
2. Ask for the new rate and term — unless the customer emailed...
3. Build the proposal. Repayment = P&I, unless interest-only...
4. Fill the lender form. ANZ uses fields X/Y; ASB uses...
5. Draft the sign-off email. Warm but professional...
...and ~300 more lines of rules, edge cases, and tone notes.

It was simple, predictable, and legible. Then it met production reality, and fractured under two fatal flaws:

Cognitive Overload: The longer the workflow, the more the prompt ballooned with edge cases, scattering the model's attention. It assumed a fixed sequence, but brokers don't work that way.
Tangled Coupling: One prompt carrying fetch, validate, draft, and edge-case logic became an entangled wall of text. You couldn't test "build the proposal" without running the whole machine.

The recurring disruption that exposed these flaws: "The customer replies three days later." In the monolith, there was simply no place for this. The run was over, or it was blocked waiting for a turn that wasn't coming.

The Diagnosis: A monolithic prompt fuses three things that should never mix.

Fused together, changing one risks breaking the others.

The mandate for our first refactor was clear: Pull these three apart, and stop hardcoding the sequence.

Architecture 2: Planner + Steps (Decoupling Flow and State)

We shattered the monolith into Steps. A step is a small, single-purpose agent with a typed contract: a focused prompt, a whitelist of tools, and strictly defined I/O schemas for state.

A Planner produced an ordered sequence, and an orchestrator executed them. This cleanly separated the monolith's responsibilities:

The planner orders typed steps; each runs its own small, scoped LLM prompt.

The Breakthrough: Context as Memory

The most critical insight of this era was how we handled state. Steps didn't get the whole conversation history; they got a scoped view.

refix_proposal reads parties, writes proposal, and never sees the rest. This isn't just access control; it's memory management. A model reasons far more accurately over a small, relevant context than a massive one. Scoping the view didn't just tidy the code—it made the steps sharper.

Architecture 2 was a massive leap forward. We shipped a lot of features on it. But it carried a hidden, fatal assumption: It assumed work is synchronous and continuous.

To start work, the broker had to land on a homepage and select the workflow type from a dropdown (e.g., "Refix"). Choosing "Refix" launched the fixed plan. That demands the broker knows the shape of the work before anything runs. (You don't pick "refactor" from a menu before Cursor listens—you just start typing).

Furthermore, the plan assumed run-to-completion. So, our recurring disruption returned: The customer replies three days later. By then, the pipeline had finished or stalled. We tried bolting on a separate "inbox" to triage incoming items, but it felt like stapling an async patch onto a synchronous system. We deleted it.

The Diagnosis: A refix is not a workflow you execute. It's a long-lived case, fed by disconnected events over days.

The mandate for the next refactor: Stop making the broker declare the work up front. Stop pretending work runs to completion.

Architecture 3: The Open Case (Event-Driven Paradigm)

We collapsed the dropdown menu of workflow types into one universal Case. Every inbound event—email, chat message, late reply—flows into the same entry point. The first thing that runs is the planner, but it now has a completely different job: Triage.

Instead of sequencing a known workflow, the planner looks at a single event and picks a path:

Triage routes each event — answer directly, wake a case, or compose steps. A late reply just re-enters triage.

Three mechanisms make this architecture work:

1. Long-Lived Cases with Implicit Waiting
After acting, a case goes quiet. In our system, the absence of new messages is the waiting state—there is no dangling pipeline. What broke Architecture 1 and stalled Architecture 2 is now trivial.

2. First-Class "Do Nothing" Path
When a broker asks "when does their rate expire?", the triage planner answers it directly and stops. No four-step plan is generated. Often, no plan is the right plan.

3. Declarative Playbooks
Domain knowledge moved out of code and into Markdown playbooks. This allows mortgage experts to edit the "how" without touching the engine code.

Guarding the Freedom: Data-Flow Validation

Handing a language model the freedom to compose its own step sequences is dangerous. We put a hard floor under it: a data-flow validator that rejects an incoherent plan before a single token of work is spent.

If the planner emits refix_proposal before parties, the validator hands back a precise error, and the planner self-corrects.

This introduces a brilliant mechanism we call Soft Reads vs. Hard Reads:

Hard Read (Correctness): An ordering constraint. form_filler must wait for parties. If it's missing, the validator blocks it.
Soft Read (Composability): Visibility-only. form_filler can see refix_proposal if it exists, but won't force it into the plan. This lets the step be dropped into a one-off request without dragging the whole refix sequence along.

One reads declaration scopes the step's memory and acts as an edge in the dependency graph. One declaration, two jobs.

This is where twio finally felt like the free-form agent we set out to build. It’s the synthesis of our journey: Freedom at the top, structure at the bottom, knowledge in prose.

Why Not Just Use LangGraph?

We evaluated LangGraph and similar agent frameworks before building our own. They are excellent tools, but designed for a fundamentally different paradigm: the static graph. Our workflow is a dynamic, event-driven beast.

Forcing our architecture into a graph exposed three core mismatches:

1. The engine isn't the graph; it's the contracts.
Frameworks give you nodes, edges, and shared state. But twio's real value isn't topology—it's the typed step contracts (reads/softReads), the triage planner's judgment, and the prose playbooks. We'd still have to write all of that, just wrapped in someone else's StateGraph syntax. We'd take on a heavy dependency while our actual core logic (like the 40-line data-flow validator) becomes harder to tweak.

2. Pre-compiled graphs vs. runtime composition.
A graph requires defining paths ahead of time. Our planner composes a fresh step sequence at runtime for every single event, checked by a validator before execution. You can force a static framework to do dynamic routing, but then the graph stops being the source of truth. You end up fighting the framework's core abstraction.

3. Shared state vs. strict isolation.
Our central thesis is that a step should only see its declared context slices. Graph frameworks default to passing a massive shared state object through every node. We’d be constantly fighting the framework's default behavior to enforce the one rule we care about most.

If our workflows were mostly static and predictable, LangGraph would save us real work. Ours aren't.

But the deeper point is this: our event-driven case design entirely sidesteps the need for durable execution. Frameworks earn their keep by handling suspend/resume, timers, and exactly-once side effects. Because "no new messages" is our waiting state, the system naturally suspends without needing underlying infrastructure to maintain it. If we ever actually need durable execution, we’ll reach for a dedicated engine like Temporal—not an agent framework.

What's Still Hard

Top-level freedom introduces new failure modes. The planner can match the wrong playbook. It can compose a valid but suboptimal plan. Prose playbooks can drift from what the steps actually execute. Our guardrails catch a majority of this, but this is a live frontier, not a solved problem.

The through-line of this whole journey is that you don't arrive at the architecture—you get pushed into it by reality. Ultimately, the customer who replies three days late wrote more of our architecture than we did.

Architectural Takeaways

If you are building long-lived, agentic systems:

Don't make the model hold what the harness holds better. Offload sequencing, durable state, and lifecycle management to traditional code.
Treat context as memory. Strictly scope each step's view. Models reason better over small, relevant contexts.
Model the arriving unit as an event, not a task. Work is fragmented; your architecture should expect interruption.
Build a first-class "just respond" path. Forcing a workflow when a direct answer suffices destroys the user experience.
Earn freedom with guardrails. Use static validation (like our data-flow checker) to make self-composing agents safe.

We're building twio, an AI assistant for mortgage brokers. If you're wrestling with the same questions about long-lived, event-driven agent architectures, we'd love to compare notes.

Why Twio Chose Vertex AI Search over pgvector for Production RAG

Twio_AI — Mon, 22 Jun 2026 00:00:58 +0000

When we first built RAG at Twio, pgvector was the obvious pick. Our business data was already in PostgreSQL, and dropping embeddings into the same database was the fastest path to a working product.

For the first version, that was right. As we scaled, the problem stopped being "how do we store vectors?" and became "how do we reliably understand thousands of broker documents, emails, and attachments in production?" That changed the answer. Today, Vertex AI Search is our main retrieval layer.

RAG is Twio's memory layer, not a search feature

Twio is an AI SaaS for loan brokers. A single client case is a mess of fragmented information:

email threads
payslips, bank statements, identity documents
loan forms, lender requirements
handwritten notes, follow-up emails, missing-document requests

The AI needs to answer questions like:

What documents has this client already sent?
Which email mentioned the missing requirement?
Does this bank statement support the income claim?
Summarize all documents related to this borrower.

If retrieval is weak, the answer is weak. If indexing lags, context is missing. If parsing is wrong, the model sees the wrong evidence. RAG isn't a feature on the side — it's the memory layer of the product.

Why pgvector was the right first choice

Twio is a multi-tenant SaaS, so retrieval can't just return "similar content" — it has to return similar content scoped to the right user, client, application, or file. pgvector made that trivial: embeddings sat next to the business records, joined cleanly, and filtered with plain SQL.

The early wins were real:

no new infrastructure
low cost, easy local dev
SQL inspection for debugging
straightforward metadata filtering
fast to ship

It let us build the first version quickly and learn from actual usage. That matters more than people give it credit for.

Where pgvector stopped paying off

pgvector didn't fail. It did exactly what it's designed for. The issue was that vector storage is only one slice of the RAG pipeline, and pgvector left every other slice to us:

download attachments
extract text from PDFs, run OCR on scans
chunk documents, generate embeddings
design metadata, build retrieval queries
tune indexes, improve ranking
monitor Postgres load, debug retrieval quality

A clean PDF is easy. A scanned bank statement isn't. An email body is easy. An email with five attachments, lender forms, tables, and partial OCR isn't. A demo dataset is easy. A real broker workspace with years of historical emails isn't.

With pgvector, every weakness in that pipeline was ours to fix. When retrieval quality dropped, the suspect list ran all the way from OCR through chunking and embedding to vector distance, SQL filtering, ranking, and DB performance. The extension is simple. The production RAG system around it isn't.

The cost shifted from cloud bill to engineering time — and engineering time was the constrained resource.

pgvector vs Vertex AI Search, in Twio's terms

Scenario	pgvector	Vertex AI Search
Clean text PDF	We own extraction, chunking, embedding, storage, search	Vertex handles most of the indexing and retrieval workflow
Scanned document	We build or integrate OCR ourselves	Vertex absorbs much of the document-processing logic
Broker asks a document question	We own query design, ranking, filtering	Managed search with stronger out-of-the-box quality
Attachment bursts	Postgres carries more search and indexing load	Search workload lives outside the main database
Debugging	Excellent SQL visibility, but many custom layers to inspect	Less low-level control, but far less custom infra to debug
Cost	Lower direct service cost	Higher service cost, lower engineering and maintenance cost
Production readiness	Significant custom work required	Easier to operate as a managed layer

pgvector was cheaper as a database extension. Vertex is cheaper as a product decision. The cloud bill is one input; engineering time, reliability, and iteration speed are the bigger ones at our stage.

Why Vertex fits Twio's shape of problem

Twio's RAG problem is document-heavy. We aren't searching short snippets — we're dealing with messy broker PDFs, scans, forms, tables, and forwarded attachments. Vertex helps in four concrete ways:

Less infrastructure to own. Indexing and retrieval are handled by the managed layer, so we don't rebuild that surface.
Less document-processing logic to maintain. OCR and parsing for messy broker files is one of the harder parts of the pipeline to keep healthy. Vertex covers much of it.
Postgres stays focused on what it's good at — business data, transactions, workflow state — instead of competing with OLTP work for the same resources.
It scales more naturally as document volume grows.

Vertex isn't free, but the alternative isn't either. Building OCR, indexing, ranking, monitoring, and tuning ourselves has its own bill — paid in engineer-weeks.

What pgvector still does well

pgvector is still a strong choice when:

data volume is moderate
you're already on Postgres and want retrieval close to your data
your documents are already clean text
you need tight SQL filtering and full control
you want a fast, low-cost first version

For us, it was the right first implementation — and it taught us what retrieval the product actually needed. It may stay in the stack for internal or fallback use cases.

Takeaway

The lesson from Twio's RAG evolution is simple:

Start with the tool that helps you learn fastest. Move to the tool that helps you operate best.

pgvector got us to a working RAG system quickly. As the product matured, the real challenge shifted to document processing, indexing quality, and operational reliability — and at that point, Vertex AI Search became the better fit. It costs more as a service and less as a system to maintain. For a SaaS at Twio's stage, that's the trade that matters.

From pg-boss to Cloud Tasks: Fixing Queue Bursts and DB Connection Failures on Serverless

Twio_AI — Tue, 02 Jun 2026 03:23:40 +0000

At Twio we picked pg-boss for our job queue, ran into trouble when we went serverless, looked at Pub/Sub, and ended up on Google Cloud Tasks. This is what each queue got right, what it got wrong for our workload, and the rule we landed on for choosing between them.

The workload

Twio is an AI SaaS for loan brokers. The piece that needs a job queue is email processing: download an email, parse the body and attachments, OCR, classify with an LLM, write structured data, and index for RAG. One email with five attachments easily becomes 30+ background jobs. A batch upload becomes hundreds.

Why pg-boss worked — until it didn't

Our database was Postgres on Neon, so pg-boss was the obvious starting point. No extra infrastructure, and one feature we genuinely loved: transactional enqueue. Because jobs live in the same database as business data, you can create a job in the same transaction as the row that triggered it. No dual-write problem, no "DB succeeded but the queue API failed" inconsistency.

It also gave us retries, delayed jobs, dead-letter queues, dedup keys, and full SQL visibility into stuck or failed jobs. For a Postgres-first app on always-on infra, it's an excellent tool.

Then we moved heavy processing to Cloud Run, and the cracks showed up.

pg-boss polls. Neon suspends. They want opposite things.

pg-boss runs a query roughly every 1–2 seconds to look for the next job, plus maintenance queries. Neon autosuspends compute when nothing touches the database. If the queue is polling every second, Neon's idle timer never expires — you pay for always-on compute even when the queue is empty.

Worse, when Neon did manage to suspend, the next poll had to wake it. That wake-up takes hundreds of ms to a few seconds, and queries that triggered it would fail with Connection terminated, ECONNRESET, or timeouts. Pooled connections made it worse: the pool kept sockets that the server had already closed during suspend, and the next polling cycle picked one up and broke.

This isn't a pg-boss bug. It's an architectural mismatch.

Why Pub/Sub wasn't the answer

Pub/Sub is event-driven — no polling against Postgres, Neon can suspend freely. That fixed the obvious problem, but introduced a worse one for our shape of work.

Pub/Sub is built to move messages fast. We needed a queue that moves messages carefully.

Two specific failure modes hit us:

Retry amplification. A parent import job publishes 100 child parse messages, then crashes before acking. Pub/Sub redelivers the parent. The parent re-publishes 100 children. After a few retries, you have hundreds of duplicate child jobs.
No native job-level pacing. If 300 messages land at once, subscribers consume them as fast as they can — slamming our parser, Neon, the LLM provider, and third-party APIs simultaneously. Pub/Sub has flow control on the subscriber side, but it's not the kind of per-queue dispatch throttle we needed.

Plus the ack-deadline problem on long parse jobs, where a missed lease extension causes redelivery while the original is still running.

All of these are solvable with idempotency keys, outboxes, and bounded retries — but at that point you're rebuilding what a job queue should give you out of the box.

Why Cloud Tasks fit

Cloud Tasks is push-based: when a task is due, Google sends an HTTP request to our handler. When there are no tasks, nothing touches our database. That alone resolved the pg-boss/Neon conflict — Neon suspends, costs drop, no more wake-up connection errors.

But the real reason it fit was per-queue dispatch control:

# queue.yaml
- name: email-parse
  rateLimits:
    maxDispatchesPerSecond: 10
    maxConcurrentDispatches: 20
  retryConfig:
    maxAttempts: 5
    minBackoff: 10s
    maxBackoff: 600s
    maxDoublings: 4

Enqueue 300 tasks in a second and Cloud Tasks won't deliver them all at once — it paces dispatch to the limits we set. Our parsers, Neon, and the LLM provider stay protected from bursts.

It also gives us operational levers Pub/Sub doesn't: list tasks, inspect depth, pause a queue, purge a bad batch. When a fan-out goes wrong, we can stop it.

What Cloud Tasks doesn't solve

Two things, both important.

It's still at-least-once. A handler can finish the work and Cloud Tasks can still redeliver if the HTTP response is lost. Handlers must be idempotent.

Fan-out duplication is still possible. If the parent creates 100 child tasks and then fails before returning 200, the retried parent creates them again. The fix here is deterministic task names:

parse-{emailId}-{attachmentId}

Cloud Tasks rejects duplicate names within its retention window, so the second attempt is a no-op. But you have to design for it — it's not automatic.

And it doesn't recover transactional enqueue. Cloud Tasks lives outside the database, so creating a task after a DB write is a dual-write. If you need strict atomicity, the answer is still an outbox: write the business row and an outbox row in one transaction, have a relay publish to Cloud Tasks and mark the row published. No external queue makes this go away.

The rule we landed on

Queue selection isn't about finding the best queue. It's about matching the queue to the runtime model.

pg-boss for small internal jobs in always-on services where Postgres transactionality matters.
Cloud Tasks for cross-system, serverless workflows where we need to protect Neon, LLM providers, and third-party APIs from bursts.

And three rules that apply regardless:

Every handler is idempotent.
Fan-out children have deterministic keys.
If enqueue must be atomic with a business write, use an outbox.

Cloud Tasks fixed our infrastructure mismatch, but the real win was clarifying what the queue is responsible for. Infrastructure handles scheduling, retries, and rate limits. Correctness belongs to the application.