Raphael Mendes

Posted on Oct 12

Measure Before You Scale: SofiaVoice, Cortex-Flow and the Commerce Pipeline

#ai #python #opensource #programming

Measure Before You Scale: SofiaVoice, Cortex-Flow and the Commerce Pipeline

RS4 Lab · Rs4Machine · Technical article · October 2026
Experiments covered: EXP005 (SofiaVoice v2.0), EXP006 (Cortex-Flow v1 → v2), EXP007 (Commerce Pipeline v3.0)

Experiment first. Measure. Understand. Only then scale.
— RS4 Lab operating principle

TL;DR

Over roughly three weeks (12 Sep – 3 Oct 2026), RS4 Lab took three systems through the same loop: write down a hypothesis, instrument everything, run it, read the numbers, and only then decide what to change. Every run left a JSON metrics file and a Markdown report next to the code.

System	What it was	What it became	Signal from the data
SofiaVoice	Synchronous voice assistant, gTTS, ~5.8 s per turn	Async FastAPI service, Edge-TTS Neural, API guardrails	Production audit: 5.81 s → 2.80 s (−51.7%). TTS was 76% of the old latency.
Cortex-Flow	A linear script with 4 LLM calls, no memory	A typed state graph with a deterministic router, vector memory seeded with the lab's philosophy, 8 nodes, two routes, an evaluator and a human gate	Full run on a CPU: Route A 8.0 min, Route B 33.9 min, 2/2 routes SUCCESS, with a final verdict left pending human decision
Commerce Pipeline	Ad hoc collection of offers and freelance leads	A deterministic, LLM-free feeder with a strict payload contract into Cortex-Flow's inbox	84 assertions passing, 0 exceptions, 16–17 items exported per daily run in 5–9 s

The more useful result is the method. All three systems ended up with the same shape: deterministic code where determinism is possible, models only where judgment is needed, telemetry on every boundary, and a human at the end of the chain.

1. The method: how Rs4Machine runs an experiment

RS4 Lab is not a product team and not an agent playground. It is a lab with a method, and the method is written down. In Cortex-Flow v2 it was literally ingested into the system's memory (Section 3.4). These are the rules that shaped every decision in this article:

Metrics are mandatory. Every project that involves local models gets a Metrics/ folder, and every run writes a machine-readable JSON and a human-readable report. (In the lab's DNA file this is called "Nova Regra de Filosofia RS4 — CEO RS4MACHINE".)
Living Checklists. Every build cycle starts, advances and closes through a checklist. Item numbers such as Item 07 — Seed DNA or Item 13 — End-to-end test show up inside the metrics files.
Human approval is part of the protocol. The working sequence is understand → discuss → plan → scope → human approves → execute → verify. No agent closes its own loop.
Zero-cost constraint (R$ 0.00). Local models through Ollama, the Python standard library where possible, SQLite instead of servers, no paid APIs in the experimental loop.
Four projects > a million projects. A few systems, taken deep, beat many shallow demos.
Responsibility. "Technology can expand our capability. It should not replace our responsibility."

Figure 1 — The RS4 experimental loop

2. SofiaVoice: from a slow pipeline to a measured one (EXP005)

2.1 What it was

SofiaVoice v1 was a classic three-stage voice loop: speech-to-text → LLM → text-to-speech, run synchronously. The first job of the experiment was not to optimize. It was to find out where the time actually went.

Stage (v1)	Engine	Latency	Share of total
STT	Whisper	0.718 s	12.4%
LLM	LLaMA 3.3	0.673 s	11.6%
TTS	gTTS	4.415 s	76.1%
Total		5.806 s

That table set the agenda. The intuitive target in an AI app is the model, but here the LLM was the cheapest stage. Three quarters of the user's wait was speech synthesis. A faster LLM alone could have saved at most 0.67 s, while TTS had 4.4 s to give back.

2.2 What it became

SofiaVoice v2.0 was rebuilt as an async service with every stage swapped for a faster engine, and the TTS stage got the most attention:

Stage	v1.0	v2.0	Engine change	Gain
STT	0.718 s	0.613 s	Whisper → Whisper Large V3 (Groq)	−14.6%
LLM	0.673 s	0.412 s	LLaMA 3.3 → `openai/gpt-oss-20b` (Groq)	−38.8%
TTS	4.415 s	1.850 s	gTTS → Edge-TTS Neural (FranciscaNeural)	−58.1%
End-to-end	5.806 s	2.800 s		−51.7%
Audio payload	—	30.8 KB		−52.2%

Source: test05-sofia-v2.0.0/metrics.json, production audit (Render + Vercel), 12 Sep 2026.

The release also added API guardrails, and they were audited as part of the release rather than treated as a footnote: 10 MB payload limit, XSS sanitization, restricted CORS, and a null-audio guard.

Figure 2 — SofiaVoice v1 vs v2 architecture

2.3 What the local stress tests added

After the release, the lab pointed a stress harness (test_stress_sofia.py) at the /voice endpoint on a local machine and set two SLOs: p95 total latency ≤ 15 s and error rate ≤ 5%. Four runs followed, and each one taught something different:

Run	Input	Result	p50	p95	SLO
test01	synthetic	5/5 `ConnectionRefusedError` (server not running)	—	—	❌
test02	real WAV (`test02-sofia.wav`)	3/3 HTTP 200	6.46 s	23.33 s	❌
test03	synthetic ("Submarino")	3/3 HTTP 200	5.70 s	6.35 s	✅
test04	synthetic ("Submarino")	3/3 HTTP 200	5.78 s	6.64 s	✅

The runs support three findings:

The harness fails honestly. test01 is a failed environment, not a failed product, but it was recorded and kept instead of deleted. A measurement system that can say "100% error" is one you can trust when it says "0%".
HTTP 200 does not mean success. In test02, request #2 returned 200 with an empty ai_response and a 148-byte body, and request #1 took 25.2 s and returned only 720 bytes of audio for a full sentence. By status code the run was "100% success". By content, one in three responses was empty and one was degenerate. This is the strongest argument in the dataset for content-level assertions (non-empty text, minimum audio bytes per character) on top of status codes. It also suggests the null-audio guard needs a sibling: an empty-reply guard.
Local end-to-end ≈ 2× the production audit. The local p50 (~5.7–5.8 s) is about twice the 2.80 s production figure. The two numbers measure different things: the audit measures per-stage time on the server, while the stress test measures the full client round trip from a Windows workstation, including upload, base64 encoding and the network hops to the inference and TTS providers. Both are valid, but they should never be quoted interchangeably.

Payload size tracks answer length: decoded audio ranged from 22 KB for a one-line answer to 85 KB for a long one. The cheapest latency optimization left for SofiaVoice is probably prompting for shorter spoken answers, not a faster engine.

Figure 3 — Latency distribution across SofiaVoice test runs

3. Cortex-Flow: from a script to a graph with memory and judgment (EXP006)

Cortex-Flow is the lab's local thinking engine. It takes a raw idea written by a human (the ideia bruta) and runs it through specialized agents that map, research, criticize, synthesize and, in v2, build and evaluate. Everything runs on one Windows 11 workstation, CPU-only, through Ollama, at zero cost.

3.1 What it was: v1 (the "Claudio Project")

v1 was a straight line: four agents called in order by an orchestrator script, each receiving the raw idea and the previous outputs.

ideia_bruta → [1] Structural Mapper → [2] Tech Scout → [3] Acid Critic → [4] Synthesizer → Refinado_*.md

The first runs used a 256-token cap per agent, and the reports show the cost: the 14 Sep report is flagged "Modo: Chunking Controlado", and Agent 1's output stops mid-word. The fix was to raise the cap to 1,024 tokens. Then the lab measured what that cost:

v1 metric (18 Sep run)	Value
Model	`qwen2.5:7b`, temperature 0.2
Agents OK	4 / 4
Prompt / response tokens	2,363 / 3,078
Pipeline time	605.9 s (vs 219.7 s with the 256 cap, +176%)
Effective generation rate	~5.1 tokens/s
Output artifact	`Refinado_20260918_180917.md`, 14.2 KB

Agent	Latency	Response tokens
Structural Mapper	92.2 s	416
Tech Scout	192.8 s	1,024
Acid Critic	195.2 s	1,024
Synthesizer	123.4 s	614

The run was green, but the table shows a problem: two of the four agents returned exactly 1,024 tokens, which means they hit the cap. Raising the limit moved the truncation point without removing it. v1 also had structural limits that more tokens could not fix:

No memory. Each run started from zero, and past decisions were not available.
No routing. A copywriting request and a code request went through the same four steps.
No judgment of its own output. The synthesizer's answer was the end of the line.

3.2 What it became: v2 architecture

v2 was rebuilt from scratch as an isolated package (cortex_flow_v2) and organized as a compiled state graph (app.invoke(...)) over a typed shared state:

class CortexState(TypedDict):
    entrada_bruta: str        # raw human idea
    frente_alvo: str          # route / intent
    analise_mapeador: str
    analise_techscout: str
    analise_critico: str
    sintese_final: str
    contexto_rag: list        # retrieved memory
    erros: list               # errors travel with the state, not in stderr

Two routes leave a deterministic triage node:

Route A — COPY_OFFER:        triagem → redator → critico → evaluator → END
Route B — BUILDER_TEMPLATE:  triagem → mapeador → techscout → critico
                             → sintetizador → builder → evaluator → END

Figure 4 — Cortex-Flow v1 vs v2

3.3 Design choices and what the data says about them

Choice 1: Routing is deterministic. No LLM is used to pick the route.
The triage node scores the input against keyword lists (copy, oferta, landing page, call to action versus freela, código, template, …) plus explicit Frente Alvo markers.

Triage run	Copy score	Builder score	Route	Time
Email-campaign offer	13	0	COPY_OFFER	0.17 ms
Freelance landing-page template	1	7	BUILDER_TEMPLATE	0.33 ms

On this hardware, any LLM call costs between 30 s and 5 min. Routing with a model would have added minutes to every run and introduced non-determinism at the most consequential branch in the graph. The keyword router costs under a millisecond, is fully explainable (every matched clue is logged), and is right on both test inputs. The rule is the lab's: use the model where judgment is needed, and code everywhere else.

Choice 2: One model family, with temperature tuned per role.

Node	Model	Temperature	`num_predict`
Mapper	qwen2.5:7b	0.1	512
Tech Scout	qwen2.5:7b	0.2	1,024
Copywriter	qwen2.5:7b	0.7	1,024
Critic	qwen2.5:7b	0.2	1,024
Synthesizer	qwen2.5:7b	0.3	1,536
Builder	qwen2.5-coder:7b	0.2	2,048
Evaluator	qwen2.5:7b → fallback qwen2.5:3b	0.2	1,024

The analytical roles run cold (0.1–0.3). Only the copywriter is allowed to be creative (0.7), and the builder gets a code-specialized model. Token budgets grow along the chain because later nodes have more to consolidate.

Choice 3: Hardware guards are part of the architecture.
The integration test verifies them before running: 1,024 tokens max per request, 240 s timeout per request, and a 2 s pause between node transitions so the CPU can cool down and Ollama's memory can settle. They proved their worth in the real run (see 3.5).

Choice 4: Each node reports its own telemetry.
Every node writes its own metrics_<node>_v2.json with RAG hits and distances, prompt size, HTTP status, total latency, eval_duration, prompt and response tokens, done_reason, every model attempt (including failed ones), artifact paths, and a py_compile check of its own source. The integration test proves which nodes ran by checking which metrics files were updated in that execution, so the route is shown by evidence rather than by log text.

3.4 Where the philosophy goes into the system: memory seeded with the lab's DNA

The most distinctive v2 decision is the Seed DNA step (Checklist Item 07). The lab's philosophy document, filosofia_rs4.txt (15,027 characters, 537 lines), was split on its numbered sections into 19 chunks, embedded with nomic-embed-text (768 dimensions), and stored in a persistent ChromaDB collection named decisoes. Ingestion took 8.3 s, and a re-run left the count at 19 → 19, so the ingestion is idempotent.

Every LLM node then queries that collection before it prompts the model. In practice, the lab's principles are retrieved and placed into the context window of every agent. The telemetry shows which principles each node actually retrieved:

Node	Top retrieved DNA sections
Copywriter	Offer-tone rules ("never invent results, prices or statistics"), §14 Greater objective of RS4
Tech Scout	§14 Greater objective, §16 Living Checklists, §15 Mandatory Metrics/
Critic	§15 Mandatory Metrics/, §16 Living Checklists, §11 How AI must work with Raphael
Synthesizer	§15, §11, §16
Builder	§15, §16, §11
Evaluator (k=5)	§15, §29 Principle of Continuity, a past Tech Scout decision, §31 Final Rule, §2 Fundamental Philosophy

Three sections dominate: measure everything (§15), work through living checklists (§16) and a human approves (§11). Those three ideas are also the core of the lab itself. The Critic and the Evaluator do not judge against a generic idea of "quality". They judge against the RS4 criteria, so the Evaluator's rubric reads like the lab's manifesto: real utility, radical simplicity, R$ 0.00 cost, security, privacy/LGPD, metrics/evidence, clarity, immediate executability.

The memory also learns. After each run, the Tech Scout writes its decision back to decisoes (for example techscout_d6e36f2720fd40aa, 3,080 characters). On 3 Oct, the Evaluator retrieved a Tech Scout decision written on 1 Oct as its third-closest context. The system's own past decisions now inform its later judgments.

Figure 5 — The DNA memory loop

3.5 The end-to-end run (3 Oct 2026)

The integration test pulled two real pending opportunities from the Commerce inbox (Section 4) and ran both routes on live models:

	Route A — COPY_OFFER	Route B — BUILDER_TEMPLATE
Opportunity	`COM-2026-001` Email campaign for the annual offer	`COM-2026-002` Landing-page template for a portfolio freelance gig
Nodes executed = expected	4 / 4 ✅	7 / 7 ✅
Wall time	481.6 s (8.0 min)	2,031.9 s (33.9 min)
Hardware pauses	8 s	14 s
Evaluator score	3 / 5 → REVISE	4 / 5 → APPROVE
Final decision	`PENDENTE_HUMANO`	`PENDENTE_HUMANO`
Status	SUCCESS	SUCCESS

Total: 2,513 s (~42 min), 2/2 routes, 0 state errors.

The run produced four results worth noting.

1. The pipeline refused its own copy. Route A's Evaluator scored the generated ad 3/5 and recommended REVISE. It flagged missing evidence for claims, no mention of data privacy/LGPD, and an unclear value proposition. That is the intended behavior: the system's own critic rejected the system's own marketing output against the lab's rules ("never invent results"). A pipeline that approves everything has no value as a gate.

2. Where the time goes (Route B):

Node	Time	Share
Triage	0.3 ms	~0%
Mapper	100.2 s	4.9%
Tech Scout	175.4 s	8.6%
Critic	357.4 s	17.6%
Synthesizer	387.4 s	19.1%
Builder	599.9 s	29.5%
Evaluator	397.6 s	19.6%

Figure 6 — Route B time budget

3. Latency is a function of tokens generated. Every 7B node decoded at almost the same speed:

Node	Decode rate
Mapper	6.5 tok/s
Tech Scout	6.3 tok/s
Builder (coder)	6.0 tok/s
Copywriter	5.9 tok/s
Synthesizer	5.9 tok/s
Critic	5.8 tok/s
Evaluator (3B)	9.7 tok/s

About 6 tokens/s is the ceiling of this machine for a 7B model. Prompt engineering will not make a node faster per token. The levers that work are fewer output tokens, smaller prompts, or a smaller model. There is also a second-order effect: nodes ran 1.9–2.7× slower inside the full route than in their isolated unit tests (Critic 150 s → 357 s, Builder 287 s → 600 s). The shared state grows along the chain, so downstream prompts get longer and prompt pre-fill becomes a real cost. It was 25% of the Critic's time and 55% of the Evaluator's.

4. The guards fired, and the run still completed. The Evaluator's prompt had grown to 3,676 tokens (an 11.9 K-character rubric plus 6.7 K characters of deliverable). The 7B attempt hit the 240 s timeout, the node fell back to qwen2.5:3b, and it finished in 152 s with a valid score. All of this was recorded in tentativas[]. The guards turned a potential hang into a logged degradation. The lesson is uncomfortable but useful: the quality gate is currently the most expensive node, and under load it is judged by the smallest model. That is now an explicit item for v3.

4. The Commerce Pipeline: a deterministic feeder (EXP007)

4.1 What it was

Opportunities (affiliate offers, freelance leads) were found and handled manually or by ad hoc scripts. There was no single record of what had been seen, no guarantee of correct affiliate tagging, and no way to hand work to Cortex-Flow.

4.2 What it became

RS4 Commerce v3.0 is a small, LLM-free pipeline (ollama_acionado: false in every run) with one job: produce clean, typed, de-duplicated opportunities and drop them in Cortex-Flow's inbox.

coletor_ofertas ─┐
                 ├─► utils_links (sanitize + tag) ─► db_manager (SQLite, SHA-256 id, decision history)
coletor_multifrentes ┘                                    │
                                                          ▼
                                     exportador_cortex ─► cortex_flow_v2/inbox/payload_commerce.json
                                                          │
                                                          ▼
                                     conector_commerce ─► Cortex-Flow triage (Route A / Route B)

The design choices are clear in the test suites (84 assertions across 3 suites, all passing):

Link sanitization is idempotent and conservative (13 checks). Affiliate tags are injected per network (Amazon tag, Shopee af_web_id, Mercado Livre as). Existing tags are replaced, never duplicated. Fragments are preserved, hosts are normalized, and unknown domains and short links pass through unchanged. The rule is never break a link you do not understand.
A strict payload contract (27 checks). Exactly six keys (origem, timestamp, tipo_entrada, titulo, link_sanitizado, entrada_bruta), UTC ISO-8601 timestamps, and an enumerated tipo_entrada (OFERTA_AFILIADO / OPORTUNIDADE_FREELA). An invalid type raises ValueError and nothing is written to disk. The entrada_bruta field is already written in Cortex-Flow's Frente Alvo format, so the downstream triage reads it natively.
The database records history, not just current state (44 checks). The primary key is the SHA-256 of the sanitized link, so duplicates are rejected structurally (IntegrityError) and guarded procedurally (existe_oportunidade()). Decisions live in their own table as an append-only history (PENDENTE → ENVIADO_CORTEX), so the system can always show when and why an item changed state. Test and production databases are path-isolated.

Daily run	Items	Offers	Freelance	Exported type	Duration
28 Sep 12:03	16	12	4	OFERTA_AFILIADO	8.81 s
30 Sep 04:29	17	12	5	OFERTA_AFILIADO	8.46 s
1 Oct 19:15	17	12	5	OPORTUNIDADE_FREELA	5.03 s

Collectors run in sub-millisecond time, and the cost is almost entirely the export step. The pipeline does not need to be smart. Its job is to make sure that, when a model is finally called, it gets a clean and well-formed input.

Figure 7 — The RS4 system map: Commerce feeds Cortex, Cortex asks a human

5. Philosophy → mechanism → evidence

This is the core claim of the article. Each principle maps to something concrete in the code, and each mechanism left evidence in the repository:

RS4 principle	Mechanism in the systems	Evidence in the data
Experiment first, measure, understand, then scale	v1 baselines before any rebuild. Cortex nodes unit-tested in isolation before integration	SofiaVoice v1 table (TTS = 76%). Cortex `00`–`10` node metrics before `11` integration
Mandatory `Metrics/`	Every node, test and run writes JSON + Markdown	14 Cortex v2 metrics files, 5 Commerce runs, 5 SofiaVoice runs
Living Checklists	Checklist item numbers embedded in metrics (`Item 07`, `08.x`, `13`)	`dose` / `item` fields in every v2 metrics file
Human makes the final decision	Evaluator only recommends. The output status is fixed to `AGUARDANDO_DECISAO_HUMANA`	`decisao_final: PENDENTE_HUMANO` in both routes
R$ 0.00 / local first	Ollama on CPU, ChromaDB on disk, SQLite, stdlib-only setup	`stdlib_only: true`. No paid API in Cortex or Commerce
Simplicity: code where possible, model where necessary	Keyword triage, deterministic Commerce, LLM only for judgment	Triage 0.17–0.33 ms. `ollama_acionado: false`
Honest evidence	Failed runs kept. Fallbacks and timeouts logged	SofiaVoice test01 (100% error) kept. Evaluator `TimeoutError` in `tentativas[0]`
Never invent results	DNA-backed copy rules and an evaluator that penalizes unsupported claims	Route A ad scored 3/5 → REVISE
Continuity of the philosophy	Philosophy embedded as retrievable memory	§11, §15, §16 retrieved by every downstream node

6. What the data does not prove yet

The same discipline requires listing the gaps. These are the open issues, taken from the lab's own files:

Sample sizes are tiny. SofiaVoice runs have n = 3, so "p95" and "p99" are interpolations, not tail measurements. Cortex-Flow's integration result is one run per route. The numbers show feasibility, not statistical stability.
Concurrency was never tested. Every SofiaVoice run used concurrency = 1. Throughput (0.17 req/s) is a single-user figure.
Status codes overstate success. SofiaVoice test02 counted an empty reply as a success. Content-level assertions are needed.
Truncation is logged, not enforced. In Cortex v1, two agents returned exactly 1,024 tokens (cap hit). In v2, every isolated node ended with done_reason: "stop", but nothing would fail the run if one did not. done_reason should be promoted from a logged field to an asserted one.
The web search returned nothing. The Tech Scout's DuckDuckGo search returned 0 results (SEM_RESULTADOS), yet the memory it wrote was labeled origem: chroma+web. Provenance labels should reflect what actually contributed.
Memory hygiene. Probe documents created by tests (probe_test: true) still live in the decisoes collection and were retrieved by the copywriter in a real run. Model-written decisions are also re-read by later evaluations, which is a feedback loop that needs a curation step (TTL, human sign-off before a memory becomes "decision", or a separate collection).
The judge is the weak link under load. The Evaluator has the largest prompt, it timed out on 7B, and it ran on 3B. A shorter rubric, a summarized deliverable, or a dedicated evaluator budget is needed.
Local vs production numbers. SofiaVoice's 2.80 s (server-side, per stage) and ~5.7 s (client round trip, local) describe different systems, and both should be reported with their scope.

7. Next experiments

SofiaVoice: add content assertions (non-empty text, bytes-per-character floor) to the stress harness. Run concurrency 1/5/10 with n ≥ 30. Test shorter spoken answers as a latency lever.
Cortex-Flow v3: pass summarized state between nodes to cut prompt pre-fill. Give the Evaluator a compact rubric and its own timeout budget. Fail the node on done_reason != "stop". Separate test probes and machine-written memories from curated decisions. Repeat each route at least 10 times to get real latency distributions.
Commerce: move from a single exported item to a batched inbox with back-pressure, so Cortex-Flow at ~34 min per builder route is not flooded by a 17-item daily run.

Closing

The three systems were very different (a cloud voice assistant, a local multi-agent reasoner, and a deterministic data pipeline), but they converged on one shape:

deterministic where possible → model where judgment is needed → measured everywhere → human at the end.

That shape was not designed up front. It came out of reading the numbers. Measurement showed that TTS, not the LLM, was SofiaVoice's bottleneck. It showed that a 7B model on a CPU decodes at about 6 tokens per second whatever the prompt says, so routing had to be code. It showed that the pipeline's own critic would reject a weak ad. And the lab's philosophy, now stored as vectors in a database, made sure that the final call on any of it still belongs to a person.

Technology can expand our capability. It should not replace our responsibility.

Appendix: data sources

Claim	File
SofiaVoice v1/v2 per-stage latency, guardrails	`experiments/005-ai-system-improvement/001-sofiavoice/v2.0.0/test05-sofia-v2.0.0/metrics.json`
SofiaVoice stress runs	`…/test01…test04-sofia-v2.0.0/metrics_sofia_test_.json` and `Result_.md`
Cortex v1 run and agent telemetry	`experiments/006-cortex-flow/v1.0.0/metrics_*_20260918_180917.{json,md}`
Cortex v1 truncated outputs	`experiments/006-cortex-flow/v1.0.0/refining-report/Refinado_20260914_192107.md`
Cortex v2 setup, state schema	`experiments/006-cortex-flow/v2.0.0/00-metrics_setup_v2.json`
Vector memory, Seed DNA	`…/v2.0.0/01-metrics_chroma_v2.json`, `04-metrics_seed_dna_v2.json`
Per-node telemetry	`…/v2.0.0/02`–`10`, `12`, `13-metrics_*_v2.json`
End-to-end integration	`…/v2.0.0/11-metrics_integracao_v2.json`
Evaluator verdicts	`…/v2.0.0/drafts/avaliation/avaliacao_20261003_*.md`
Commerce test suites and daily runs	`experiments/007-commerce-pipeline/0[1-5]-metrics_.json`, `result/Result_.md`