Shipping the Capstone: PDF Reports, Chaos Testing, and the Architecture of a System That Fails Gracefully

SokoFlow Build Log , Month 4 of 4

---

Sixteen weeks ago, I opened an empty directory, ran git init, and started building something I wasn't entirely sure I could finish.

This is the final build log for SokoFlow a conversational ERP for small Kenyan shopkeepers that lets them manage inventory and record sales entirely through WhatsApp chat. No app, no training, just natural language.

If you've been following along: Month 1 was the business core, built with TDD. Month 2 was infrastructure: Docker, CI/CD, and a live staging deployment. Month 3 was the conversational engine: FSMs, Redis sessions, fuzzy matching, and a secured webhook.

Month 4 was the capstone: async report delivery, a chaos testing suite, internationalization, and final integration. Everything built over the previous 12 weeks came down to these four weeks.

---

Goals for Month 4

• Report PDF generated within 5 seconds and delivered as a document message via the mock sender.
• The system handles all chaos scenarios without crashing or corrupting state.
• A shop registered with locale='sw' receives Swahili responses.
• Repository clone-to-running in under 5 minutes following the README.

---

Week 13 Async Report Delivery: ReportLab, PDF Bytes, and the report_tasks Queue

Two Worlds, One Translator

In Month 1, Week 4, I built the sales recording, aggregation, and report data generation. Month 4 built the final layer on top: turning that data into something a human can actually use.

Imagine a shopkeeper asking SokoFlow for an end-of-day summary. The system can calculate this:

Today's Sales:     KSh 45,000
Number of Sales:   37
Top Product:       Blue T-Shirt
Profit:            KSh 12,000

But data is not documentation. The shopkeeper might want to download it, print it, email it to their accountant, or archive it for the month. A Python dictionary serves none of those needs.

This is where a PDF comes in; and the PDF service's job is conceptually simple: translate between the computer's world and the human's world.

Computer World	Human World
{"sales": 45000, "orders": 37, "profit": 12000}	SOKOFLOW DAILY REPORT , Sales: KSh 45,000, Orders: 37, Profit: KSh 12,000

ReportLab

For PDF generation, the options range from manual word processors to print-to-PDF workarounds; both of which have no place in an automated pipeline. ReportLab is a Python library built precisely for this: generating complex, data-driven documents programmatically from any data source.

Since SokoFlow is already a Python system, the integration is natural. ReportLab lets us express document structure directly in code , no external processes, no format conversions, no third-party API round trips.

Where Does the PDF Live? A Three-Way Architecture Decision

Once ReportLab builds the PDF, there's an immediate question: where does the resulting data live while it waits to be sent to WhatsApp?

This is a real infrastructure decision. I evaluated three options:

Option A , In-memory (BytesIO)

The PDF exists purely as bytes in the Python process's memory. No files written to disk, no cleanup required.

Advantages: Simplest implementation. No Disk I/O. No orphaned temporary files.

Drawbacks: Memory is finite and volatile. Under concurrent load, large PDFs could create memory pressure. A worker crash before delivery means the PDF is gone permanently.

Option B , Temporary file on the server

Write a /tmp/daily-summary-[job-id].pdf file, send it, then delete it.

Advantages: Doesn't consume application memory for stored documents. More manageable for very large files.

Drawbacks: Disk I/O costs. Temporary files accumulate if a crash prevents cleanup. Multiple application instances don't share the same filesystem.

Option C , Object storage (e.g., S3)

Generate the PDF, upload it to a storage bucket, retrieve the URL for delivery.

Advantages: Durable, scalable, shared across all application instances. Enables report history and audit trails.

Drawbacks: Network round-trip latency. Storage costs. Additional infrastructure and credential management.

The choice: Option A , in-memory delivery for the MVP.

The spec's requirement is to generate a shop's daily summary (typically under 200KB) and immediately deliver it. That's a precise, time-bounded operation. Introducing S3 at this stage would add network latency, infrastructure cost, and credential complexity for a problem that in-memory delivery already solves cleanly. If report sizes grow or delivery patterns change, moving to Option C is a clear migration path , the PDFService abstraction makes that swap a localized change, not a systemic one.

Database → Aggregate data → ReportLab → BytesIO → WhatsApp API

Extending the MessageSender

Until Week 13, the MessageSender protocol only knew how to send text. Adding document delivery meant extending the protocol with a send_document method:

send_document(
    recipient,
    document_bytes,
    filename,
    caption
)

The sender constructs an HTTP multipart request, posts the raw PDF bytes to the WhatsApp API, receives a media ID in return, and attaches that ID to an outbound message. WhatsApp's infrastructure handles the rest.

The Async Pipeline: Why Report Generation Is Architecturally Different

Every other flow built in Month 3 , ADD_PRODUCT, RECORD_SALE, CHECK_STOCK , followed the same pattern: message arrives, FSM transitions through states across multiple turns, business logic executes at CONFIRM, session resets to IDLE.

The report flow broke that pattern in an important way: it's genuinely one-turn, but the task is heavy.

Sales aggregation, PDF generation, and document delivery are not operations that belong inside a synchronous Celery task serving the conversation_tasks queue. That queue is optimized for fast, lightweight FSM transitions. Tying it up with a potentially long-running report job would degrade response times for every other active conversation.

My first instinct was to handle this like any other flow , hook it into the handler map and route by state. But that doesn't work. A standard flow looks like:

Incoming message → FSM engine → dispatch by state → handler → transition → FSMResult

For the report, by the time the next message arrives, the job is already running in the background. There's no follow-up state to route to.

The design I landed on:

1. When the Intent Resolver detects Intent.GENERATE_REPORT in the IDLE handler, log the transient REPORT_PENDING state to session history for observability purposes.
2. Immediately delegate to GenerateReportFlow.handle_daily_report(...) , a dispatcher that does exactly one thing: enqueue the job into report_tasks and return.
3. Return an immediate acknowledgement to the user: "Your report is being generated and will arrive shortly."
4. The session resets to IDLE. The user can continue interacting.
5. The report_tasks worker picks up the job independently, generates the PDF, and delivers it asynchronously.

The REPORT_PENDING state is never used for routing. It exists purely as a breadcrumb in the session history , useful for debugging and observability, not for FSM logic.

The result: the conversation worker is never blocked, the user isn't waiting, and the report arrives when it's ready.

---

Week 14 , Chaos Testing: Proving That SokoFlow Fails Gracefully

"But I Already Have Tests. Why Do I Need a Chaos Runner?"

Going into Week 14, SokoFlow had over 100 tests spanning unit and integration suites. State transitions, invalid inputs, cancellation flows, normal paths , all covered. The question that genuinely confused me was: what gap is a chaos runner filling that those tests aren't?

The answer required stepping back and looking at what SokoFlow actually is.

It's a system where every message arrives through a stateless HTTP webhook, gets validated, deduplicated, pushed into Celery, processed by an FSM, persisted across Redis and PostgreSQL, and sometimes triggers asynchronous background work. The components are distributed. The state is shared across processes. The inputs come from an unreliable external network.

That changes the question from "does this piece of code behave correctly?" to something harder:

"Does SokoFlow remain correct when reality starts being annoying?"

The individual components can all be correct while the system as a whole can still behave badly when things happen in the wrong order, happen twice, fail halfway through, or disappear temporarily. That's the gap existing tests don't cover , and it's exactly the gap the chaos runner was designed for.

I implemented tools/chaos_runner.py as a dedicated script that runs against the actual local stack , not mocked functions, not isolated units. It injects specific failure scenarios and checks whether the system preserves its important invariants.

The Scenarios

DUPLICATE_MESSAGE

WhatsApp can deliver the same message more than once under normal network conditions. The chaos runner sends the exact same message_id twice and checks what doesn't happen afterward.

The duplicate must not create a second FSM transition, execute the business operation twice, or send another response. SokoFlow should acknowledge the second delivery and silently drop it. This tests the Redis deduplication layer end-to-end , not just the dedup function in isolation, but the entire path from webhook to worker to FSM to response gate.

DELAYED_DELIVERY

A message arrives after the user's session TTL has expired in Redis:

User:    "add product"
System:  "What's the product name?"
         ... session expires (Redis TTL) ...
User:    "Milk 500ml"

The dangerous failure mode here is the system resurrecting the old state and treating "Milk 500ml" as an ADD_PRODUCT_NAME response. Instead, an expired session should produce a clean IDLE initialisation: "Your previous session timed out. Let's start fresh."

This matters because Redis TTLs are silent. There's no expiry event, no callback, no notification , the key simply disappears. The system has to handle that disappearance gracefully on the next incoming message.

MALFORMED_JSON

Send intentionally broken JSON to the webhook. The important check isn't just that FastAPI returns 422 , it's that no Celery task is enqueued. Invalid input should die at the front door. Business logic should never see it.

This aligns directly with the architectural rule from Month 3: the webhook is responsible for validation and dispatch. The worker is responsible for business logic. Malformed input is the webhook's problem to reject, not the worker's problem to handle.

INVALID_HMAC

Take a valid request payload and tamper with the HMAC signature. The expected path is strict:

Tampered request → HMAC verification → 401 → no task → no FSM transition → no database work

A normal FSM unit test doesn't prove this. The FSM can be perfectly implemented while the security boundary is completely absent. The chaos runner validates both at once.

WORKER_CRASH

Inject an exception after the session has been written to Redis but before the outbound response is sent. When Celery retries the task, two invariants must hold: the FSM state must not roll backwards to its pre-transition position, and the retry must not execute the operation a second time.

This directly tests the CAS (compare-and-swap) architecture from Month 3's Week 9. The Lua script's expected_old_state check means a retry encountering an already-advanced state either continues safely or surfaces a StateMismatchError , never silently re-applies a transition.

CONCURRENT_MESSAGES

Submit five messages for the same phone number simultaneously. One phone number maps to one active FSM instance. If multiple Celery workers race to read and write the same session key without coordination, the conversation state becomes incoherent. The invariant being tested: exactly one valid state exists in Redis at the end of all five operations.

DATABASE_FAILURE

Pause PostgreSQL in the middle of a task. The worker should fail, retry with exponential backoff, and succeed once Postgres recovers , without leaving corrupted application state. Structured Celery retry configuration looks correct in code; the chaos runner verifies it works against the actual running database.

INVALID_FSM_LOOP

Send invalid input repeatedly while the FSM holds a state awaiting a specific input type , for example, feeding non-numeric strings to ADD_PRODUCT_PRICE. After three consecutive failures in the same state, the FSM should force-reset to IDLE and apologise. This scenario verifies that guard, end-to-end, with real messages through the full stack.

Without this, a shopkeeper who accidentally types the wrong thing three times in a row is trapped in an infinite re-prompt loop , one of the most frustrating experiences possible in a conversational interface.

REDIS_UNAVAILABILITY

Pause Redis while a conversation is active. Redis isn't just a cache in SokoFlow , it's session memory, Celery's message broker, and the deduplication store. Losing Redis means losing the FSM's ability to remember anything. The test verifies graceful degradation and correct recovery once Redis comes back online.

REPORT_TIMEOUT

Inject a timeout into document delivery during the async report path:

Report task → generate PDF → delivery fails → retry → idempotency check → safe continuation

The retry check isn't just "does it try again?" It's "does trying again produce a duplicate report?" Each mutating Celery task carries an idempotency key. The chaos runner validates both sides: that the retry fires correctly, and that idempotency prevents duplicate delivery.

The Results

❯ uv run -m tools.chaos_runner

SokoFlow Chaos Testing Suite v1.0
============================================================
Environment: local
Started: 2026-09-09T10:59:33.613254+00:00

Running scenario: DUPLICATE_MESSAGE (1/10)       PASSED (0.23s)
Running scenario: DELAYED_DELIVERY (2/10)        PASSED (3.03s)
Running scenario: MALFORMED_JSON (3/10)          PASSED (0.04s)
Running scenario: INVALID_HMAC (4/10)            PASSED (0.03s)
Running scenario: WORKER_CRASH (5/10)            PASSED (0.03s)
Running scenario: CONCURRENT_MESSAGES (6/10)     PASSED (1.12s)
Running scenario: DATABASE_FAILURE (7/10)        PASSED (8.34s)
Running scenario: INVALID_FSM_LOOP (8/10)        PASSED (0.67s)
Running scenario: REDIS_UNAVAILABILITY (9/10)    PASSED (12.44s)
Running scenario: REPORT_TIMEOUT (10/10)         PASSED (14.03s)

Summary: 10/10 passed | Duration: 40.0s | Coverage: 100.0%
Report saved: chaos_reports/chaos_2026-09-09T11-00-13.html

The Reframe That Changed Everything

My original question going into this week , "why do I need chaos testing when I already have unit tests?" , had the wrong premise. The better question was:

"What kinds of failures can my existing tests structurally not see?"

Unit tests answer: "does this piece of code behave correctly in isolation?"

Chaos tests answer: "does the system remain correct when a Celery task retries, Redis temporarily disappears, and another message arrives for the same phone number , all at the same time?"

Those are completely different questions. Both matter. Neither replaces the other.

---

Week 15 , Internationalization and Low-Stock Alerts

Internationalization (i18n)

The spec required that a shop registered with locale='sw' receives Swahili responses, while locale='en' receives English. Straightforward requirement. The interesting engineering challenge was figuring out where in the architecture this belongs.

Scoping the Problem First

Before touching any code, I had to decide: inbound Swahili parsing, or outbound Swahili responses only?

Inbound internationalization , understanding incoming Swahili messages like "uzia maziwa 3" , would require redesigning the entire Intent Resolver. That's a separate, significant project. It would also expand the intent-resolution problem rather than implement the i18n requirement.

The scope for Week 15 was outbound messages only: whatever the shop's locale, the FSM responses go out in the correct language. No changes to incoming message parsing.

Finding the Right Architectural Boundary

With scope defined, the next question: where does localization live?

The most obvious place was inside the FSM handlers themselves. Each handler already returns an FSMResult with a reply_text field. Why not just make that text language-aware?

The problem: hardcoding localized strings across 20+ FSM flows means that adding a new locale in the future requires editing every handler. More critically, it means the FSM is doing two jobs at once , determining state transitions and producing presentation text. Those are different responsibilities, and mixing them creates the coupling that makes systems brittle.

The right answer kept pointing to the same place: the edge. Specifically, the MessageSender , the one layer every outbound message passes through before it leaves the system.

This keeps the FSM entirely unaware of language. The FSM produces meaning. The localization layer translates meaning into the correct language. The MessageSender delivers it.

The FSMResult Refactor

This is where things got architecturally interesting. The existing FSMResult looked like:

FSMResult(
    previous_state=...,
    new_state=...,
    context=...,
    reply_text="Sale recorded successfully"   # ← English hardcoded in business logic
)

The problem: reply_text is a presentation decision baked into a domain object. The FSM is saying how to say something instead of what happened.

The refactored version:

FSMResult(
    previous_state=...,
    new_state=...,
    context=...,
    message_key=MessageKey.SALE_RECORDED,             # ← what happened
    message_params={                          # ← the data to render it with
        "product": "Milk 500ml",
        "quantity": 5,
        "remaining_stock": 12
    }
)

At the outbound boundary, message_key + message_params + shop.locale renders the final text:

en: "Sale of 5 Milk 500ml recorded. Remaining stock: 12 units."
sw: "Mauzo ya Milk 500ml 5 yamehifadhiwa. Stoke iliyobaki: vitengo 12."

This is the principle i18n exposed that was always latent in the design:

Application code should declare what happened. Not how to say it.

One key decision I was careful about: the localization logic lives in a dedicated localization.py service, not directly inside MessageSender. Transport responsibility and localization responsibility are different things, and conflating them would just move the coupling problem from one place to another.

What i18n Actually Taught Me

Features like internationalization are worth building not just for the user-facing value , they expose architectural boundaries that were previously invisible. The original FSMResult design wasn't wrong. It was correct and efficient for one language and fast iteration. But it had a hidden assumption: that presentation and domain logic were the same thing. i18n made that assumption impossible to ignore.

Low-Stock Alerts

The spec promised shop owners an alert when a product crosses its defined low-stock threshold. The interesting design question was deceptively simple: what exactly is the event that triggers the alert?

The naive approach , send an alert after every sale if stock is currently below threshold , sends multiple alerts for a product that stays low across several consecutive sales. If a threshold is 10 and stock goes 12 → 8 → 6 → 4, the naive approach fires three times. That's spam.

The correct approach is threshold crossing detection , the alert fires only when a sale transitions a product from above to below the threshold. Not when it's already below and goes lower.

old_is_low = inventory.quantity <= inventory.low_stock_threshold

# Deduct inventory for the sale

new_is_low = inventory.quantity <= inventory.low_stock_threshold

low_stock_triggered = (not old_is_low) and new_is_low

The logic:

Stock Before	Stock After	Result
Above threshold	Above threshold	❌ No alert
Above threshold	Below threshold	✅ Alert fires
Below threshold	Below threshold	❌ No alert (already alerted)
Below threshold	Above threshold	❌ No alert

Only the False → True transition fires the alert. It's detecting a state change event, not a condition being true. The alert enqueues as a Celery task into report_tasks, which handles delivery through the same localized MessageSender pipeline built for i18n.

---

Week 16 , Final Integration: GUI Simulator, Demo, and Shipping

The final week was deliberately narrow. The project was functionally complete. Week 16 was about integration, documentation, and one addition I hadn't anticipated: a visual demo interface.

The GUI Chat Simulator

For 15 weeks, I'd been testing conversations through a CLI REPL , a terminal process that mimicked a WhatsApp session. It worked perfectly for development. For a demo, it looked like a developer tool, not a product.

The decision: build a lightweight web-based chat simulator that looks like a WhatsApp conversation, wired to the actual SokoFlow backend. The constraints were strict , final week means shipping, not feature-adding. No new backend endpoints, no new dependencies.

The stack: pure HTML, CSS, and vanilla JavaScript. Python's built-in http.server to serve static files and handle API requests. No FastAPI changes.

How a message flows from the browser to SokoFlow and back:

Inbound (user → SokoFlow):

1. User types a message in the browser UI.
2. UI posts { phone, message } to the simulator's POST /api/send endpoint.
3. The simulator constructs a valid WhatsAppWebhook payload, signs it with HMAC-SHA256 (X-Hub-Signature-256), and forwards it to http://localhost:8000/webhook/whatsapp.
4. FastAPI validates the payload, verifies the signature, deduplicates via Redis, and dispatches to the appropriate Celery queue.

Outbound (SokoFlow → browser):

1. When the Celery worker completes processing, MockMessageSender.send_text() or send_document() POSTs the response to the simulator's receiver endpoint.
2. The simulator appends the message to an in-memory store keyed by phone number.
3. The browser polls GET /api/messages?phone=... every second and renders new messages as chat bubbles.

The backend never knew the difference. Every message went through the real webhook, the real HMAC verification, the real FSM, the real database. The simulator was a cosmetic wrapper around the same production-shaped infrastructure that Month 2 built.

The Demo

I recorded an 11-minute walkthrough using Loom covering the full system , adding products, recording sales, generating a PDF report, and demonstrating FSM edge cases including invalid input recovery and session cancellation. You can watch it here.

Docker and README

The Docker work was largely done in Month 2 , multi-stage builds, an optimised image under 200MB, and a docker-compose.yml orchestrating FastAPI, PostgreSQL 15, Redis 7, and the Celery worker pools. Week 16 ensured everything was documented clearly enough that a fresh clone reached a running system in under 5 minutes.

---

The Complete Journey: All Four Months

	Month 1	Month 2	Month 3	Month 4
Focus	Business core	Infrastructure	Conversational engine	Integration + polish
Environment	Local development	Containerized + cloud	Stateful conversations	Fully integrated
Testing	TDD, 40+ unit tests	CI pipeline, 85% gate	FSM unit + integration	100+ tests + chaos suite
Key technology	FastAPI, PostgreSQL, Alembic	Docker, GitHub Actions, Railway	Redis FSM, Lua CAS, pg_trgm, HMAC	ReportLab, i18n, chaos runner
State of system	Runs locally	Runs in the cloud	Holds a conversation	Deployed, tested, documented

---

Final Thoughts: What 16 Weeks Actually Taught Me

The Answer to the LLM Question

In Month 1, I said SokoFlow wouldn't use an LLM for intent parsing. In Month 3, I said the explanation would come in Month 4.

Here it is.

An LLM would have resolved "uzia maziwa 3" into RECORD_SALE, product=milk, qty=3 in milliseconds. It would have handled typos, ambiguity, and Swahili code-switching out of the box. It would have been faster to build.

The reason I didn't: SokoFlow deals with real inventory and real sales for real shop owners. When a system records that 5 units of milk were sold, that number needs to be reliable. An LLM's output is probabilistic , it produces the most likely answer, not a guaranteed one. Debugging a hallucination in production at a shop owner's expense is not an acceptable failure mode.

The FSM approach gives me something the LLM approach cannot: every decision is traceable. If a sale is recorded incorrectly, I can read the state transition log and find exactly where the interpretation went wrong. There's no probability distribution to interrogate, no temperature to tune. The system is deterministic by design, and for a system that moves inventory and revenue, that's not a constraint , it's a feature.

That said, "no LLMs" was a design choice for SokoFlow's MVP, not a position. The right tool depends on the problem. Which is exactly why Month 5 will add one carefully scoped AI-powered feature , and why that distinction matters.

The Bigger Lesson

Before this project, I knew the names of these tools. FastAPI, Celery, Redis, Docker, PostgreSQL , I could have listed them in a sentence. What I didn't have was any intuition for why a system is designed the way it is. Why the webhook is dumb and the worker is smart. Why naive datetimes are a silent landmine. Why localization belongs at the edge and not inside business logic. Why chaos testing asks different questions than unit testing.

Those aren't things you learn from documentation. They show up when you build something real and the wrong design creates a problem you have to solve.

What's Next

SokoFlow is complete as an MVP. But it doesn't end here.

For the next month, I'll be adding one AI-powered feature on top of the existing FSM foundation: natural-language shop analytics. A shopkeeper will be able to ask free-form questions like "which products are running low?" or "how does this week compare to last week?" , and an LLM, using function calling against SokoFlow's existing internal APIs, will resolve the question into a real query and a grounded answer.

No new data store. No separate product. The AI layer is additive , it works with the deterministic FSM, not instead of it.

That's the right way to introduce AI into a system: find a specific problem it solves better than rules-based logic, add it at the edge, and keep the core predictable.

Keep following for when that episode drops.

---

SokoFlow is open source. The full codebase, architecture diagrams, and documentation are available on GitHub.