Teaching SokoFlow to Remember: Building a Conversational Engine with FSMs, Redis, and WhatsApp Webhooks

SokoFlow Build Log — Month 3 of 4

---

Welcome back. If you're new here — I'm an IT student running a structured, project-based learning plan toward becoming a production-grade backend engineer. Last semester I built SimPesa, a local-first M-Pesa STK Push simulator. This semester, the theme is from controlled environments to the messy real world.

SokoFlow is my flagship project: a conversational ERP for small Kenyan shopkeepers that lets them track inventory and record sales entirely through WhatsApp chat — no app, no training, just natural language.

Month 1 was the business core, built with strict TDD. Month 2 was infrastructure — Docker, CI/CD, and a live staging deployment on Railway. If you haven't read those, I covered them in the previous build logs.

Month 3 was the part I'd been building toward since the beginning: the Conversation Engine.

At the end of the Month 1 blog, I left off with a question:

How do you turn messy human language into deterministic system actions without the whole thing becoming cursed spaghetti?

The answer I committed to: Finite State Machines (FSMs) + controlled intent parsing + conversational context.

And then you might be wondering — why not just throw an LLM at this? That answer is coming. Keep reading.

---

Goals for Month 3

The acceptance criteria going into this month were specific and measurable:

• Session survives a worker restart. An expired session resets to IDLE correctly.
• Full add-product flow completes via chat simulator in fewer than 5 messages.
• A sale is recorded correctly for an ambiguous product name (e.g., "milk" → "Milk 500ml").
• Simulator payload accepted. Invalid signature returns 401. Duplicate dropped silently.

---

Prerequisites: Two Mini-Projects Before the Real Work

Before starting any FSM implementation, there were two self-contained tools I had to build first. These weren't part of the weekly milestones — they were infrastructure the rest of the month depended on. Both were interesting enough that I'll dedicate a separate post to each. For now, here's the short version.

1. Chat Simulator

Relying on a real WhatsApp API during local development would have been painful. So I built a CLI tool that mimics a WhatsApp conversation directly from the terminal.

After exploring a few options, I settled on an interactive REPL with a local webhook server:

• The CLI step turns the simulator into a long-running process and spins up a lightweight background HTTP server using Python's built-in http.server.
• The worker step: when Celery finishes processing a state transition, it fires a standard HTTP POST request back to that background server.
• The loop closes: the running CLI thread captures the inbound payload and prints it to the terminal instantly.

The result is a single terminal screen that behaves exactly like an active WhatsApp conversation — without needing a phone, a verified business account, or any real API calls.

2. Intent Resolver

When a user's message hits the system, it's raw natural language. "sold milk", "check stock", "Hi" — the system needs to decide what the user wants before deciding what to do.

The Intent Resolver sits in front of the FSM and classifies incoming text into a discrete intent the system can act on. Think of it as a receptionist at a building entrance. The receptionist doesn't solve your problem — they just say "You're here for Finance" or "You're here for HR" and point you to the right floor.

The resolver maps something like "Sale" into Intent.RECORD_SALE. The FSM takes it from there.

Crucially, the Intent Resolver is only active when the FSM is in the IDLE state. Once a user is inside a flow like:

ADD_PRODUCT_NAME → ADD_PRODUCT_PRICE → ADD_PRODUCT_QTY

...the current state provides all the context needed. There's no ambiguity to resolve.

---

Week 9 — Redis Session Layer: FSM State Serialization, TTL Handling, and Dedup Keys

The Core Problem: HTTP Has No Memory

The key mental model to start with: HTTP is stateless by design. If a user sends a message to SokoFlow, the system processes it and responds. When the next message arrives seconds later, the system has no idea who this person is or what they were doing.

For a back-and-forth conversation to work, the system needs persistent memory between requests. That points to a database — but not just any database. It needs to be fast and lightweight. Everything pointed to Redis, an in-memory data store already in SokoFlow's stack as the Celery broker. No new dependencies, just a Redis client and a clear schema.

The Session Schema: Pydantic vs. Dataclasses

The first implementation decision was how to model the session structure. In Python, this immediately surfaces the Pydantic vs. dataclasses debate. They look similar on the surface — both package typed fields — but they serve fundamentally different purposes.

Dataclasses are for clean internal data structures. Pydantic is for enforcing schema integrity across untrusted boundaries.

Redis stores raw strings. Celery workers read raw messages off a queue. These are untrusted boundaries. Pydantic wins here for four concrete reasons:

Reason	Detail
Automatic type coercion	Converts raw string timestamps from Redis directly into datetime objects — no manual parsing
Nested deserialization	Rebuilds complex nested JSON into deep Python objects automatically
Runtime validation	Guarantees Celery workers never process corrupted or malformed session state — raises ValidationError before business logic touches bad data
Native JSON compatibility	Serializes complex Python types directly into JSON strings Redis can store

Handling Edge Cases from the Start

Before writing the first handler, I defined how the system behaves when things go sideways. These aren't afterthoughts — they're first-class design decisions.

User Interruptions. If a user sends cancel or menu from any non-IDLE state, the FSM transitions unconditionally to IDLE, clears the working context, and confirms cancellation. This override works from any state, no exceptions.

Session Expiry. When a session's TTL expires in Redis, the next message from that phone number finds nothing. The FSM initialises fresh from IDLE with: "Your previous session timed out. Let's start fresh." Partial operations are never committed to PostgreSQL until the CONFIRM state is explicitly reached.

Invalid Inputs. Each state defines its own acceptance criteria. If a user enters a non-numeric value when a price is expected, the FSM holds the current state and re-prompts with a helpful message, incrementing an error counter. After 3 consecutive invalid inputs in the same state, the FSM transitions to IDLE with an apology — preventing infinite loops.

Duplicate Messages. WhatsApp can deliver the same message multiple times under poor network conditions. Each incoming message is checked against a Redis set of recently processed message IDs (TTL: 60 seconds). If the message_id already exists in the set, the request is acknowledged with 200 OK and silently dropped:

redis.set(f"dedup:{message_id}", "1", nx=True, ex=ttl)

The NX flag makes this atomic — the key is set only if it doesn't already exist, which is exactly the check we need.

The ConversationStore: Abstracting Redis Away

The ConversationStore is the Data Access Object (DAO) for session management. Its job is to completely hide Redis implementation details from the rest of the application. The FSM engine and Celery tasks don't care how sessions are saved — they just call the store.

Method	Responsibility
get_session(phone)	Fetches raw string from Redis, parses JSON into a Pydantic model, returns None on cache miss
save_session(...)	Updates session state using atomic compare-and-swap to prevent race conditions
delete_session(phone)	Removes the key explicitly when a user completes a flow or cancels

The save_session method is where things get interesting — and where the most important architectural decision of the week lives.

The Race Condition Problem: Why save_session Needed Special Treatment

Anywhere a system reads state, modifies it in memory, and writes it back is dangerous. The pattern looks innocent:

Read state → modify in memory → write state back

But the state you read may no longer be the state that exists when you write. The classic failure mode is a lost update: two workers read the same state simultaneously, both modify it independently, and the second write overwrites the first — producing a state that was never a valid transition.

Imagine two requests arriving near-simultaneously from the same phone number:

Worker A reads state: IDLE
Worker B reads state: IDLE
Worker A transitions to ADD_PRODUCT_NAME in memory
Worker B transitions to ADD_PRODUCT_NAME in memory
Worker A writes ADD_PRODUCT_NAME → correct
Worker B writes ADD_PRODUCT_NAME → overwrites A's work

According to the FSM, the next valid state was ADD_PRODUCT_PRICE, not a second ADD_PRODUCT_NAME. The conversation is now corrupt.

The Solution: Redis Lua Scripting (Optimistic CAS)

I had two genuine options:

Option 1 — Optimistic CAS via Lua Script: At write time, atomically verify that the state in Redis still matches what the worker expected. If another worker has already advanced the state, the write fails and the error is handled explicitly.

Option 2 — Pessimistic Distributed Locking: Acquire a Redis lock on the user's phone number at the beginning of task processing, preventing any concurrent access.

I chose Option 1. Here's why Option 2 was the wrong pick for this architecture:

SokoFlow is built on the principle of a dumb webhook receiver and a smart async worker pool (Principle 4 in the Developer Manifesto: "Failure Is a First-Class Input"). Pessimistic locking introduces a hard coordination problem between the lock TTL, the Celery task timeout, and the 30-minute Redis session TTL. If a worker crashes while holding a lock, that user's entire chat session is frozen until the lock expires — a terrible experience for an SME user who's in the middle of recording a sale.

Optimistic CAS aligns with the architecture naturally. Concurrent messages from the same phone number are rare (the deduplication layer already handles retransmissions). When a CAS failure does occur, the recovery path is explicit: fetch the new state, log the conflict with a correlation_id, and re-prompt the user — no blocked threads, no frozen sessions.

The implementation uses a Lua script registered at startup that executes atomically inside Redis:

save_session(phone, expected_old_state, new_payload, ttl=1800) → bool

The Lua script receives three arguments: the expected old state, the new serialized session JSON, and the TTL. It compares the current state in Redis against the expected state before writing. The return codes map directly to typed domain exceptions:

• 1 → Success
• 0 → StateMismatchError (stale state — another worker advanced it first)
• -1 → CorruptedSessionError (invalid JSON in Redis)

---

Week 10 — FSM Core: The ADD_PRODUCT Flow End-to-End

With the session layer solid, Week 10 was about wiring up the actual state machine.

The Architectural Strategy: State Pattern over if-else Chains

The naive approach to routing a message to its handler looks like this:

if state == IDLE:
    ...
elif state == ADD_PRODUCT_NAME:
    ...
elif state == ADD_PRODUCT_PRICE:
    ...

This works until the FSM has 20 states. Then process_message() becomes a 300-line method that knows everything. Adding, removing, or renaming a state means surgically editing one giant block.

The better approach is a dispatch table — a dictionary that maps each state directly to its handler function:

handlers = {
    IDLE: handle_idle,
    ADD_PRODUCT_NAME: handle_product_name,
    ADD_PRODUCT_PRICE: handle_product_price,
    ADD_PRODUCT_QTY: handle_product_qty,
    CONFIRM_ADD_PRODUCT: handle_confirm_add_product,
}

handler = handlers.get(state)
if handler:
    return handler(message)

The engine now answers exactly one question: which handler belongs to this state? Adding a new state means adding one line to the map and writing the handler independently. Each handler can evolve, be tested, and be reasoned about in complete isolation.

The dispatch table keeps the engine generic and the state-specific logic where it belongs: in the handler.

Every message turn through the engine follows a strict pipeline:

1. Pre-processing (Universal Guards): Check for global commands — cancel, menu, exit, stop.
2. State Evaluation: Look up the current SessionState in the dispatch table.
3. Input Validation: Validate the message according to that state's rules.
4. State Transition & Context Mutation: On valid input, update UserSession.context and advance UserSession.state.
5. Error Counter Management: On invalid input, increment context.error_count. At 3 consecutive failures, force reset to IDLE.
6. Atomic Persistence: Save the updated session back to Redis via ConversationStore.save_session().

The ADD_PRODUCT State Blueprint

Current State	Input Accepted	Target State	System Response
IDLE	Intent trigger ("add product", "add", "new product")	ADD_PRODUCT_NAME	"What is the product name? (e.g., Milk 500ml)"
ADD_PRODUCT_NAME	Non-empty string (2–100 chars)	ADD_PRODUCT_PRICE	Stores name. "What is the selling price in KES?"
ADD_PRODUCT_PRICE	Positive float/decimal (> 0)	ADD_PRODUCT_QTY	Stores price. "How many units in stock?"
ADD_PRODUCT_QTY	Positive integer (≥ 0)	CONFIRM_ADD	Stores qty. "Confirm add: {name} at KES {price}, {qty} units? (yes/no)"
CONFIRM_ADD	"yes" / "y"	IDLE	Commits product to Postgres, clears context, returns success
CONFIRM_ADD	"no" / "n"	ADD_PRODUCT_NAME	Clears context, restarts from name prompt
ANY NON-IDLE	"cancel" / "menu"	IDLE	Resets state, clears context, confirms cancellation
ANY NON-IDLE	3 consecutive invalid inputs	IDLE	Resets state, apologises, sends guidance message

Input Validators: Handling the Messy Reality of Kenyan SME Text

Before wiring up the engine, I wrote pure, stateless helper functions to clean up the real-world text a shopkeeper would type:

• parse_product_name(raw_text) — trims whitespace, validates 2–100 character length.
• parse_price(raw_text) — handles Kenyan currency formats like "KES 150", "150/=", and plain "150". Parses to Decimal and validates greater than zero.
• parse_quantity(raw_text) — parses positive integers, validates ≥ 0.
• parse_confirmation(raw_text) — accepts positive triggers ("yes", "y", "ndio", "1") and negative triggers ("no", "n", "zii", "2").

These are the entry points to every state transition. They raise typed InvalidInputError with user-facing messages, so the FSM engine never has to write error messages itself — the validators do it.

A 5-Message Execution Flow, End-to-End

User:  "add product"
FSM:   IDLE → ADD_PRODUCT_NAME       | "Great, let's add a product. What is the product name?"

User:  "Uji Flour 2kg"
FSM:   ADD_PRODUCT_NAME → ADD_PRODUCT_PRICE  | "Nice. What is the price in KES?"

User:  "150/="
FSM:   ADD_PRODUCT_PRICE → ADD_PRODUCT_QTY  | "Got it. How many units are in stock?"

User:  "20"
FSM:   ADD_PRODUCT_QTY → CONFIRM_ADD | "Confirm: Uji Flour 2kg at KES 150.00, 20 units. Reply yes or no."

User:  "yes"
FSM:   CONFIRM_ADD → IDLE            | "Product added: Uji Flour 2kg at KES 150.00, opening qty 20."

The Biggest Lesson of Week 10: Process Separation and Database Sessions

After completing the flow, I ran into a design question that tripped me up. When a user confirms a product, the data needs to move from Redis into PostgreSQL permanently. But here's the problem: Celery workers and the FastAPI web app are completely separate operating system processes — often separate Docker containers in production. They cannot share memory, engines, or connection pools.

In the FastAPI app, database sessions are managed through dependency injection — Depends(get_db) yields a request-scoped session tied to the HTTP request lifecycle. Celery tasks are triggered by messages off the Redis queue, not HTTP requests. You can't use Depends(get_db) inside a worker.

The solution was a dedicated async context manager for background tasks:

async_session_factory = async_sessionmaker(
    bind=engine,
    expire_on_commit=False,
    class_=AsyncSession
)

@asynccontextmanager
async def get_worker_db() -> AsyncGenerator[AsyncSession, None]:
    async with async_session_factory() as session:
        try:
            yield session
            await session.commit()
        except Exception:
            await session.rollback()
            raise

Inside any Celery task, wrapping execution in async with get_worker_db() as db: gives a task-scoped transactional session. This wins on three fronts:

• No idle DB locks: State transitions that don't persist (like NAME → PRICE) open and close the session instantly.
• Atomic commits: The context manager automatically commits on clean exit, making product creation atomic at the CONFIRM_ADD step.
• Automatic rollback: If PostgreSQL raises a constraint error or connection error mid-task, the rollback is guaranteed — no corrupted state.

---

Week 11 — RECORD_SALE and CHECK_STOCK Flows: Fuzzy Product Name Matching

With ADD_PRODUCT working end-to-end, Week 11 expanded the FSM to cover the two flows shopkeepers would use daily: recording a sale and checking stock.

The wrinkle that made this interesting: users don't type exact product names. They type "milk", "mllk", "Fresh Milk". The system needs to figure out what they actually meant.

What is Fuzzy Matching?

Fuzzy matching is a technique for finding strings that are approximately equal rather than exactly equal. A standard database index treats a one-letter typo as a completely different word. Fuzzy matching provides a mathematical way to say: "These two strings aren't identical, but they're a 90% match."

It's the same technology behind Google's "Did you mean...?" feature and autocorrect.

The Core Metric: Levenshtein Distance

The most common distance measure is Levenshtein Distance — the minimum number of single-character operations (insertion, deletion, substitution) needed to transform one word into another.

• cat → cats: distance 1 (one insertion)
• bread → bred: distance 1 (one deletion)
• bred → brad: distance 1 (one substitution)

Lower distance = closer match.

The Performance Problem

Calculating Levenshtein distance against a full product table is expensive. On a table with 1,000 products, a naive query calculates the distance for every single row. A standard B-Tree index can't help here — it only understands exact alphabetical ordering. At scale, this would spike the database CPU immediately.

The Solution: PostgreSQL pg_trgm and GIN Indexes

Rather than introducing a separate fuzzy search library, I used what was already in the stack: PostgreSQL's pg_trgm extension.

Instead of full-string comparison, pg_trgm breaks strings into overlapping trigrams — groups of three consecutive characters:

• "bread" → b, br, bre, rea, ead, ad
• "bred" → b, br, bre, red, ed

The database measures similarity by counting shared trigrams. A GIN (Generalized Inverted Index) indexes these chunks, so PostgreSQL can skip 99% of the table and surface the closest matches in milliseconds.

Why not a dedicated search library or service? Because the product catalogue for a small shop is not a Google-scale problem. Everything already lives in PostgreSQL. Adding pg_trgm means zero new infrastructure, and the queries are fast enough for the domain.

Confidence-Based Routing

When a user types a product name, the system returns similarity-scored matches and routes them based on confidence:

Score	Action
≥ 0.8	Auto-select product, proceed
0.3 – 0.8	Disambiguation: list up to 3 options — "Did you mean: 1. Fresh Milk 500ml 2. Mala Milk 1L? Reply 1 or 2"
< 0.3	Re-prompt user, no match found

This required one addition I hadn't planned for in the original FSM spec: a new RECORD_SALE_PRODUCT_SELECTION state. When multiple candidates are returned, the FSM needs to pause and wait for the user's selection — it can't just advance to quantity collection. The candidates are stored in the session context, and the user's numeric response ("1", "2") is mapped to the correct product.

The RECORD_SALE Flow

IDLE → RECORD_SALE_PRODUCT → [RECORD_SALE_PRODUCT_SELECTION] → RECORD_SALE_QTY → CONFIRM_SALE → IDLE

Key business rules built into the flow:

Stock validation guard. Before transitioning to CONFIRM_SALE, the system verifies requested_qty <= available_stock. If insufficient stock exists, the FSM holds at RECORD_SALE_QTY and warns the user — it never lets the sale proceed into confirmation with stock it doesn't have.

Atomic transaction execution. On "yes" at CONFIRM_SALE:

• Insert into the sales table.
• Atomically decrement inventory.quantity.
• Check if remaining stock hits the low-stock threshold and queue a notification task if so.

The CHECK_STOCK Flow

A lightweight 1–2 turn flow:

IDLE → CHECK_STOCK_QUERY → IDLE

Fuzzy lookup → return current quantity and unit price → reset to IDLE. Simple, but uses the same trigram matching infrastructure as the sale flow.

---

Week 12 — WhatsApp Webhook: HMAC Verification, Payload Schema, and Deduplication

The final week of Month 3 was about building the front door — the point where the outside world starts talking to SokoFlow.

Until now, messages entered through the internal chat simulator. Week 12 replaced the simulator with a production-shaped webhook that processes requests the way the real WhatsApp Business Platform would send them.

The core architectural principle for this week, pulled directly from the Developer Manifesto:

"The Webhook Endpoint Is Dumb. The Worker Is Smart."

The endpoint validates, deduplicates, enqueues, and responds. It does not touch the database. It does not make business decisions. That's the entire architectural thesis of Week 12.

Understanding HMAC: Authentication and Integrity

A message arriving at the server doesn't prove it came from a trusted sender. A webhook endpoint is a publicly reachable URL — anyone who knows it can send requests.

HMAC (Hash-based Message Authentication Code) solves this with a shared secret known only to the sender and receiver:

• The sender combines the raw message body and the secret to produce a signature, which travels with the request.
• The receiver independently computes the expected signature from the same raw body and the same secret, then compares the two.

If the signatures match, two things are proven:

1. Authentication — the request was created by someone who possesses the shared secret.
2. Integrity — the message body was not modified after it was signed.

HMAC does not prove freshness. Replay protection requires additional mechanisms — timestamps and event IDs — layered on top.

The mental model: HMAC is a mathematical secret handshake between two systems. The secret stays private. Only the resulting signature travels over the wire.

Why the Raw Bytes Matter

One implementation detail with real consequences: signature verification must happen against the raw request body bytes — before any JSON parsing.

Parsing the JSON and then re-serializing it can silently alter whitespace, key ordering, or escape sequences. The re-serialized body produces a different HMAC digest even though the data appears identical. Signature mismatch → 401 → dropped message.

The verification step uses hmac.compare_digest() rather than a plain string comparison. This provides constant-time comparison, which eliminates timing-based side-channel attacks.

No valid signature → no entry. No exceptions.

Understanding the WhatsApp Webhook Payload

The WhatsApp Business Platform sends a specific JSON envelope for inbound messages. The key components SokoFlow cares about:

• messaging_product: Always "whatsapp".
• metadata: Business identification — display_phone_number and phone_number_id.
• contacts: Sender profile — profile.name and wa_id (the sender's phone number).
• messages: The actual inbound message content — id, type, text.body.
• statuses: Delivery receipt tracking (sent, delivered, read, failed) — ignored by SokoFlow's webhook since it only processes inbound messages.

The Pydantic contract models only what SokoFlow actually needs — not the entire WhatsApp universe. The goal was clean: feed the simulator payload through Pydantic and reliably extract phone, message_id, message_text, and message_type without the webhook knowing anything about FSM logic.

Redis Deduplication at the Webhook Layer

External networks are unreliable. The same webhook can arrive twice under normal operating conditions — retransmissions, network jitter, WhatsApp's own retry logic. The system must not record a sale twice because the same event showed up twice.

The same is_duplicate mechanism from Week 9 handles this at the webhook layer:

# SET key "1" only if it does NOT already exist (NX), with a 60-second TTL (EX)
redis.set(f"dedup:{message_id}", "1", nx=True, ex=ttl)

Duplicate message_id → return 200 OK with status: ignored — no FSM processing, no Celery task dispatch.

The 200 OK matters: returning an error code would signal to WhatsApp that the delivery failed, causing it to retry — which is exactly the cycle we're trying to prevent.

The Complete Webhook Pipeline

By the end of Week 12, a valid inbound message travels through this sequence:

1. FastAPI receives the HTTP POST.
2. Pydantic validates the payload shape. Malformed → 422.
3. HMAC-SHA256 signature is verified against the raw body. Invalid → 401.
4. Redis checks the message_id for duplicates. Duplicate → 200 OK, dropped.
5. The message is enqueued into Celery's conversation_tasks queue.
6. FastAPI returns 200 OK immediately.
7. The Celery worker picks up the task, fetches or initialises the Redis session, runs the FSM, persists to PostgreSQL if needed, and sends the response back through the chat simulator loop.

---

Progress Since Month 1

Month 1	Month 2	Month 3
Core business logic	Automated workflows	Conversational engine
Local development	Containerized development	Stateful conversations
Manual test runs	CI pipeline	Redis-backed sessions
Single synchronous process	Background workers via Celery	Finite state machine
Runs on my machine	Runs in the cloud	Simulated WhatsApp integration

---

Final Thoughts and What's Next

Month 3 is done. That's 12 weeks of building and shipping SokoFlow, and for the first time it actually feels like a conversational system rather than just a backend with ambitions.

The interesting engineering challenge this month wasn't the code — it was the design. How do you impose deterministic structure on something as unpredictable as a human conversation? The answer turned out to be: you don't fight the unpredictability. You contain it. State machines give each message exactly one valid set of responses depending on where the conversation is. Redis gives the conversation memory. Lua scripting makes that memory safe under concurrency. Pydantic keeps every input honest before it touches business logic.

And the reason I didn't use an LLM? That's a full answer worth its own section, and it's coming in the Month 4 wrap-up. The short version: an LLM would have been easier to build, but harder to operate, debug, and trust for a system that moves real money and inventory for real shop owners.

Month 4 moves into async report generation, chaos testing, Swahili language support, and final integration.

The system can now have a conversation. Next month, I want to see how well it holds up when things go wrong.

Stay locked in.