DEV Community: JOOJO DONTOH

How My personal Agent Alfred talks to my vacuum

JOOJO DONTOH — Mon, 23 Mar 2026 08:48:20 +0000

The Idea Came First

Hi guys, I'm here again. After building Alfred here I wanted him to be able to control Juliana, my Xiaomi X20+ robot vacuum. I did not know how that was going to work and I did not have a clear path forward, but the goal was clear tbh. Ask Alfred something and then he can act on the available tools to him. So I started where most network curiosity starts. I ran an nmap scan on my LAN to see what Juliana was actually exposing to the network. All TCP ports were closed. But UDP port 54321 was open and listening. Bingo!.

If you have not read about Alfred yet, I wrote about how I built him here. This feature is a direct result of what I called the Floodgate Effect in that article. The moment Alfred works in one area of your life, you immediately want to connect everything else. Juliana was next on the list. Its weird that I have names for things in my house but yh that's me.

Speaking Juliana's Language

Having a port is not the same as having a conversation unfortunately. I needed to understand what protocol was running on that port and how to speak it. After some research I discovered that Xiaomi smart home devices communicate using MiIO, a proprietary protocol that runs over UDP port 54321. That was the first real breakthrough.

MiIO follows a three step flow.

First you do a handshake. You send a 32 byte "hello" packet made entirely of 0xFF bytes and the device responds with its device ID and a timestamp called a stamp.
Second you send a command. Every command is a 32 byte header combined with an AES-128-CBC encrypted JSON body. The encryption uses an MD5 derived key and IV generated from the device token. The header carries magic bytes, the packet length, the device ID, the stamp incremented by one, and an MD5 checksum.
Third you receive a response in the exact same format and you decrypt it using the same token to get your JSON back.

The JSON itself follows a JSON-RPC style structure. A request looks like this:

{ "id": 1, "method": "get_properties", "params": [...] }

And a response comes back like this:

{ "id": 1, "result": [...] }

It is a clean protocol once you understand it, but getting to that understanding took some work.

The Roadblocks

Getting the Token

Understanding the protocol was one thing. Actually talking to Juliana required a device token, This is the way Xiaomi handles auth in communication with the device, and getting that token turned out to be its own challenge. My first instinct was to extract it programmatically through the Xiaomi cloud, but that path was immediately blocked by captcha. I had to find another way.

I ended up using the Xiaomi Cloud Tokens Extractor tool, which supports a QR code based login flow. That got me what I needed. The token, the device ID, and confirmation that Juliana was registered under the MiIO protocol. With those two values in hand, I could finally start sending real commands.

The MiOT Property System

Modern Xiaomi devices do not use simple named commands. They use MiOT, the Mi IoT specification, where every property and action is addressed by a service ID called siid and either a property ID called piid or an action ID called aiid. Reading the battery level means sending a get_properties request with siid 3 and piid 1. Setting the suction to strong means sending a set_properties request with siid 4, piid 4, and a value of 2. Starting a cleaning session means triggering an action with siid 2 and aiid 1. Every single thing the vacuum does maps to one of these numeric combinations.

Trial and Error All the Way Down

There is no official documentation that maps these IDs for the X20+. I had to probe them manually by iterating through siid values from 1 to 30 and piid values from 1 to 30 and cross referencing what came back against what I could see in the Xiaomi app. It was tedious work. Some of the latest firmware implementations returned values that did not line up with what you would logically expect, which made matching them to real behaviour even harder.

Cleaning History Lives in the Cloud

One limitation I hit fairly early was around maps and cleaning history. Both are stored in the Xiaomi cloud rather than on the device itself. The only thing you can reliably read from Juliana directly is the last cleaning session. I turned this into an advantage by tracking every session result myself and building a local history. That way Alfred always has full context about when Juliana last cleaned, how long it took, and how much area was covered, without depending on the cloud for any of it.

The Real Value

It Is Not About the Commands

Sending commands to Juliana via an API is not the interesting part. The Xiaomi app already does all of that. You can start a clean, dock the vacuum, set suction levels, and check the battery from your phone in seconds. Replicating that through code alone would not be worth writing about.

The interesting part is what happens when those commands become tools that an AI agent can reason about and invoke on your behalf.

Tools Alfred Can Use

Every capability I built around Juliana was wrapped into a tool that Alfred can call. There are 3 of them:

executeVacuumStatus reads the current state of the device including battery level, cleaning mode, error codes, and consumable wear levels.
executeVacuumCommand sends operational commands like start, stop, pause, resume, dock, and locate.
executeVacuumHistory pulls from the locally tracked session log so Alfred can reason about when and where Juliana has cleaned.

What This Actually Looks Like

With those tools in place, my conversations with Alfred around the vacuum feel completely natural. I can ask things like:

Status and maintenance

"What is Juliana's battery level?"
"Does Juliana need any maintenance?"
"How are Juliana's consumables holding up?"

Control

"Start cleaning the living room"
"Clean the master bedroom and the office"
"Send Juliana home"
"Find Juliana" (this makes her announce her location out loud)

History

"When did Juliana last clean?"
"How much has Juliana cleaned today?"
"Show me Juliana's cleaning history"

Combined and conversational

"How is everything at home?"
"Start cleaning the guest bedroom and let me know when it is done"
"Is Juliana's mop pad due for replacement?"

Alfred does not just relay commands. He reads the context, decides which tools to call, and responds with a full picture. That is the difference between a smart home app and an agent.

From a Single Open Port to a Talking Vacuum

What started as curiosity about controlling my robot vacuum and an open UDP port turned into a fully functional integration between Alfred and Juliana. Along the way there were real obstacles and each one had to be solved before the next step was even possible.

Getting the device token could not be automated due to captcha blocks on the Xiaomi cloud, so I used the Xiaomi Cloud Tokens Extractor with QR code login instead. With no official documentation for the property IDs on the X20+, I probed siid 1 through 30 and piid 1 through 30 manually and matched results against the Xiaomi app. UDP has no built in request response correlation so I built a command serialization queue that keeps exactly one command in flight at a time. Hello responses were mixing with command responses so I added an isHelloResponse() check to skip those 32 byte packets. Timeouts were killing subsequent commands so I reset the stamp to zero on timeout to force a fresh handshake. Sending too many properties at once caused failures so I batched them into groups of ten. The locate command was not triggering a beep until I found the right combination at siid 7, aiid 1, and piid 1 set to 1. Consumable values were coming back wrong because the correct properties were at siid 9, 10, 11, and 18 rather than siid 4 where I originally looked.

The one limitation I could not fully solve is map data. The X20+ stores its maps in the Xiaomi cloud and not on the device itself. Valetudo would solve this but that means a total OS flush of the robot which voids my warranty.

Every solution in this list is a direct result of building things the right way rather than the fast way. The command queue, the constants file, the local history store, all of it exists because the goal was never just to control a vacuum. The goal was to give Alfred enough context and capability to reason about the home the same way he reasons about a calendar or an inbox. Juliana is now part of that picture.

An Autonomous, Agentic, AI Assistant, Meet Alfred and this is how I built him.

JOOJO DONTOH — Mon, 16 Mar 2026 15:39:44 +0000

Introduction

My people it's me again. This time I have built something fun but mostly useful. I gave building an autonomous agent a chance and it's turning out well. I know it's a cliché but his name is Alfred. The thing is AI agents are no longer a novelty. It all started out as simple chatbots chaining a few prompts together. Now it has evolved into something far more capable. These systems that can "reason" (I know it's just a lot of math and not actual reasoning), plan, use tools, and execute multi-step workflows with minimal human intervention. Agentic flows, where an AI iteratively breaks down a goal, takes actions, evaluates results, and course-corrects, are quickly becoming the backbone of serious productivity tooling.

But the thing is not all models are created equal. The market is crowded. GPT-4o, Gemini, Mistral, Llama, DeepSeek all have their own strengths, trade-offs, and devoted user bases. Picking the right model for a given task has become something of an art form in itself. Especially because the benchmarks keep getting blurrier and blurrier.

For me, that choice keeps coming back to Anthropic's Claude and specifically to Opus. As an engineer, I spend a significant portion of my day thinking in systems: abstractions, edge cases, failure modes and architecture trade-offs. Opus is the only model that consistently feels like it's doing the same while cleverly grabbing my immediate system context. Where other models can produce code that technically compiles but misses the intent entirely, Opus tends to understand the why behind what I'm building, not just the what. That distinction, subtle as it sounds, makes an enormous practical difference when you're deep in a complex codebase. Opus has downsides, especially because sometimes it takes shortcuts without adhering to the principles you intended.

On the bright side what sealed it for me, though, was the CLI experience. Claude's command-line interface is genuinely pleasant to use: fast, composable, and unobtrusive in a way that fits naturally into my existing workflow. It doesn't feel like a detour. It feels like a tool that belongs in your terminal alongside the rest of my stack.

In this article I'm going to talk about why I needed Alfred, the problem he solves for me, how I built him and how I improve him on this ever changing landscape where engineering meets productivity.

The Monday Morning Problem Every Developer Knows

It is Monday, 8:30 AM. Before I have written a single line of code, I already have a full-time job just figuring out where to start.

Over the weekend, 47 new Gmail messages came in. Some are spam. Some are newsletters I never unsubscribed from. But buried somewhere in that pile is an escalation that needs urgent attention and a teammate asking for a code review. I do not know which email it is yet. I have to dig for it.

That is just Gmail. I also have 12 Outlook emails from work: meeting updates, an HR policy change, and my manager asking about feature progress. Then there are 8 Teams messages spread across 3 different channels covering a production incident from Saturday, a design review thread, and standup notes. On top of that, 3 pull requests were opened against repos I review, and 2 calendar conflicts appeared for Tuesday that I need to sort out before the day gets going.

None of these systems talk to each other. So my morning routine becomes a manual context-switching exercise. I open Gmail, scan subject lines, try to mentally rank urgency. Then I switch to Outlook and do the same. Then Teams. Then Azure DevOps. By the time I have a rough picture of what actually needs my attention, 45 to 60 minutes have passed. And that client escalation? Still buried under newsletters when I finally find it.

The frustrating part is that most of that time is not real work. It is just triage. It is the overhead that comes before the actual job even starts. The other option is to close everything and wait for someone to walk to my table. Lmao I do this all the time.

But well, this is the problem I built Alfred to solve.

What do I want from Alfred?

Unification! Alfred is a personal AI agent built around a single idea: collapsing the chaos of my digital workday into one intelligent, unified system. It continuously polls Gmail at configurable intervals and receives Outlook emails and Microsoft Teams messages via Power Automate webhooks, storing everything locally in SQLite so that regardless of the source, nothing slips through the cracks.
Every incoming email is then put through an AI classification pipeline that assigns it one of six categories (Urgent, Personal, Work, Newsletter, Transactional, or Spam), gives it a priority level from 1 to 5, generates a human-readable summary, extracts action items with optional due dates, and flags whether a follow-up is needed.
From there, a configurable rules engine evaluates each classified email and proposes an appropriate action: archive it, delete it, forward it, draft a reply, or surface it for attention via a notify action with quick-action buttons.
Destructive actions like deletions, sends, and PR approvals wait behind an explicit approval gate in the dashboard, while non-destructive ones like classification and drafting execute automatically.
Every action is tracked through a full lifecycle from proposed to executed, with timestamps, rollback data, and execution results all stored in an append-only audit log.

Beyond email, Alfred integrates deeply with the rest of my work toolchain. It connects to Google Calendar and Outlook Calendar for listing, creating, updating, and searching events, and handles Azure DevOps for querying and managing work items, approving pull requests, tracking pipeline runs, and browsing repositories. When a pull request is opened, a dedicated webhook handler automatically fetches the PR details, checks pipeline status, attempts to link related work items from branch name patterns, generates an LLM summary, and proposes approval or work item creation actions accordingly. Microsoft Teams is covered too, with channel message search and webhook-based ingestion keeping Alfred aware of conversations happening outside of email. Tying everything together is a conversational chat interface powered by an agentic loop that extracts intents from natural language, executes them across services, and returns structured, context-aware responses.

Let's look at some of Alfred's core flows in detail

Email Polling and Synchronization

Alfred's background worker is built around an AgentLoop flow. When the server starts, the agentLoop runs an initial poll immediately, then sets a repeating setInterval timer at a configurable cadence. Each tick calls a listMessages request emailPort.listMessages("in:inbox", 50) to fetch up to 50 messages from Gmail via the Gmail API. 50 is a reasonable number for my personal workflow

To avoid reprocessing emails Alfred has already seen, the loop maintains an in-memory string set of message IDs. Every polled message is checked against this set, and only genuinely new messages pass through:

const newMessages = messages.filter((msg) => !this.seenIds.has(msg.id));
for (const msg of newMessages) {
  this.seenIds.add(msg.id);
}

New messages are immediately persisted to SQLite through EmailRepo.upsert(). The upsert uses SQLite's INSERT ... ON CONFLICT(id) DO UPDATE pattern, which means if Alfred encounters the same email ID twice (for example after a server restart), it updates the existing row rather than creating a duplicate. The repository stores the full email body, sender, recipients, labels, attachments as serialized JSON, and a source field that distinguishes Gmail emails from Outlook emails. I cover the exact upsert schema in the Data Integrity section.

Before sending any email to the classifier, the loop applies a set of skip rules. Social media notifications from Facebook, Instagram, Twitter, TikTok, Reddit, Discord, and similar platforms are matched by regex against the sender address. Emails carrying Gmail's CATEGORY_PROMOTIONS or CATEGORY_SOCIAL labels are also skipped. LinkedIn is explicitly exempted from this filter because its emails often contain actionable professional content. This pre-filtering avoids burning LLM API calls on emails that would reliably classify as low priority anyway.

The loop also checks whether each email already has a classification in the database before sending it to the classifier. If a record exists, the email is skipped entirely. This means restarting the server does not trigger re-classification of previously processed emails. I wrote it this way to ensure minimum cost and idempotency.

When the classifier encounters a fatal error such as an expired API key, exhausted credit balance, or a 429 rate limit response, the loop enters a paused state rather than crashing or retrying in a tight loop. It sets classifierPaused = true and stops classifying. This is sort of a circuit breaker. On subsequent polls, it still persists new emails to the database so no mail is lost, but it attempts a single test classification to check whether the service has recovered. Once the test succeeds, classification resumes automatically. Error messages are also deduplicated so the same error is only logged once regardless of how many polls occur while paused.

For Outlook, Alfred does not poll directly. Instead, an adapter calls a Power Automate flow that returns Outlook messages. A dedicated payload mapper normalizes Microsoft field names, timestamp formats, and nested structures into the same EmailMessage domain object that Gmail produces. This means the rest of the pipeline, including classification, action rules, and chat, works identically regardless of whether an email originated from Gmail or Outlook. I wrote it this way so that I can later extend email providers by just adding a normalization mapper and then it should be plug and play.

Action Proposal, Approval, and Execution

Actions in Alfred follow an event-sourced lifecycle. Every state transition is recorded as an append-only entry in action log in an SQLite table. No rows are ever updated in place or deleted. The lifecycle flows through a fixed set of ActionStatus states: Proposed → Approved → Executed, or alternatively Rejected or RolledBack. This is purely for auditing so that I can track autonomous actions from the agent.

Proposal

The ProposeAction use case starts with an idempotency check. It queries the action log for any existing entry with the same resourceId and type. If one already exists, it returns null and stops. Otherwise, it appends a new entry with status: Proposed.

From there, the action's RiskLevel determines what happens next. Low-risk actions like Classify, Draft, and Notify carry RiskLevel.Auto and execute immediately without my input. High-risk actions like Archive, Delete, Send, and Forward carry RiskLevel.ApprovalRequired and sit in the proposed state until I act on them from the dashboard:

const risk = ACTION_RISK_LEVELS[action.type];
if (risk === RiskLevel.Auto) {
  const strategy = this.strategies.find((s) => s.source === action.source);
  if (strategy?.canExecute(action.type)) {
    resultData = await strategy.execute({ type, resourceId, payload });
  }
  await this.actionLog.updateStatus(actionId, ActionStatus.Executed, new Date().toISOString());
}

If the action produces result data such as a created draft ID or classification details, that data is stored alongside the log entry via updateResultData().

Approval and Execution

When I click Approve in the dashboard, the ApproveAction use case first updates the log entry's status to Approved with a timestamp, then immediately attempts execution. It finds the correct ActionExecutionStrategy by matching the action's source field. Three strategies exist: GmailActionStrategy handles archive, delete, send, and draft operations via the Gmail API; OutlookActionStrategy handles equivalent operations through Power Automate; and DevOpsActionStrategy handles work item creation and PR approval via the Azure DevOps REST API. This is based on the open-closed principle to allow for the extension and registration of multiple strategies.

Each strategy declares which action types it supports through a canExecute() method. If a strategy exists but cannot execute the specific action type, the action is marked as executed without performing any real mutation. If execution succeeds, the status moves to Executed. If it fails, the error is returned to the caller but the action remains in Approved state so the user can retry without losing the approval.

The Notify action type is intentionally a no-op at the execution level. It exists so the rules engine can propose surfacing an email to the user without triggering any mutation on the mailbox. The notification itself is handled by the push notification system, not the action executor.

Chat Interface (Intent and Tool Use Modes)

Alfred's chat is the primary way I interact with my workspace data through natural language. I designed it to support two distinct modes of operation, an intent extraction mode (the default) and tool_use mode powered by Anthropic's internal tool choice algorithm. Both implement a ChatStrategy interface defined in a chat-strategy file, which standardises the input (message, history, context, system prompt, dependencies) and output (response text, result strings, action steps).

Intent Extraction Mode

The IntentExtractionStrategy uses a two-LLM architecture. A fast, cheap model (Claude Haiku) handles intent extraction, while the main model (Claude Sonnet) composes the final user-facing response.

The strategy runs an agentic loop of up to 5 rounds. In each round, it sends the user's message, the last 20 conversation history entries (each truncated to 2000 characters), and any results from prior rounds to the fast LLM. The system prompt includes detailed routing rules that map natural language patterns to intent types: "check my Outlook" routes to search_emails with source: "outlook", "calendar" without a provider routes to list_calendar_events without a source, and "work items" routes to query_work_items.

The LLM returns a JSON object with an intents array. Each intent specifies a type matching a registered tool name, along with type-specific fields like query, source, and timeMin. Invalid tool names are filtered out against the ToolRegistry. The strategy then executes each intent by calling the corresponding tool's execute() function, which delegates to the appropriate IntentExecutorDeps method:

for (let round = 0; round < MAX_ROUNDS; round++) {
  const intents = await this.extractIntents(extractionLlm, message, recentHistory, priorResults, deps, validToolNames);
  if (intents.length === 0) break;
  const results = await this.executeTools(deps, intents);
  allResults.push(`--- Round ${round + 1} ---\n${results.join("\n\n")}`);
}

Multi-round execution is what makes complex queries possible. A request like "invite Sabrina to my 3pm meeting tomorrow" requires two rounds: round 1 searches for tomorrow's calendar events, and round 2 uses the event ID from that result to update the event with a new attendee. The LLM receives prior results in an ACTIONS ALREADY EXECUTED THIS TURN block and can return {"intents": [{"type": "none"}]} to signal that all needed data has been gathered and the loop should stop.

After the loop completes, the ChatService combines all gathered results with local context (email stats, pending actions, and follow-ups from the database) and sends everything to the main LLM for final response composition, with extended thinking enabled.

Tool Use Mode

The ToolUseStrategy takes a fundamentally different approach. Rather than extracting intents and executing them as a separate step, it gives the LLM direct access to tools via completeWithTools(). The LLM decides which tools to call, receives structured results, and continues the conversation until it produces a final text response.

This mode requires the LLM adapter to support the Claude tool-use API. The strategy converts all registered tools into Claude tool definitions (name, description, input schema) and passes them alongside the message. The loop runs for up to 5 rounds, checking the stopReason after each response. When the model returns end_turn, the final text becomes the response. When it returns tool calls, the strategy executes each tool, packages the results as ToolResultBlock objects with matching tool_use_id, and sends them back as a user message for the next round:

const response = await deps.llm.completeWithTools({ system: systemPrompt, messages, tools, maxTokens: 4096 });
if (response.stopReason === "end_turn") {
  return { response: response.text ?? "", results: allResults, actions: allActions };
}

If the model exhausts all 5 rounds without reaching end_turn, the strategy returns a graceful fallback message in Alfred's butler voice rather than surfacing a raw error to the user.

Tool Registry

Both modes share the ToolRegistry class in a tool-registry file, which acts as a central catalogue of all available tools. Each tool is registered with a name, description, JSON input schema, an execute function, and a summarize function that produces human-readable action steps such as "Searched Gmail for 'invoice'". The registry can export its tools in two formats: toToolDefinitions() for Claude's native tool-use API, and toIntentPrompt() for building the intent extraction system prompt.

System Prompts

All persona and mode-specific instructions are centralised in a system-prompts file. The BASE_PERSONA establishes Alfred's character as a refined English butler who addresses the user as "Master Jo" and has access to Google Workspace, Microsoft 365, and Azure DevOps. (Jeremy Irons is my favorite Alfred btw) Mode-specific instructions are appended on top: intent mode tells Alfred that actions have already been executed and results are in context so it should not pretend to be searching, while tool-use mode tells Alfred to actively call tools to fetch fresh data.

Authentication and Security

Alfred enforces security at multiple levels across both the dashboard and the agent server.

Dashboard Authentication

The dashboard uses NextAuth.js v5 configured in auth.ts with Google OAuth as the sole provider. Sessions use a JWT strategy with a 7-day maximum age. Access is restricted to a single authorised user through an email allowlist: the signIn callback compares the Google profile's email against the ALLOWED_EMAIL environment variable and rejects any mismatch:

callbacks: {
  signIn({ profile }) {
    return profile?.email?.toLowerCase() === allowedEmail;
  },
}

The auth system uses a custom sign-in page at /auth/login and redirects errors back to the same page for a clean user experience. Since Alfred is a personal, single-user tool, the allowlist approach is both simpler and more appropriate than a full role-based access system.

Server-Side Credentials

The agent server stores sensitive credentials in the macOS Keychain. Both are fetched lazily on first use and cached in memory for the lifetime of the process. This means credentials never appear in environment variables, config files, or logs.

Architectural Isolation

The dashboard is a pure client-rendered application. It contains no provider SDK imports, no direct database access, and no secret values. All data access flows through the agent server's HTTP API. I made sure that all credentials are ignored. This means that even if the dashboard source code were fully exposed, it would not leak any credentials or grant any access to the underlying data.

Resilience and Caching

Alfred applies several resilience patterns across the system to handle network failures, API rate limits, and performance constraints.

In-Memory TTL Cache

The TtlCache class in cache.ts provides a simple time-to-live cache backed by a JavaScript Map. Each entry stores its data alongside an expiresAt timestamp. The get() method checks expiration on every access and automatically evicts stale entries. The getOrFetch() method combines cache lookup with lazy population:

async getOrFetch<T>(key: string, ttlMs: number, fetcher: () => Promise<T>): Promise<T> {
  const cached = this.get<T>(key);
  if (cached !== undefined) return cached;
  const data = await fetcher();
  this.set(key, data, ttlMs);
  return data;
}

This is used for calendar events and DevOps data, both cached with a 3-minute TTL. During a multi-round chat conversation where Alfred might query the calendar several times, only the first call hits the API and subsequent calls return the cached result. The 3-minute window balances data freshness with meaningful API call reduction.

Agent Loop Resilience

The classifier pause behavior is covered in the Email Polling section above. Beyond that, the polling loop is designed so that a failure in any single stage — classification, action proposal, or action execution, does not crash or block the rest of the loop. Each stage fails independently and logs the error without taking down the whole cycle.

Power Automate Retries

The Power Automate client implements a 3-attempt retry with linear backoff (1s, 2s, 3s) for transient HTTP errors and timeouts. Non-retryable errors such as 4xx client errors (excluding 429) fail immediately without retrying. Each request uses AbortController with a 30-second timeout to prevent indefinite hangs.

Push Notification Delivery

The web push delivery mechanics including concurrent sends, Promise.allSettled(), and automatic cleanup of expired subscriptions are covered in the Push Notifications section under Discoveries where the full implementation is explained in context.

Deployment and Operations

Alfred runs as three persistent background services on macOS, managed by launchd, Apple's native process manager. The deployment system is entirely script-based with no containers, no cloud infrastructure, and no external process managers. Everything runs on a single Mac.

The Three Services

The agent server is the core process. It runs the Node.js HTTP API, the background email polling loop, the action execution pipeline, and the finance statement processor. It owns all external API calls to Gmail, Google Calendar, Anthropic, Azure DevOps, and Power Automate, along with all OAuth credentials stored in macOS Keychain and the SQLite database.

The dashboard is a Next.js application serving the client-rendered UI. In production it runs against a pre-built output directory and makes no direct calls to any external service. All data comes through the agent server's HTTP API. It receives a bearer token as an environment variable so it can authenticate its requests to the agent server.

The Cloudflare tunnel creates an encrypted outbound connection from the Mac to Cloudflare's edge network, making the dashboard publicly accessible without opening any inbound ports or touching the router. It routes HTTPS traffic from the public domain down to the local Next.js server on a local port.

launchd Service Configuration

Each service is defined as a .plist property list file. The plist files use placeholder tokens that are replaced with real values at deploy time using sed. The key properties are RunAtLoad: true to start on login, KeepAlive: true to auto-restart on crash, and ThrottleInterval: 10 to wait at least 10 seconds between restart attempts and prevent tight crash loops:

<key>ProgramArguments</key>
<array>
    <string>PROJECT_ROOT/node_modules/.bin/tsx</string>
    <string>apps/agent-server/src/index.ts</string>
</array>
<key>KeepAlive</key>
<true/>
<key>ThrottleInterval</key>
<integer>10</integer>

Each service logs stdout and stderr to separate files that can be tailed in real time for debugging.

The Deploy Script

Deployment runs through a single script that orchestrates six steps in order:

creating the log directory
sourcing the .env file to load environment variables
running npm install at the monorepo root to install all workspace dependencies
running npm run build to compile all TypeScript packages in dependency order (domain → application → infrastructure → contracts → agent server, then the Next.js dashboard)
copying each plist template into ~/Library/LaunchAgents/ with placeholders replaced by real paths,
And finally loading all three services with launchctl load to start them immediately. Before installing each plist it unloads any previously running version to prevent conflicts, resulting in a brief restart with minimal downtime:

for plist in com.alfred.agent.plist com.alfred.dashboard.plist com.alfred.cloudflared.plist; do
  launchctl unload "$LAUNCH_AGENTS_DIR/$plist" 2>/dev/null || true
  sed -e "s|PROJECT_ROOT|$PROJECT_ROOT|g" \
      -e "s|USER_HOME|$USER_HOME|g" \
      -e "s|CLOUDFLARED_BIN|$CLOUDFLARED_BIN|g" \
      -e "s|NODE_BIN_PATH|$NODE_BIN_PATH|g" \
      -e "s|BEARER_TOKEN_VALUE|${BEARER_TOKEN:-}|g" \
      "$DEPLOY_DIR/$plist" > "$LAUNCH_AGENTS_DIR/$plist"
done

The script automatically detects the Node.js binary path across nvm, Homebrew, and system installs, and locates the cloudflared binary for both Apple Silicon and Intel Homebrew paths. At the end it prints a macOS settings checklist reminding me to enable auto-login, prevent sleep, and configure startup after power failure, since the Mac effectively acts as a persistent home server.

First-Time Setup

Initial installation is handled by a setup script that checks prerequisites (Homebrew and Node.js 20 or above), installs cloudflared, creates the .env file interactively, runs the Google OAuth flow by opening a browser for consent and storing the resulting refresh token in Keychain, authenticates with Cloudflare, creates the tunnel, configures DNS routes, and then kicks off the deploy script to bring everything up.

Operational Commands

I have scripts for the full operational lifecycle. A status command shows whether each service is running, its PID, and the last 5 log lines. A teardown command unloads all services and removes the plist files from LaunchAgents while preserving logs. A universal launcher supports multiple modes: all for full production, dev for hot-reload development, agent or dashboard individually, status for health checks, and doctor for preflight validation.

Configuration

All configuration flows through environment variables loaded from a .env file at the project root. A config.ts module reads these and returns a typed AppConfig object. Three variables are required: GOOGLE_CLIENT_ID, GOOGLE_CLIENT_SECRET, and ANTHROPIC_API_KEY. Everything else is optional and enables features progressively. Setting AZURE_DEVOPS_ORG enables DevOps integration. Setting PA_FLOW_MAIL_SEARCH enables Outlook. Setting VAPID_PUBLIC_KEY enables push notifications and so on. If an optional config block is absent, the composition root simply skips registering those adapters and use cases, so the system degrades gracefully rather than failing to start.

Data Integrity

Ensuring that Alfred handles data meticulously was very important to me. It does not make sense to build an assistant that is sloppy with the information it presents. Therefore I wrote Alfred in such a way that he prevents duplicate and inconsistent data through idempotency checks, upsert semantics, and schema separation at every data boundary.

Idempotent Action Proposals

Before creating a new entry in the action log, the proposal system queries for any existing entry with the same resourceId and type. If a match is found, the proposal is silently skipped and returns null. This means the polling loop can encounter the same email multiple times, such as after a server restart, without generating duplicate action proposals:

const existing = await this.actionLog.getByResourceIdAndType(action.resourceId, action.type);
if (existing) return null;

Email Upsert Semantics

Whether an email arrives via polling, a webhook, or is encountered again after a restart, the upsert guarantees exactly one row per email ID. All fields including subject, body, labels, and read status are updated to their latest values, and an updated_at timestamp records when the last refresh occurred:

INSERT INTO emails (id, thread_id, from_address, ..., updated_at)
VALUES (@id, @threadId, @from, ..., datetime('now'))
ON CONFLICT(id) DO UPDATE SET
  thread_id = excluded.thread_id,
  from_address = excluded.from_address,
  ...
  updated_at = datetime('now')

Conversation Ordering

Chat messages are stored with a created_at timestamp and always queried in chronological order using ORDER BY created_at ASC. Messages are never reordered, edited, or deleted after creation. This ensures the conversation history Alfred sees when composing a response exactly matches what the user experienced.

Normalised Schema Design

Classifications are stored in a separate classifications table linked to emails by email_id. This separation means re-classifying an email, whether due to a model update or a rule change, only touches the classification row without affecting the underlying email data. The email's original content, headers, labels, and metadata remain untouched. Follow-ups and action log entries follow the same pattern. Each table has a single source of truth for its own data, and no operation on one table can corrupt another.

Pitfalls: From Intent Extraction to Tool Use

I started Alfred's chat system with a pure intent extraction approach. The idea was straightforward: send my message to a fast LLM, ask it to return structured JSON with an intent type and parameters, then map that intent to an executor function. A message like "show me today's calendar" would produce {"type": "list_calendar_events", "timeMin": "2026-03-16", "timeMax": "2026-03-16"}, and the system would call the calendar adapter directly:

const intents = await this.extractIntents(extractionLlm, message, recentHistory, priorResults, deps, validToolNames);
for (const intent of intents) {
  const entry = deps.toolRegistry.get(intent.type as string);
  if (entry) {
    const result = await entry.execute(deps.intentExecutor, intent);
    if (result) results.push(result);
  }
}

I built this following the Open/Closed Principle. Each intent type was a self-contained ToolEntry registered in a ToolRegistry. Adding a new capability meant registering a new entry with a name, schema, executor function, and summariser. No existing code needed modification:

toolRegistry.register({
  name: "search_emails",
  description: "Search emails by query, category, or sender",
  inputSchema: { ... },
  execute: async (deps, input) => { ... },
  summarize: (input) => `Searched emails: ${input.query}`,
});

In theory this was clean and extensible. In practice, the cost of adding intents started to compound. Every new capability required writing a system prompt fragment describing the intent format, adding routing rules so the LLM knew when to select it, writing the executor function, and testing that the LLM reliably produced the right JSON structure. At 5 intent types it was manageable. By the time I had 15 (email search, calendar list, calendar create, calendar update, calendar search, work item query, work item create, PR query, pipeline list, Teams messages, follow-ups, actions, repo list, commits, branch list), the intent extraction system prompt had ballooned. The LLM was juggling too many format rules and frequently produced malformed JSON or selected the wrong intent type.

The extraction prompt had grown to include detailed routing rules, source-specific provider logic, multi-intent support, and follow-up round awareness:

const INTENT_RULES = `
ROUTING RULES:
- "check my Outlook" → search_emails with source: "outlook"
- "search Gmail" → search_emails with source: "gmail"
- "Outlook calendar" → list_calendar_events with source: "outlook-calendar"
- "work items" / "tickets" → query_work_items
- "pull requests" / "PRs" → query_source_control with subtype: "pull_requests"
...
`;

Every new intent meant updating these routing rules, testing edge cases, and hoping the model did not confuse the new intent with existing ones. The Open/Closed architecture was holding up at the code level — I was not modifying existing executors, but the prompt was a single growing artifact shared by every intent. Adding one intent risked degrading the reliability of all the others.

This led me to Claude's native tool use API. Instead of asking the LLM to produce JSON matching my custom schema, I could give it proper tool definitions and let Claude's built-in tool calling handle the routing:

const tools = deps.toolRegistry.toToolDefinitions();
const response = await deps.llm.completeWithTools({
  system: systemPrompt,
  messages,
  tools,
  maxTokens: 4096,
});

Claude's tool use was noticeably more reliable. It natively understands tool schemas, validates parameters against the input schema, and handles multi-tool calls cleanly. The model picks the right tool more consistently than my intent extraction prompt ever did, because tool selection is a first-class capability of the model rather than something I was trying to engineer through prompt instructions.

But tool use burned through API credits quickly. Each round of the conversation becomes a full API call carrying the entire tool catalogue, conversation history, and system prompt. A simple question like "what meetings do I have today?" that previously cost one cheap Haiku call for intent extraction plus one Sonnet call for response composition now cost one or more full Sonnet calls with tool definitions attached, adding significant token overhead to every request.

I balanced models to keep costs sustainable. Intent extraction uses Haiku because it only needs to produce structured JSON, not reason deeply. Final response composition uses Sonnet with extended thinking enabled because that is where quality matters:

const strategyDeps = {
  llm: this.deps.llm,         // Sonnet — reasoning and response
  fastLlm: this.deps.fastLlm, // Haiku — intent extraction
  ...
};

Rather than committing to one approach, I gave the chat system the ability to switch between both modes. The mode parameter on each request selects the active strategy:

const strategy = mode === "tool_use" ? toolUseStrategy : intentStrategy;
const strategyResult = await strategy.run({ message, history, localContext, systemPrompt, deps });

Intent mode is cheaper and faster for straightforward queries where the routing rules work well. Tool use mode is more reliable for complex, ambiguous, or multi-step requests where maintaining routing rules would be impractical. Both strategies implement the same ChatStrategy interface and share the same ToolRegistry, so all capabilities are available in both modes without any duplication.

From Single Request-Response to Reasoning Loops

Early on, the chat used a single request-response pattern. I ask a question, Alfred gathers context from the database, sends everything to the LLM in one shot, and returns the response. The quality was poor. With 15+ tools and a rich system prompt, the model would frequently miss details, give shallow answers, or fail to connect information across multiple data sources. A question like "what's my schedule like tomorrow and do I have any overdue follow-ups?" would produce a partial answer because the model was trying to handle everything in a single pass.

My first instinct was to use a better model. I switched from Sonnet to Opus for the response composition step and the quality jumped immediately. Opus reasons more carefully, connects dots across context, and produces noticeably more nuanced responses. But it was expensive. Opus costs significantly more per token than Sonnet, and every chat message was a full context window call carrying email stats, action history, follow-up data, and conversation history.

This led me to implement reasoning loops. Instead of asking the model to do everything in one pass, I let it work iteratively. In intent mode, the strategy runs up to 5 rounds. Each round extracts intents, executes them, and feeds the results back into the next round's context:

for (let round = 0; round < MAX_ROUNDS; round++) {
  const intents = await this.extractIntents(extractionLlm, message, recentHistory, priorResults, deps, validToolNames);
  if (intents.length === 0) break;
  const results = await this.executeTools(deps, intents);
  allResults.push(`--- Round ${round + 1} ---\n${results.join("\n\n")}`);
}

In tool use mode, the loop is similar but driven by Claude's stop reason. The model keeps calling tools until it decides it has enough information and returns a final text response:

for (let round = 0; round < MAX_ROUNDS; round++) {
  const response = await deps.llm.completeWithTools({ system: systemPrompt, messages, tools, maxTokens: 4096 });
  if (response.stopReason === "end_turn") {
    return { response: response.text ?? "", results: allResults, actions: allActions };
  }
  // ... execute tool calls, feed results back
}

This multi-round approach means a request like "invite Sarah to my 3pm meeting tomorrow" works naturally.
Round 1 searches tomorrow's calendar events.
Round 2 uses the event ID from that result to update the event with a new attendee. The LLM sees prior results in an ACTIONS ALREADY EXECUTED THIS TURN block and returns {"intents": [{"type": "none"}]} when everything is resolved and the loop should stop.

{"timestamp":"2026-03-16T07:11:03.210Z","level":"info","msg":"\nchat:start","component":"chat","message":"What does my outlook calendar look like ?","historyLength":16,"mode":"tool_use"}
{"timestamp":"2026-03-16T07:11:07.854Z","level":"info","msg":"llm:completeWithTools","component":"llm","model":"claude-opus-4-6","inputTokens":8168,"outputTokens":131,"durationMs":4644,"stopReason":"tool_use"}
{"timestamp":"2026-03-16T07:11:07.854Z","level":"info","msg":"chat:tool-use-round","component":"chat","round":1,"stopReason":"tool_use","toolCallCount":1,"hasText":true,"durationMs":4644}
{"timestamp":"2026-03-16T07:11:07.855Z","level":"info","msg":"chat:tool-result","component":"chat","tool":"list_calendar_events","resultLength":33,"resultPreview":"Calendar Events: No events found."}
{"timestamp":"2026-03-16T07:11:13.314Z","level":"info","msg":"llm:completeWithTools","component":"llm","model":"claude-opus-4-6","inputTokens":8318,"outputTokens":120,"durationMs":5458,"stopReason":"end_turn"}
{"timestamp":"2026-03-16T07:11:13.315Z","level":"info","msg":"chat:tool-use-round","component":"chat","round":2,"stopReason":"end_turn","toolCallCount":0,"hasText":true,"durationMs":5459}
{"timestamp":"2026-03-16T07:11:13.315Z","level":"info","msg":"chat:complete","component":"chat","totalDurationMs":10106,"mode":"tool_use","actionCount":1}

The reasoning happens where it counts. Mechanical work like deciding which tools to call uses the cheapest model that can do it reliably, and the expensive synthesis step only fires once at the end. A 3-round conversation costs 3 Haiku calls plus 1 Sonnet call rather than 3 Opus calls.

Prompt Refinement

Prompt refinement turned out to be significantly harder with intent extraction than with tool use. With intent extraction, I was responsible for the entire instruction surface: routing rules, format specifications, edge case handling, multi-intent support, source disambiguation, date inference, and conversational context awareness. Every ambiguous user message required a new rule or clarification in the prompt. The prompt became a fragile, growing document where changing one section could silently break another.

With tool use, Claude does most of the heavy lifting. I define each tool's name, description, and input schema. Claude figures out when to call it, what parameters to pass, and how to combine results across multiple tools. The refinement effort shifted from "teach the model my custom intent format" to "write clear tool descriptions and let the model's built-in tool selection do its job." This was a dramatically smaller surface area to maintain.

The persona prompt is where I spent the most deliberate effort, and I structured it to follow the Open/Closed Principle. The BASE_PERSONA defines Alfred's character, his access to workspace systems, and the critical behavioural rules that apply regardless of which mode is active:

export const BASE_PERSONA = `You are Alfred, a distinguished personal workspace assistant. 
You are an old English gentleman — impeccably dressed in a three-piece suit at all times, 
refined in manner, and utterly devoted to your employer. You always address the user as 
"Master Jo". Your speech carries the quiet authority and warmth of a seasoned butler...

CRITICAL RULES:
- ALWAYS address the user as "Master Jo"
- ONLY use the data provided to you. Do not make up emails, events, or results.
- When calendar events were CREATED, confirm this to the user with details and calendar links.
...`;

Mode-specific instructions are appended on top without touching the base. Intent mode tells Alfred that actions have already been executed and results are already in context, so he should not pretend to be searching. Tool use mode tells Alfred to actively call tools to fetch fresh data. The buildSystemPrompt() function composes these cleanly:

export function buildSystemPrompt(mode: "intent" | "tool_use"): string {
  const modeInstructions = mode === "tool_use" ? TOOL_USE_MODE_INSTRUCTIONS : INTENT_MODE_INSTRUCTIONS;
  return BASE_PERSONA + "\n" + modeInstructions;
}

This separation means I can refine Alfred's personality, add new behavioural rules, or adjust mode-specific instructions entirely independently. Adding a new mode in the future means writing a new instruction block and adding a case to buildSystemPrompt(), without touching the persona or any existing mode instructions.

The persona itself evolved through iteration. Early versions were too stiff and formal. Later versions overcorrected and became too casual. The current version balances warmth with efficiency, giving Alfred permission to be dry-witted and occasionally opinionated while staying concise and never fabricating data.

Discoveries

The Floodgate Effect

Once I had the first working version of Alfred deployed, something unexpected happened: my mind would not stop generating ideas. The initial version could poll Gmail, classify emails, propose actions, and let me approve them from a dashboard. It was functional, but using it every day exposed gaps and opportunities I had not anticipated during planning. Every morning I would open the dashboard, see how Alfred handled my overnight inbox, and think "what if he could also do this?" The backlog grew faster than I could build.

This is something I did not expect about building a personal tool. When you are the only user, the feedback loop is immediate. There is no product manager filtering requests, no sprint planning, no prioritisation meetings. You feel the friction directly, and the fix is always within reach. That immediacy is both a gift and a trap. I had to learn to be disciplined about scope, because every "quick addition" carries a maintenance cost that compounds.

Financial Statement Processing

The first major expansion came from a personal pain point. I bank with multiple banks in Malaysia, and both send monthly e-statements as password-protected PDF attachments to my Gmail. Every month I would download the PDFs, unlock them, manually scan through transactions, and try to categorise spending in a spreadsheet. I actually stopped this a long time ago. It was tedious, error-prone, and I rarely kept up with it. I realised Alfred already had the infrastructure to solve this: he polls Gmail, he can download attachments, and he has an LLM for classification.

I built a six-stage pipeline that runs automatically during each polling cycle. Alfred searches Gmail for emails from the configured bank sender addresses, filters for emails with PDF attachments, and checks each against the bank_statements table to skip already-processed ones. The idempotency check matters because the polling loop runs every 60 seconds and the same bank emails will appear in search results repeatedly:

private async findUnprocessedIds(bank: BankConfig, filters: EmailSearchFilters): Promise<string[]> {
  const ids = await this.deps.emailRead.searchFilteredIds(filters);
  const unprocessed: string[] = [];
  for (const id of ids) {
    if (!(await this.deps.statementRepo.isStatementProcessed(id))) {
      unprocessed.push(id);
    }
  }
  return unprocessed;
}

For each unprocessed email, Alfred downloads the PDF attachment and decrypts it using the bank-specific password from environment config. This is where I hit the first real bug. The pdf-parse library accepts a password option, but its internal implementation completely ignores it. It passes the raw buffer directly to PDF.js's getDocument() instead of wrapping it in { data, password }. Every statement was failing with a cryptic "No password given" error. The fix was a workaround that tricks pdf-parse by passing a PDF.js parameter object in place of the buffer:

const pdfInput = { data: new Uint8Array(pdfBuffer), password } as unknown as Buffer;
const result = await pdf(pdfInput);

After decryption, the raw text goes to a bank-specific parser. Each bank formats its statements differently, so I built a StatementParserRegistry that routes to the correct parser based on the BankProvider enum.

The parser also strips page noise including headers, footers, and the Chinese and Malay translations that some banks include on every page, and collects multi-line transaction details like merchant names and reference numbers.

Once parsed, transactions go through a hybrid classification stage. The HybridTransactionClassifier first attempts rule-based categorisation using keyword matching (merchant names like "GRAB" map to transport, "MCDONALD'S" maps to food), and falls back to Claude Haiku for ambiguous transactions. This hybrid approach keeps costs low because most transactions have recognisable merchant names that do not need LLM inference.

The pipeline also handles historical backfill. On first run, it does not just process recent statements. It walks backward through the inbox month by month, processing older statements until it reaches a configurable cutoff, defaulting to 12 months. A backfill_state table tracks the cursor position per bank so the backfill can resume across server restarts:

private async processBackfill(bank: BankConfig): Promise<void> {
  const isComplete = await this.deps.backfillStateRepo.isComplete(bank.bankProvider);
  if (isComplete) return;

  const cursor = await this.deps.backfillStateRepo.getCursor(bank.bankProvider);
  const cutoff = new Date();
  cutoff.setMonth(cutoff.getMonth() - this.deps.backfillMonths);
  // ... fetch historical emails before cursor, process, advance cursor
}

All of this produces a normalised finance_transactions table where every transaction from every bank shares the same schema: date, description, amount, type (credit or debit), balance, category, merchant name, and statement period. Two banks, different formats, one unified table.

Making Financial Data Conversational

Having the data in SQLite was useful on its own, the dashboard has a Finance page with tables and charts, but the real power came from wiring it into Alfred's chat. I registered finance-specific tools in the ToolRegistry so that both chat modes can query transaction data naturally.

The chat can now answer questions like "how much did I spend on food last month?", "what were my biggest transactions in February?", or "show me all Grab transactions this year." Alfred queries the finance_transactions table, aggregates the results, and presents them in his butler persona.

What I did not anticipate is that this naturally enabled budgeting. Once Alfred could tell me "you spent RM 2,400 on dining in February, Master Jo," I started asking follow-up questions like "is that more than January?" and "set a reminder if I go over RM 2,000 next month." The transaction data combined with the follow-up system and push notifications created a lightweight budget monitoring capability that I never explicitly designed. It emerged from the intersection of features that already existed.

Progressive Web App

The dashboard started as a standard Next.js web app accessed through a browser tab. It worked, but it felt disposable. I would forget to check it, or close the tab and lose my place. Making Alfred a Progressive Web App changed that relationship. With a PWA manifest, a service worker, and the right meta tags, Alfred became an app I could install on my phone and in my Mac's dock. It has its own window, its own icon, and it persists across reboots.

The practical difference is small since it is still the same Next.js app behind the scenes. But the psychological difference is significant. An app in the dock feels like a tool. A browser tab feels temporary. I open Alfred every morning now the way I open Slack or my email client. It has presence.

Push Notifications with Service Workers

The feature I am most proud of is the push notification system. Before I built it, Alfred was purely pull-based. I had to open the dashboard to see if anything needed attention. Proposed actions would sit in the approval queue for hours because I simply forgot to check. Follow-ups would go overdue silently.

Push notifications made Alfred proactive. When the classification pipeline proposes a new action for approval, Alfred sends a push notification to my browser. When a high-priority email arrives, he notifies me immediately. When a DevOps PR webhook fires, I get a notification with a deep link straight to the approvals page.

The implementation uses the Web Push protocol with VAPID keys for authentication. The SendNotification use case checks user preferences before sending. I can toggle notifications per event type from the Settings page, and for high-priority emails I can set a minimum priority threshold:

const pref = await this.preferenceRepo.get(event.type);
if (pref && !pref.enabled) return;

if (event.type === NotificationEventType.HighPriorityEmail && emailPriority !== undefined) {
  const threshold = PRIORITY_THRESHOLDS[minPriority] ?? PRIORITY_THRESHOLDS.high;
  if (emailPriority > threshold) return;
}

The WebPushAdapter sends to all registered browser subscriptions concurrently using Promise.allSettled(), so a failed delivery to one device does not block others. It automatically cleans up expired subscriptions when the push service returns HTTP 410 or 404, which happens when a user clears browser data or uninstalls the PWA.

On the client side, a service worker listens for push events and displays native OS notifications with the app icon, a body preview, and a deep link URL. The notificationclick handler is smart about reusing existing windows: if the dashboard is already open, it focuses that tab instead of opening a new one:

self.addEventListener("notificationclick", (event) => {
  event.notification.close();
  const url = event.notification.data?.url ?? "/";
  event.waitUntil(
    self.clients.matchAll({ type: "window", includeUncontrolled: true }).then((clients) => {
      for (const client of clients) {
        if (client.url.includes(url) && "focus" in client) return client.focus();
      }
      return self.clients.openWindow(url);
    }),
  );
});

The usePushNotifications React hook manages the entire subscription lifecycle from the UI: checking browser support, requesting notification permission, fetching the VAPID public key from the server, subscribing via the Push API, and sending the subscription details to the server for storage. Unsubscribing reverses the process, removing the subscription from both the browser and the server database.

What made this feel like a real discovery is how it changed my workflow. Before push notifications, Alfred was a dashboard I checked. After push notifications, Alfred is an assistant who taps me on the shoulder. The difference between pull and push is the difference between a tool and a colleague. When my phone buzzes with "Action: archive. Proposed archive for 'Your NIKE order has shipped', Master Jo," I smile every time. It feels like Alfred is actually there, running the household.

Further Implementations

Retrieval-Augmented Generation for Personal Knowledge

The next frontier I want to explore is giving Alfred deep knowledge of everything I have written. I publish articles, write tweets, draft technical documentation, and take notes across multiple platforms. Right now Alfred knows my emails, my calendar, and my finances, but he does not know my voice. If someone asks me to write a thread about Clean Architecture, I start from scratch every time. If I need to reference a point I made in an article six months ago, I have to search manually.

I plan to build a RAG pipeline that indexes my published content, tweets, notes, and drafts into a vector store. A good friend of mine (Edem Kumodzi) already does this, read his article here. When I ask Alfred to help me write something, he would retrieve relevant passages from my own prior work and use them as context for generation. The goal is not for Alfred to write as me, but to write with full awareness of what I have already said, how I say it, and what positions I have taken. He should be able to say: "Master Jo, you wrote about this exact topic in your March article. Shall I pull the relevant points as a starting foundation?"

This is a step toward something larger. I want Alfred to have a total embodiment of who I am — not a shallow personality clone, but a deep contextual understanding of my thinking, my writing style, my professional opinions, and my personal preferences. He should know that I care about Clean Architecture and SOLID principles, that I have strong opinions about over-engineering, and that I prefer concise explanations with concrete examples. At the same time, he should remain his own person: a distinct entity with his butler persona who assists me rather than pretending to be me. The line between "knows me well" and "impersonates me" is one I want to walk carefully.

Expanding Service Integrations

Alfred currently connects to Google Workspace, Microsoft 365, and Azure DevOps. I want to push further into the services that shape my daily life.

WhatsApp is where most of my personal communication happens. The ability to search messages, get summaries of group conversations I have missed, or draft replies through Alfred would close a major gap. The challenge is that WhatsApp's API is designed for businesses rather than personal use, so I will likely need to explore the WhatsApp Business API with creative workarounds.

LinkedIn is the integration I am most excited about. I got the idea from a podcast about the discipline of maintaining professional relationships, and it resonated because I am genuinely terrible at it. I connect with people at conferences, have great conversations, and then never follow up. Alfred could do something far more personal than LinkedIn's built-in "keep in touch" feature: track my connections, identify people I have not interacted with in a while, cross-reference them with my calendar and email history, and nudge me with context. Not just "you haven't talked to Sarah in 3 months" but "you haven't talked to Sarah in 3 months. You last discussed the migration project at her company. She posted about a promotion last week. Shall I draft a congratulations message, Master Jo?" That level of contextual nudging is what turns a contact list into actual relationships.

Spotify might seem like an odd fit for a workspace assistant, but I spend a significant amount of my commute and focus time listening to engineering podcasts. I want Alfred to suggest relevant episodes based on what I am currently working on. If I am deep in a week of building a notification system, Alfred could recommend episodes about push notification architecture, service workers, or PWA best practices. The Spotify API is well-documented with solid search and recommendation endpoints, so this should be one of the more straightforward integrations to build.

Smart Home Integration

I have been thinking about extending Alfred beyond the digital workspace and into my physical space. Apple Shortcuts provides a bridge between software and home devices. If I can trigger Shortcuts programmatically, Alfred could control lights, check device status, set scenes, and interact with HomeKit accessories through natural language.

The most entertaining use case involves Juliana, my robot vacuum. She runs on a schedule, but I never actually know if she has finished cleaning or got stuck under the couch again. If I can query her status through a Shortcut or her manufacturer's API, Alfred could include in my morning briefing: "Juliana completed her cleaning cycle at 3 AM, Master Jo. All rooms covered, no incidents to report." Or more usefully: "Juliana appears to be stuck in the bedroom. She has not moved in 40 minutes. Shall I send a rescue party?"

The broader vision is for Alfred to be aware of my home the same way he is aware of my inbox. When I ask "is everything in order?", he should be able to answer with a status report covering emails, calendar, pending approvals, financial alerts, and whether the house has been cleaned. A proper butler would never limit his awareness to just the mail.

A Second Persona

My girlfriend has watched me use Alfred. This sparked an idea I had not considered: cloning Alfred's architecture for a second persona. The entire system is built on Clean Architecture with dependency injection, which means the persona, the rules, and the connected accounts are all configurable. The core infrastructure covering polling, classification, the action lifecycle, push notifications, and chat strategies is entirely provider-agnostic and user-agnostic.

In theory, creating a second instance means standing up another agent server pointed at different OAuth credentials, a different SQLite database, a different set of action rules, and a different system prompt. The persona would not be Alfred. She would get her own character, her own name, and her own way of speaking. But underneath, the same ChatService, the same ToolRegistry, the same AgentLoop, and the same strategy pattern would power everything.

The part that interests me most is how the persona shapes the experience. Alfred's butler character is not just flavour text. It affects how he delivers bad news ("I regret to inform you, Master Jo, that your credit card statement shows a rather generous dining budget this month"), how he prioritises information, and how he handles ambiguity. A different persona for a different person would need to match their communication style and preferences entirely. This is where the buildSystemPrompt() architecture pays off. The base capabilities and mode-specific instructions stay constant, while the persona layer is a separate, swappable block. Building a second agent is less about rewriting code and more about crafting a new character who happens to run on the same engine.

Conclusion

Building Alfred started as a weekend experiment: a polling loop that checked Gmail and labelled anything that looked important. What it became, over months of iteration, is something I did not fully anticipate: a personal operating system that sits between me and the noise of digital life.

The biggest lesson was not technical. It was architectural. Clean Architecture is not just an academic exercise you draw on whiteboards. It is the reason I was able to bolt on Microsoft Teams notifications, bank statement processing, and a full chat interface without rewriting the core. When your domain layer knows nothing about Gmail, adding Outlook is just another adapter. When your use cases speak in ports, swapping Claude Haiku for Sonnet is a one-line change in the composition root. The upfront cost of drawing those boundaries paid for itself ten times over.

That said, the path was not smooth. The jump from intent extraction to native tool use humbled me. Prompt engineering is not engineering in the traditional sense. There is no compiler to catch your mistakes, no type system to lean on. You ship a prompt, watch it hallucinate a tool name that does not exist, and go back to the drawing board. The multi-round reasoning loop took more iterations than any other feature, not because the code was complex, but because coaxing an LLM into reliable, structured behaviour across multiple turns is genuinely hard. Every fix revealed a new edge case. Every edge case demanded a new constraint in the system prompt. I have a much deeper respect now for anyone building production agentic systems.

The discovery that surprised me most was how naturally financial data fit into the system. I built Alfred to manage emails. The fact that bank statements arrive as email attachments meant the entire PDF extraction and transaction classification pipeline was, architecturally, just another use case plugged into the same ports. The backfill system, the hybrid classifier, the per-bank parser registry: none of it required changes to the core domain. That is Clean Architecture doing exactly what it promises.

Running everything on a Mac on my desk with a Cloudflare Tunnel was a deliberate choice. There is no monthly cloud bill. There is no cold start. My data never leaves my network unless I am the one requesting it through an encrypted tunnel. For a personal assistant that reads your email, knows your calendar, and processes your bank statements, that is not a nice-to-have. It is a requirement.

Alfred is far from finished. RAG-powered memory, WhatsApp integration, smart home control: the roadmap is long. But the foundation is solid. Every new capability I have added has reinforced the same pattern: define a port, write the use case, build the adapter, wire it in the composition root. The system grows without becoming fragile because each piece knows only what it needs to know.

If there is one thing I would tell someone starting a similar project, it is this: invest in the boundaries early. Not the features, not the UI, not the clever LLM tricks. The boundaries. Get the dependency direction right. Make your domain layer boring. Let your infrastructure layer be the only place that knows about the outside world. Everything else follows from that discipline. Alfred taught me that the most powerful personal software is not the one with the most features. It is the one you can keep evolving without fear of breaking what already works.

See you in the next one 😁

Increasing Technical Onboarding Velocity for Your Engineering Team

JOOJO DONTOH — Fri, 05 Dec 2025 06:16:26 +0000

TLDR

Problem: Engineers change teams frequently, and slow onboarding wastes everyone's time.

Goal: Minimise the time between git clone and first meaningful pull request.

Key practices:

Setup scripts that interactively guide new engineers through environment configuration
Code formatting tools (Prettier, ESLint) committed to the repo so standards are automatic
Brief READMEs focused on "how to run this" rather than business context
Descriptive file naming (transaction.service.ts, pos.client.ts) so the codebase is navigable
Comprehensive tests that serve as living documentation and give new engineers confidence to make changes
Hooks to catch issues before they reach code review
Protected branches and required approvals to prevent accidental mistakes
Common libraries and pipeline templates for multi-service teams

Result: New engineers get productive in days, reviews are shorter, and service owners spend less time hand-holding.

Trade-off: Requires upfront investment, but pays dividends with every new team member.

Introduction

Hello my people, its me again 😄. Today I want to talk about Engineering onboarding. So what is that? 🤔 In very simple terms, it is the journey between an engineer's first introduction to a codebase and the moment they can confidently open a meaningful, safe pull request. It's that critical window where confusion transforms into contribution.

The global job landscape for software engineers is highly dynamic and volatile. According to Zippia's analysis of over 100,000 software developer profiles, 69% of software engineers have a tenure of less than 2 years at their current job. At large tech companies, this number skews even shorter, with average tenures ranging from 1 to 3 years. The tech industry also carries one of the highest turnover rates across all industries, estimated at 13.2% according to LinkedIn workforce data. This reality means that more engineers than ever will find themselves in onboarding situations throughout their careers.

Onboarding isn't limited to new hires either. It happens when engineers switch teams internally, when a service gets transferred from one squad to another, or when engineering resources are borrowed temporarily for critical projects. Each of these scenarios demands the same thing: getting someone productive in an unfamiliar codebase as quickly as possible.

As an engineering lead, I've seen firsthand how a rough onboarding experience can slow down delivery, frustrate talented people, and introduce risk into production systems. This article aims to share practical strategies for making onboarding smooth and fast, while minimising the fear of new team members accidentally breaking things.

A quick note: onboarding involves non-technical aspects as well, such as team rituals, communication norms, and stakeholder relationships. Those matter deeply, but this article will focus specifically on the technical side of getting engineers productive and confident in your codebase.

Aspects of Knowledge a New Engineer Should Be Aware Of

Before an engineer can contribute meaningfully to a codebase, there are several knowledge areas they need to get up to speed on. Some of these are explicit and documented, others are tribal knowledge passed down through code reviews and hallway conversations.

Domain Knowledge

Understanding the business domain of the service you're working on isn't always a prerequisite for making changes. You can fix a bug in a fuel pricing service without fully understanding the intricacies of how pump prices are calculated. However, when it comes to adding features or making architectural decisions, domain knowledge becomes crucial for quality contributions and fewer review round trips.

Consider this example: an engineer is tasked with adding a "price override" feature to a convenience retail POS system. Without understanding the domain, they might implement it as a simple field that replaces the scanned price. But someone with domain knowledge would know that price overrides in retail need to account for manager approval workflows, audit trails for loss prevention, tax recalculations, loyalty point adjustments, and integration with the back-office reporting system. They'd also know that certain items like fuel, tobacco, and alcohol often have regulatory restrictions on price modifications. The engineer lacking this context might go through three or four review cycles before landing on the right approach, while someone with domain understanding gets it right the first time.

This knowledge transfer is typically handled through a buddy system where an assigned team member walks the new joiner through the current architecture at a high level. One important note here: keeping architecture diagrams up to date can feel like thankless work with no short-term rewards, but it pays dividends every time someone new joins the team.

Team Rituals

Stand-ups, sprint ceremonies, retrospectives, RCAs (Root Cause Analyses) and other team rituals are also part of onboarding. These won't be covered in this article since we're focusing on technical aspects, but they're worth mentioning for completeness.

Tech-Stack Familiarity

Tech-stack familiarity is usually filtered for during hiring or internal transfers. If you're hiring a backend engineer for a Java-based integration team, you're likely looking for candidates with Java or similar JVM experience. Knowledge of the stack naturally makes onboarding smoother.

That said, smooth onboarding practices become even more critical when tech-stack familiarity is low. If you've hired a strong engineer from a Python background into your Apache Camel and Spring Boot codebase, your onboarding process needs to carry more of the load.

Coding Standards

Every team develops conventions around how code should be written and organised. These include file naming standards, variable naming conventions, indentation preferences, and file structure patterns.

Some teams prefer their folder structure to mirror API endpoints. For example, in a retail integration service:

src/
├── api/
│   ├── v1/
│   │   ├── transactions/
│   │   │   ├── transactions.controller.ts
│   │   │   ├── transactions.service.ts
│   │   │   └── transactions.routes.ts
│   │   ├── inventory/
│   │   │   ├── inventory.controller.ts
│   │   │   ├── inventory.service.ts
│   │   │   └── inventory.routes.ts
│   │   └── fuel-prices/
│   │       ├── fuel-prices.controller.ts
│   │       ├── fuel-prices.service.ts
│   │       └── fuel-prices.routes.ts

With this structure, if a new engineer needs to work on the GET /api/v1/inventory endpoint that returns current tank dip readings, they immediately know to look in src/api/v1/inventory/. The cognitive load of navigating the codebase drops significantly.

Authentication, Environment Variables, and Secrets

This is often where onboarding gets frustrating. Different companies handle secrets and environment configuration in vastly different ways, and the friction here can make or break someone's first few days.

More mature organisations orchestrate access at an enterprise level using tools like ServiceNow, HashiCorp Vault, or AWS Secrets Manager, where permissions are tied to identity and granted automatically based on team membership. The less manual this process is, the better.

For teams without enterprise-grade tooling, here are some common approaches:

Symmetric encryption within the codebase: Some teams encrypt their .env files using a tool like git-crypt or sops and store them directly in the repository. New engineers just need the decryption password to access everything. This approach is convenient but carries risk since the password becomes a single point of compromise. A sensible mitigation is to only encrypt secrets for lower environments like dev and staging, keeping production secrets in a more secure system.

Encrypted files outside the codebase: Secrets are stored in a shared location (like an S3 bucket or internal file store) with company-wide access controls. Engineers with the right permissions can download what they need.

Manual sharing: The most primitive approach. Someone on the team carefully shares env files via secure channels. It works, but it doesn't scale and is prone to human error.

Whichever approach your team uses, the goal should be minimising the time between "I've cloned the repo" and "I have everything I need to run this locally."

Things Needed to Ensure Fast and Clean Onboarding

This section covers the practical tooling and processes that make onboarding frictionless. I've split it into two parts: getting engineers set up quickly (code pickup), and enabling them to make changes safely (change integration).

Part 1: Smooth Code Pickup

The goal here is simple: minimise the time between git clone and "I have a working local environment."

Code Formatting Standardisation

Inconsistent code formatting creates unnecessary noise in pull requests and wastes mental energy. A new engineer shouldn't have to guess whether to use tabs or spaces, or whether to add trailing commas.

Prettier is one of the most popular tools for solving this. Commit a .prettierrc file to your repository and every engineer's code gets formatted the same way:

{
  "semi": true,
  "singleQuote": true,
  "tabWidth": 2,
  "trailingComma": "es5",
  "printWidth": 100
}

Don't forget a .prettierignore file to prevent formatting generated files or dependencies:

node_modules/
dist/
coverage/
*.generated.ts

When a new engineer opens the codebase, these config files immediately communicate the team's standards without anyone needing to explain them.

Handy Scripts

Instead of a README that says "run npm install, then set up your .env file, then run docker-compose up, then..." wrap all of this in scripts. Scripts are executable documentation.

Setup Script

A setup script handles dependency installation and environment preparation. Here's an example for a retail POS integration service:

#!/bin/bash
# setup.sh - Interactive setup script for POS Integration Service

set -e

echo "🚀 Setting up POS Integration Service..."

# Check Node version
REQUIRED_NODE_VERSION="18"
CURRENT_NODE_VERSION=$(node -v | cut -d'v' -f2 | cut -d'.' -f1)

if [ "$CURRENT_NODE_VERSION" -lt "$REQUIRED_NODE_VERSION" ]; then
    echo "❌ Node.js version $REQUIRED_NODE_VERSION or higher is required."
    echo "   Current version: $(node -v)"
    echo "   Install via: https://nodejs.org/ or use nvm"
    exit 1
fi
echo "✅ Node.js version: $(node -v)"

# Check for .env file
if [ ! -f .env ]; then
    echo ""
    echo "📋 No .env file found."
    echo "   Would you like to:"
    echo "   1) Copy from .env.example (recommended for new setup)"
    echo "   2) Decrypt from .env.encrypted (requires team password)"
    echo "   3) Skip (I'll set it up manually)"
    read -p "   Enter choice [1-3]: " env_choice

    case $env_choice in
        1)
            cp .env.example .env
            echo "✅ Created .env from .env.example"
            echo "   ⚠️  Remember to update placeholder values"
            ;;
        2)
            read -s -p "   Enter decryption password: " password
            echo ""
            openssl aes-256-cbc -d -in .env.encrypted -out .env -pass pass:"$password"
            echo "✅ Decrypted .env file"
            ;;
        3)
            echo "⏭️  Skipping .env setup"
            ;;
    esac
else
    echo "✅ .env file exists"
fi

# Validate critical env vars
if [ -f .env ]; then
    source .env
    MISSING_VARS=()

    [ -z "$POS_API_BASE_URL" ] && MISSING_VARS+=("POS_API_BASE_URL")
    [ -z "$AZURE_SERVICE_BUS_CONNECTION" ] && MISSING_VARS+=("AZURE_SERVICE_BUS_CONNECTION")
    [ -z "$S3_BUCKET_NAME" ] && MISSING_VARS+=("S3_BUCKET_NAME")

    if [ ${#MISSING_VARS[@]} -gt 0 ]; then
        echo "⚠️  Missing required environment variables:"
        for var in "${MISSING_VARS[@]}"; do
            echo "   - $var"
        done
    else
        echo "✅ All required environment variables are set"
    fi
fi

# Install dependencies
echo ""
echo "📦 Installing dependencies..."
npm ci
echo "✅ Dependencies installed"

echo ""
echo "🎉 Setup complete! Run './start.sh' to start the service."

Notice how the script is interactive. It doesn't just fail silently when something is missing. It guides the engineer through decisions and helps them understand what the system needs. This is far more educational than a wall of README text.

Startup Script

A startup script gets the application running locally. It should aim to be system-agnostic by leveraging containers where possible:

#!/bin/bash
# start.sh - Start the POS Integration Service locally

set -e

echo "🚀 Starting POS Integration Service..."

# Check if setup has been run
if [ ! -d "node_modules" ]; then
    echo "❌ Dependencies not installed. Run './setup.sh' first."
    exit 1
fi

# Start local dependencies (mocked external services)
echo "📦 Starting local dependencies..."
docker-compose up -d localstack mockpos

# Wait for dependencies to be healthy
echo "⏳ Waiting for dependencies..."
sleep 5

# Start the service
echo "🏃 Starting service in development mode..."
npm run dev

Some teams combine setup and startup into a single script. Others keep them separate so you don't re-run setup every time you want to start the service. Either approach works as long as it's consistent.

example output from another script:

Functionality Test Script (Optional)

For extra confidence, you can provide a script that runs a quick smoke test against the locally running service:

#!/bin/bash
# smoke-test.sh - Verify the service is running correctly

BASE_URL="http://localhost:3000"

echo "🧪 Running smoke tests..."

# Health check
HEALTH=$(curl -s -o /dev/null -w "%{http_code}" "$BASE_URL/health")
if [ "$HEALTH" -eq 200 ]; then
    echo "✅ Health endpoint responding"
else
    echo "❌ Health check failed (HTTP $HEALTH)"
    exit 1
fi

# Test transaction endpoint with mock data
RESPONSE=$(curl -s -X POST "$BASE_URL/api/v1/transactions" \
    -H "Content-Type: application/json" \
    -d '{"storeId": "TEST001", "items": [{"sku": "MOCK123", "quantity": 1}]}')

if echo "$RESPONSE" | grep -q "transactionId"; then
    echo "✅ Transaction endpoint working"
else
    echo "❌ Transaction endpoint failed"
    exit 1
fi

echo ""
echo "🎉 All smoke tests passed!"

Brief but Useful README

People don't read long READMEs. Keep yours focused on operability rather than explaining the business problem the service solves. Save that for Confluence or your internal docs.

A good README structure:

# POS Integration Service

Handles transaction processing between store POS systems and central data lake.

## Quick Start

bash
./setup.sh # First time only
./start.sh # Start the service


Service runs at `http://localhost:3000`

## Useful Commands

- `npm run test` - Run unit tests
- `npm run test:integration` - Run integration tests
- `./smoke-test.sh` - Verify local setup works

## Documentation

- [Architecture Diagram](https://confluence.internal/pos-integration/architecture)
- [API Specification](https://confluence.internal/pos-integration/api-spec)
- [Runbook](https://confluence.internal/pos-integration/runbook)

That's it. A new engineer can get running in under a minute and knows where to find deeper documentation when needed.

Part 2: Smooth Change Integration

Once an engineer is set up locally, the next challenge is enabling them to make changes confidently without breaking things.

Descriptive File and Function Naming

Clear naming conventions reduce the learning curve dramatically. When files are named descriptively, new engineers can navigate the codebase intuitively.

Consider a retail integration service with these common file patterns:

src/
├── clients/
│   ├── pos.client.ts           # Handles POS API communication
│   ├── serviceBus.client.ts    # Azure Service Bus operations
│   └── s3.client.ts            # S3 storage operations
├── services/
│   ├── transaction.service.ts  # Transaction business logic
│   └── inventory.service.ts    # Inventory business logic
├── utils/
│   ├── date.utils.ts           # Date formatting helpers
│   └── validation.utils.ts     # Input validation helpers
└── builders/
    └── transaction.builder.ts  # Builds transaction payloads

If a new engineer needs to modify how transactions are sent to S3, they know to look in s3.client.ts. If they need to change business logic, they check the services folder. The naming convention acts as a map.

Treat these as principles rather than rigid rules. The goal is descriptive, predictable naming that helps people find what they need.

Unit Tests

All those clients, utils, helpers, and services should have accompanying tests. When a new team member modifies transaction.service.ts, they can run the tests to verify they haven't broken existing functionality:

// transaction.service.test.ts
describe('TransactionService', () => {
  describe('processTransaction', () => {
    it('should calculate correct total for multiple items', () => {
      const items = [
        { sku: 'FUEL001', quantity: 45.5, unitPrice: 1.89 },
        { sku: 'SNACK001', quantity: 2, unitPrice: 3.50 }
      ];

      const result = transactionService.processTransaction(items);

      expect(result.total).toBe(92.995);
    });

    it('should apply fuel discount for loyalty members', () => {
      const items = [{ sku: 'FUEL001', quantity: 40, unitPrice: 1.89 }];
      const loyaltyId = 'LOYALTY123';

      const result = transactionService.processTransaction(items, loyaltyId);

      expect(result.fuelDiscount).toBe(0.10);
      expect(result.total).toBe(71.60);
    });
  });
});

Tests serve as living documentation. A new engineer can read the test file to understand what a function is supposed to do without digging through implementation details.

Pre-commit and Pre-push Hooks

Git hooks catch issues before they reach the remote repository. Tools like Husky make this easy to set up:

// package.json
{
  "husky": {
    "hooks": {
      "pre-commit": "npm run lint && npm run format:check",
      "pre-push": "npm run test"
    }
  }
}

A typical setup runs linting and format checks on commit (fast feedback), and runs tests before push (thorough validation).

One word of caution: keep these hooks fast. If your pre-commit takes 30 seconds, engineers will start bypassing it with --no-verify. Aim for under 5 seconds on pre-commit.

Common Libraries (For Multi-Service Teams)

When your team owns multiple services, you'll notice patterns emerging. The same S3 client code, the same transaction builder, the same logging setup. Instead of copy-pasting across repositories, extract these into a shared library.

// @my-org/retail-common
import { S3Client } from '@my-org/retail-common';
import { TransactionBuilder } from '@my-org/retail-common';

const s3 = new S3Client({ bucket: process.env.S3_BUCKET });
const transaction = new TransactionBuilder()
  .withStore('STORE001')
  .withItems(items)
  .build();

A few guidelines for common libraries:

Use semantic versioning with alpha/beta releases so teams can test changes before they go stable
Write rigorous tests. A bug in a common library affects every consuming service
Document breaking changes clearly in your changelog

Pipeline Repositories (For Multi-Service Teams)

GitHub Actions, GitLab CI, and Azure Pipelines all support reusable workflow definitions. Instead of duplicating deployment logic across repositories, centralise it:

# In your service repository
jobs:
  deploy:
    uses: my-org/pipeline-templates/.github/workflows/deploy-to-aws.yml@v2
    with:
      environment: staging
      service-name: pos-integration

When deployment processes change, you update one repository instead of twenty.

Branching Strategies and Policies

Protect your main branch from direct commits. This is non-negotiable for team safety. Configure your repository to require:

Pull request reviews (at least one approver)
Passing CI checks before merge
No force pushes to main

This protects new engineers from accidentally pushing directly to production. The guardrails are there before they even make their first commit.

Environment Strategies and Policies

Development environments should be open for experimentation. Engineers need a place to break things safely.

Staging and UAT environments should mirror production as closely as possible, with stricter deployment controls:

# azure-pipelines.yml
stages:
  - stage: DeployDev
    condition: eq(variables['Build.SourceBranch'], 'refs/heads/develop')
    jobs:
      - deployment: DeployToDev
        environment: development  # Auto-deploys, no approval needed

  - stage: DeployStaging
    condition: eq(variables['Build.SourceBranch'], 'refs/heads/main')
    jobs:
      - deployment: DeployToStaging
        environment: staging  # Requires manual approval
        strategy:
          runOnce:
            deploy:
              steps:
                - script: ./deploy.sh staging

This ensures that code can flow freely to dev, but staging deployments require explicit approval.

Integration Tests

Unit tests verify individual components. Integration tests verify that services work together correctly.

For a retail integration service, an integration test might verify that a transaction flows correctly from the POS mock through your service and into S3:

describe('Transaction Flow Integration', () => {
  it('should process POS transaction and store in S3', async () => {
    // Arrange
    const mockTransaction = createMockPOSTransaction();

    // Act
    await posClient.sendTransaction(mockTransaction);
    await waitForProcessing(5000);

    // Assert
    const storedTransaction = await s3Client.getTransaction(mockTransaction.id);
    expect(storedTransaction).toBeDefined();
    expect(storedTransaction.status).toBe('PROCESSED');
  });
});

Integration tests give new engineers confidence that their changes haven't broken contracts with other systems.

Culture of Maintenance

Finally, build a culture that keeps quality high as new engineers join:

Code coverage thresholds: Configure your pipeline to fail if coverage drops below a threshold. This ensures new code comes with tests:

# jest.config.js
coverageThreshold: {
  global: {
    branches: 80,
    functions: 80,
    lines: 80,
    statements: 80
  }
}

Integration tests as part of feature work: A feature isn't done until its integration tests are written. Make this explicit in your definition of done.

Strict but fair reviews: Code reviews should enforce standards consistently, but reviewers should also be helpful and educational. A review that just says "wrong" teaches nothing. A review that explains why something should change helps engineers grow.

Advantages and How This Helps My Teams

All the upfront investment in scripts, tests, and automation pays dividends quickly. Here's what I've seen in practice.

Saves Time for Onboarded Members

New engineers on my team don't spend their first day wrestling with environment setup or hunting down secrets. They clone the repo, run ./setup.sh, and follow the interactive prompts. Within an hour, they have a working local environment and can start exploring the codebase.

Compare this to the alternative: a new joiner pinging five different people on Slack asking where to find the database credentials, discovering their Node version is wrong after hitting a cryptic error, and spending half a day just getting the service to start. That frustration compounds and sets a negative tone for the entire onboarding experience.

Saves Time for Service Owners

Before I invested in these practices, onboarding a new engineer meant hours of hand-holding. "Where's the config for X?" "How does Y work?" "Why is Z failing?"

Now, when someone asks me a question, I can often point them to a specific file or test. "Check transaction.service.test.ts, the third test case covers exactly that scenario." The tests become documentation. The scripts become guides. I'm not the bottleneck anymore.

This is especially valuable when you're leading a team and your time is split across architecture decisions, stakeholder meetings, and code reviews. Every hour saved on repetitive explanations is an hour you can spend on higher-leverage work.

Reduces Time in Change Management and Knowledge Transfer

When an engineer leaves the team or moves to another project, the knowledge transfer burden is significantly lighter. The important patterns are encoded in common libraries. The deployment process is captured in pipeline templates. The business logic is documented through tests.

New engineers inheriting a service don't need a week of shadowing sessions. The codebase is largely self-explanatory.

Reviews Are Short and Sweet

This one might be my favourite. When automated checks handle formatting, linting, test coverage, and integration verification, code reviews can focus on what actually matters: logic, architecture, and edge cases.

I no longer leave comments like "missing semicolon" or "incorrect indentation." Prettier handles that. I don't have to verify that tests exist. The coverage threshold enforces it. The review becomes a conversation about the change itself rather than a checklist of mechanical issues.

Pull requests that used to require three rounds of back-and-forth now get approved on the first or second pass.

Disadvantages

No approach is without trade-offs. Here are the downsides I've encountered and some honest reflections on them.

A Lot of Initial Work

Setting up robust scripts, configuring pipelines, writing comprehensive tests, and building common libraries takes time. Time that could otherwise go toward feature delivery.

I personally don't find this burdensome because I've seen the compounding benefits across multiple teams. But I understand the hesitation. When you're under pressure to ship a fuel pricing integration before the end of the quarter, spending two days writing a setup script feels like a luxury you can't afford.

The reality is that this investment is easier to justify on greenfield projects or during quieter periods. Retrofitting these practices onto a legacy codebase with looming deadlines is genuinely difficult. Sometimes you have to be pragmatic and introduce improvements incrementally rather than all at once.

Can Cause Friction in Delivery If Not Managed Well

Standards are helpful until they become obstacles. If your automated checks are too strict or too slow, they start blocking legitimate work.

Consider this scenario: your team has a rule that every pull request must have 90% code coverage. An engineer is fixing a critical bug in the loyalty points calculation that's causing customers to lose discounts at checkout. The fix is two lines, but to satisfy the coverage requirement, they'd need to write fifteen new tests for an untested legacy function they happened to touch. The bug sits in production for an extra day while they write tests for unrelated code.

Another example: you've established a convention that all API responses must follow a specific format. But the convention lives only in a Confluence page that nobody reads. Without automated schema validation, engineers keep forgetting. Reviews become tedious nitpicking sessions, and resentment builds. "Why did my PR get blocked for a formatting issue when the last three PRs got merged without it?"

The lesson here is that standards need automated enforcement to be sustainable. If it can't be checked by a machine, it will eventually be ignored by humans.

OS Compatibility Issues

Scripts written on macOS often break on Windows, and vice versa. This is a constant source of friction for teams with mixed development environments.

A simple example:

#!/bin/bash
# Works on macOS and Linux, fails on Windows

# macOS sed syntax
sed -i '' 's/old/new/g' config.json

# Linux sed syntax (different from macOS)
sed -i 's/old/new/g' config.json

Or path handling:

# Unix-style paths
CONFIG_PATH="./config/local/settings.json"

# Windows needs backslashes (or Git Bash to translate)
CONFIG_PATH=".\config\local\settings.json"

Mitigation strategies include using cross-platform tools like Node.js scripts instead of bash, containerising your development environment with Docker, or maintaining separate scripts for different platforms. None of these are perfect, but they reduce the pain.

Script Sprawl

When you start automating everything, you can end up with a dozen scripts scattered across your repository:

scripts/
├── setup.sh
├── setup-windows.ps1
├── start.sh
├── start-docker.sh
├── run-tests.sh
├── run-integration-tests.sh
├── deploy-dev.sh
├── deploy-staging.sh
├── generate-mocks.sh
├── update-snapshots.sh
├── clean.sh
└── seed-database.sh

A new engineer clones the repo and has no idea which script to run first. "Do I run setup.sh or start.sh? What's the difference between start.sh and start-docker.sh?"

The solution is consolidation and documentation. Consider a single entry point script with subcommands:

./run.sh setup      # First-time setup
./run.sh start      # Start the service
./run.sh test       # Run unit tests
./run.sh test:int   # Run integration tests

Or use a Makefile, which is language-agnostic and self-documenting:

.PHONY: help setup start test

help:
    @echo "Available commands:"
    @echo "  make setup    - First-time environment setup"
    @echo "  make start    - Start the service locally"
    @echo "  make test     - Run unit tests"
    @echo "  make test-int - Run integration tests"

setup:
    ./scripts/setup.sh

start:
    docker-compose up -d
    npm run dev

test:
    npm run test

Running make help gives engineers a clear menu of options. No more guessing.

Conclusion

Engineering onboarding isn't a one-time event. With average tenures shrinking and teams constantly evolving, it's a recurring challenge that deserves intentional investment.

The practices outlined in this article aren't revolutionary. Setup scripts, automated formatting, comprehensive tests, and protected branches are all well-established ideas. The difference lies in treating them as a cohesive system rather than isolated improvements. Each piece reinforces the others. Scripts reduce setup friction. Tests enable confident changes. Automation shortens reviews. Together, they create an environment where a new engineer can go from git clone to meaningful pull request in days rather than weeks.

The upfront cost is real. Writing that first setup script takes time you could spend on features. Configuring pipeline templates isn't glamorous work. But every engineer who joins your team after that benefits. The investment amortises quickly, and the compound returns are substantial.

Start where you are. If your team has none of these practices, don't try to implement everything at once. Pick one pain point. Maybe it's the two hours new joiners spend setting up their environment. Write a setup script. Maybe it's the endless formatting debates in code reviews. Add Prettier. Small improvements stack up.

Your future team members will thank you. And honestly, so will your future self the next time you have to onboard someone new.

This article is formatted and grammatically enhance with AI.

Your Integration Layer is Probably Over-Engineered (Let's Fix It with Camel DSL)

JOOJO DONTOH — Tue, 28 Oct 2025 15:01:17 +0000

Introduction

Hi guys it's me again 😁. If you've ever wished you could describe complex integration flows in plain, readable language rather than wrestling with boilerplate code, you're going to love DSLs. A Domain-Specific Language (DSL) is essentially a specialized mini-language designed for a particular task. Think of it as a shorthand that lets you express what you want to do without getting lost in the how. Easy peasy

Apache Camel embraces this philosophy wholeheartedly and offers developers multiple flavors of DSLs to choose from. Whether you're a fan of the Java DSL with its fluent builder style, prefer the structured clarity of XML DSL in Camel XML files, or lean toward Spring XML for classic Spring configurations, there's something literally for everyone. You can define routes using YAML DSL for a clean, human-readable format, build RESTful services with Rest DSL (including contract-first approaches with OpenAPI specs), or even keep things annotation-based with the Annotation DSL right in your Java beans.

In this article, we're narrowing down on the YAML DSL. It's a lightweight, intuitive way to define integration routes that feels more like writing a configuration file than coding. It follows the declarative way of engineering. We'll explore how to define routes, configure endpoints, and wire up beans, all while keeping our setup refreshingly simple. If you want to dive deeper into the technical details, the official Camel YAML DSL documentation is a great resource. But for now, let's keep things practical and see what makes YAML DSL such a joy to work with.

What Are Enterprise Integration Patterns?

Before we dive into Camel DSL, let's talk about the "why" behind it all. Enterprise Integration Patterns (EIPs) are tried-and-true solutions to common problems that pop up when you're connecting different systems. These are the plumbing of most of the systems you use. Think of them as design patterns, but specifically for the messy world of enterprise integration where you're constantly moving data between APIs, databases, message queues, and legacy systems that were never meant to talk to each other.

The concept was popularized by Gregor Hohpe and Bobby Woolf's seminal book, Enterprise Integration Patterns: Designing, Building, and Deploying Messaging Solutions. Published in 2003, this book catalogued around 65 patterns that have since become the lingua franca for integration architects. Apache Camel was built from the ground up to implement these patterns, making them accessible through its DSL rather than forcing you to reinvent the wheel every time.

Here are five of the most common EIPs you'll encounter:

1. Content-Based Router

Routes messages to different destinations based on their content. For example, you might route orders to different processing queues based on their total amount—high-value orders go to manual review, while smaller ones get auto-processed. Read more here

2. Message Filter

Acts as a gatekeeper, allowing only messages that meet certain criteria to pass through. Think of it as a bouncer for your data—only messages with valid formats or specific properties get through. Read more here

3. Splitter

Takes a single message containing multiple items (like a batch of orders) and breaks it into individual messages. This is handy when you need to process each item independently or send them to different endpoints.Read more here

4. Aggregator

The opposite of a splitter—it combines multiple related messages into a single cohesive message. Perfect for scenarios where you're collecting responses from multiple services before sending a unified reply.
Read more here

5. Dead Letter Channel

Your safety net for when things go wrong. Messages that fail processing get routed to a special "dead letter" queue where you can inspect them, fix issues, and potentially reprocess them later.
Read more here

You can explore the full catalog of patterns in the Apache Camel EIP documentation, but these five alone will cover a huge portion of your integration needs.

What is Apache Camel DSL?

A Brief History

Apache Camel didn't appear out of nowhere. It was born from a real need to implement those Enterprise Integration Patterns we just discussed. Here's how it evolved:

2003 marked the beginning when Gregor Hohpe and Bobby Woolf's Enterprise Integration Patterns book was published, laying the groundwork for what would become Apache Camel's DNA.

June 27, 2007 saw Camel's initial release. From day one, the DSL was baked into its core design philosophy. The goal was simple: make it ridiculously easy to describe integration flows without drowning in boilerplate code.

In those early years, Camel established itself as the go-to framework for implementing EIPs in Java. The DSL wasn't just an afterthought, it was the way you worked with Camel, turning complex integration logic into readable, maintainable code.

As adoption grew, so did the DSL. It evolved beyond Java, expanding to support XML, YAML, Groovy, and other syntaxes. This gave developers the freedom to pick the language that fit their team's style and existing infrastructure.

2014 (Camel 2.14) brought a game-changer: extensions to the routing DSL specifically for REST endpoints. Suddenly, building RESTful integrations became as straightforward as defining any other route.

Today, ongoing development keeps the DSL at the forefront of Camel's priorities. The community continuously refines it, making routes easier to write, test, and maintain. New features and improvements roll out regularly, keeping pace with modern integration challenges.

How Does Camel YAML DSL Work?

From YAML to Running Integration

Here's where things get really interesting. You write a simple YAML file declaring your integration patterns, and somehow it becomes a running Java application. But here's the key distinction: this isn't compilation—it's interpretation.

When you run a Camel YAML route, you're not generating Java source code, compiling it, and then executing bytecode. Instead, Camel reads your YAML file at runtime, parses it, and dynamically constructs the integration flow in memory. Think of it like the difference between translating a book (compilation) versus having a real-time interpreter translate as you speak (interpretation). The YAML DSL is interpreted into Camel's internal routing model on the fly.

Enter JBang

So how do we actually run these YAML files? Meet JBang—a tool that lets you run Java applications with zero ceremony. No project setup, no build files, no IDE required. Just a simple command and you're off to the races.

JBang is essentially a launcher and script runner for Java. It can download dependencies, manage classpaths, and execute Java code—all from a single command. For Camel, this means you can run integration routes as easily as running a Python script.

The Interpretation Process

When you execute jbang camel@apache/camel run route.yaml, here's what happens under the hood:

JBang Downloads & Starts: JBang fetches the Camel catalog if it's not already cached and initializes the runtime environment.
File Detection: JBang detects that you've provided a YAML file and determines which parser to use.
YAML Parsing: The YAML content is parsed into a structured format that can be processed.
Convert to Camel Model: The parsed YAML is transformed into Camel's internal route model—essentially Java objects that represent your integration flow.
Component Resolution: Camel identifies which components you're using (HTTP, Kafka, databases, etc.) and loads them if needed.
Endpoint Creation: Based on your route definitions, Camel creates endpoint instances that represent the actual connections to external systems.
Processor Pipeline Creation: Any transformations, filters, or logic in your route are assembled into a processing pipeline.
Consumer Creation: Camel sets up consumers that will trigger your route (like an HTTP listener or a file watcher).
Route Registration & Start: Finally, the complete route is registered with Camel's context and started, ready to process messages.

All of this happens in seconds, and you're left with a fully functional integration running in a JVM.

For those curious about the inner workings, the Apache Camel GitHub repository is a treasure trove of information.

Hands-On: Building a Weather Service

Let's get practical. We'll build a simple REST service that fetches weather information for a given country. This example will demonstrate route definition, endpoint definition, and how Camel handles external API calls.

Step 1: Install JBang

First, install JBang. On macOS or Linux:

curl -Ls https://sh.jbang.dev | bash -s - app setup

On Windows (using PowerShell):

iex "& { $(iwr -useb https://ps.jbang.dev) } app setup"

Verify the installation:

jbang version

Step 2: Create Your Route

Create a file called weather-service.yaml:

- route:
    id: weather-api-route
    from:
      uri: "platform-http:/weather"
      parameters:
        httpMethodRestrict: "GET"
      steps:
        - setProperty:
            name: country
            simple: "${header.country}"

        - choice:
            when:
              - simple: "${header.country} == null"
                steps:
                  - setHeader:
                      name: Content-Type
                      constant: application/json
                  - setBody:
                      constant: '{"error": "Country parameter is required"}'
                  - setHeader:
                      name: CamelHttpResponseCode
                      constant: "400"
                  - stop: {}

        - removeHeaders:
            pattern: "*"

        - toD:
            uri: "https://wttr.in/${exchangeProperty.country}"
            parameters:
              format: "j1"
              bridgeEndpoint: "true"

        - unmarshal:
            json:
              library: Jackson

        - setBody:
            simple: "The weather in ${exchangeProperty.country} is ${body[current_condition][0][temp_C]} degrees Celsius with ${body[current_condition][0][weatherDesc][0][value]}"

        - setHeader:
            name: Content-Type
            constant: text/plain

Let me break down what's happening here:

Route Definition: We define a route with ID weather-api-route that starts from an HTTP endpoint.

Endpoint Definition: The platform-http:/weather endpoint creates an HTTP server listening on the /weather path, restricted to GET requests only.

Processing Steps:

We extract the country parameter from the request header and store it as an exchange property
We validate that the country parameter exists, returning a 400 error if it doesn't
We remove all incoming HTTP headers with removeHeaders to prevent them from interfering with our external API call
We use toD (dynamic to) to call the wttr.in weather API with the country variable. The D in toD means it evaluates expressions at runtime
We set bridgeEndpoint: "true" to properly bridge the HTTP connection between our endpoint and the external API
We unmarshal the JSON response using Jackson
We transform it into a readable sentence using Camel's Simple language
We set the response content type to plain text

Important Notes:

We use toD instead of to because we need to evaluate the ${exchangeProperty.country} expression dynamically at runtime

The removeHeaders step is crucial - without it, headers from the incoming request would be passed to the external API, causing errors

In YAML DSL, query parameters must be in the parameters section, not embedded in the URI

Step 3: Run Your Route

Execute your route with JBang:

jbang camel@apache/camel run weather-service.yaml

You'll see Camel start up, and it will tell you which port it's listening on (typically 8080). The first startup might take a bit longer as JBang downloads dependencies.

Step 4: Test It Out

Open another terminal and try it:

curl "http://localhost:8080/weather" -H "country: London"

You should get a response like:

The weather in London is 15 degrees Celsius with Partly cloudy

Try different cities:

curl "http://localhost:8080/weather" -H "country: Tokyo"
curl "http://localhost:8080/weather" -H "country: NewYork"
curl "http://localhost:8080/weather" -H "country: Paris"

Test the error handling:

curl "http://localhost:8080/weather"

This should return:

{"error": "Country parameter is required"}

Note on Response Times: You might notice requests take 5-15 seconds to complete. This isn't your Camel route being slow—it's waiting on wttr.in, a free weather service that aggregates data in real-time. If you test the API directly (curl "https://wttr.in/London?format=j1"), you'll see it takes about the same time. Your Camel route itself is fast; it's just waiting on the external dependency. In production, you'd typically use a faster commercial weather API or implement caching to improve response times.

Step 5: Visualize with Karavan

Want to see your route visually? Install the Karavan extension in VS Code:

Open VS Code
Go to Extensions (Ctrl+Shift+X or Cmd+Shift+X on Mac)
Search for "Karavan"
Install the "Karavan" extension by Apache Software Foundation

Once installed:

Right-click on your weather-service.yaml file in VS Code
Select "Karavan: Open" from the context menu
You'll see a visual representation of your route with all the steps laid out graphically

The Karavan designer shows your route as a flowchart—from the HTTP endpoint through validation, header removal, the external API call, JSON unmarshalling, transformation, and finally the response. It's incredibly helpful for:

Understanding complex routes at a glance
Debugging integration flows
Communicating integration logic to non-technical stakeholders
Editing routes visually instead of writing YAML by hand

You can even use Karavan to build routes from scratch by dragging and dropping components, and it will generate the YAML for you!

Why Use Camel YAML DSL? Real-World Advantages

Rapid Project Spinning with Proven Patterns

Camel YAML DSL gives you instant access to 65+ pre-built Enterprise Integration Patterns. Need a content-based router, message filter, or splitter? It's just a few lines of YAML. What typically takes weeks to build from scratch—complete with error handling and retry logic—becomes an afternoon of configuration. You're describing what you want, and Camel handles the how.

MVPs and Proof of Concepts at Lightning Speed

For startups and innovation teams, speed matters. Build a functioning integration layer in hours, not weeks. Connect to Stripe, send Twilio notifications, sync to your CRM—all achievable in a single afternoon. When business requirements change (and they will), your routes change with them. No massive refactoring, just edit YAML and redeploy.

Democratizing Integration Development

You don't need to be a Java expert to build integrations anymore. JBang and YAML DSL lower the barrier dramatically. Business analysts, product managers, or junior engineers with basic YAML knowledge can create and modify routes. This shifts senior engineers from implementing routine integrations to building reusable components and solving genuinely complex problems.

Centralizing Integration Logic Across Teams

Companies struggle with integration sprawl—every team building their own connectors and reinventing the same patterns. Camel DSL solves this. A platform team maintains common components and patterns, while product teams compose them as needed. Want all APIs to use exponential backoff? Configure it once. Need centralized logging? Build it into your base template. One update benefits every route. This is a personal favorite of mine

Building Integration Platforms on Kubernetes

Here's where Camel truly shines at scale. Companies build entire integration platforms using this pattern:

The Setup: Multiple Kubernetes clusters divided into namespaces (workspaces), each owned by a team. Each integration runs as a pod within these namespaces.

Smart Routing: Route traffic between namespaces or clusters using Kubernetes networking. A payment request flows through payment → order → notification namespaces seamlessly. Add content-based routing to direct European orders to EU clusters for data residency.

Security: Kubernetes secrets hold API keys, passwords, and tokens. Each pod only accesses its required secrets—payment pods can't see shipping credentials, notification pods can't access payment gateways.

Team Autonomy: Teams deploy their integrations independently. They write YAML routes, push to Git, and CI/CD handles the rest. Teams set their own resource limits, autoscaling rules, and health checks while following platform-level guardrails.

The Payoff: Self-service integration platform. Fast team velocity. Enforced security and governance. Kubernetes handles scaling. Clean, readable YAML that stakeholders understand. Enterprises already use this pattern to manage thousands of integrations across distributed teams.

The Other Side: When DSLs Fall Short

High Abstraction Hides Complexity

YAML DSL sits at a very high level of abstraction. You declare what you want, and Camel figures out how to make it happen. This is powerful—until something breaks or behaves unexpectedly.

Debugging becomes detective work. Your YAML looks correct, but the route isn't working. Is it a timing issue? A header not being set? A component default you didn't know about? Suddenly you need to understand:

How Camel interprets and executes routes
Underlying component implementations
Message exchange patterns and contexts
How properties and headers flow through pipelines

The abstraction that made development fast now makes troubleshooting slow. You end up diving into Camel documentation, examining verbose logs, and sometimes reading Java source code to understand what's really happening under your simple YAML declarations.

It's Java All the Way Down

Here's the uncomfortable truth: Camel is a Java framework. When things get complex or custom, you're writing Java code.

Need a transformation more complex than Simple language supports? Write Java. Want a custom component or specialized error handling? Java. Need to debug a route deadlocking under load? Better understand Java concurrency, exception handling, and threading models.

This creates a skills gap. You might have team members comfortable with YAML who hit a wall when they need to drop into Java. Or you have Java experts who view the DSL as unnecessary abstraction. Either way, the promise of "just write YAML" only holds for straightforward integrations. Anything non-trivial requires Java expertise.

Organizational Startup Costs

Building an integration platform around Camel DSL isn't trivial. There's real investment required:

Learning Curve: Teams need to learn YAML syntax, Camel concepts, EIP patterns, component configurations, and debugging techniques. This takes time—expect weeks or months before teams are productive.

Infrastructure Setup: You need CI/CD pipelines, container registries, Kubernetes clusters (if going that route), monitoring systems, and logging infrastructure. Setting this up properly takes effort.

Governance and Standards: Someone needs to establish coding standards, route templates, security policies, and deployment procedures. Without this, you'll end up with inconsistent routes that are hard to maintain.

Cultural Change: Non-engineers writing integrations sounds great in theory, but requires buy-in. Engineers might resist sharing control. Business analysts might not want the responsibility. Establishing this new way of working takes organizational effort.

The payoff is significant, but don't underestimate the upfront investment needed to make it work.

Configuration Has Its Limits

YAML DSL is fantastic for standard patterns, but sometimes you need something unique. Maybe you're integrating with a legacy system that has quirky authentication. Or you need custom data transformation logic that's too complex for Simple language. Perhaps you're dealing with binary protocols that don't fit Camel's HTTP-centric model.

When configuration isn't enough, you have to write custom Java components or processors. Now you're maintaining both YAML routes and Java code, which defeats some of the simplicity. You also create a two-tier system: simple integrations anyone can handle, and complex ones that require Java developers.

The line between "configurable" and "needs code" isn't always clear until you're deep into implementation. What starts as "just a simple route" can evolve into something requiring custom Java, at which point you might wonder if starting with code would have been cleaner.

Wrapping Up

Apache Camel YAML DSL represents a pragmatic approach to enterprise integration. It takes decades of integration patterns, wraps them in readable YAML, and makes them accessible to a broader audience than traditional Java frameworks ever could.

Is it perfect? No. You're trading explicit control for declarative simplicity, and that tradeoff won't suit every team or every project. But for organizations drowning in integration complexity, or startups needing to move fast, or platform teams trying to democratize development—it's a compelling option.

The key is knowing when to reach for it. Building a weather API demo? Perfect. Connecting your e-commerce platform to payment gateways, inventory systems, and shipping providers? Absolutely. Implementing something so custom that you're fighting the framework every step of the way? Maybe reconsider.

Start small. Spin up JBang, write a simple route, see how it feels. You might find that what once took your team weeks now takes hours, and that's worth paying attention to. The patterns are proven, the tooling is mature, and the community is active. Give it a shot—your future self (and your integration backlog) might thank you. See you in the next one

Skip the Database: Building Analytics Dashboards Directly from S3 Files

JOOJO DONTOH — Sat, 27 Sep 2025 07:57:42 +0000

Introduction

Hello guys, my name is Jo and welcome back to my engineering blog!.
I'm gonna talk about something cool today, let's go!😎.
In today's data-driven landscape, organizations collect vast amounts of information from multiple sources in various formats including structured databases, APIs, IoT sensors, and application logs, with much of this data landing in formats like CSV and JSON files that need to be stored, processed, and analyzed efficiently. While Amazon S3 serves as an excellent scalable storage solution for these files, it's a significant engineering challenge 🤕 to extract meaningful insights from data sitting in S3 buckets, especially when stakeholders need regular access to visualizations and reports without the overhead of complex data pipeline management.
Traditional approaches often involve provisioning expensive database instances, writing ETL jobs to move data around, and maintaining multiple data stores just to enable business intelligence tools like Power BI to access the information they need. In this article, I'll walk you through a cost-effective, serverless architecture that uses AWS Glue for data cataloging, Amazon Athena for SQL querying, and Power BI for visualization. This creates a streamlined solution that maintains S3 as your single source of truth while providing the SQL interface that Power BI expects, all without the complexity and cost of traditional database provisioning. Let's dive into it!

While Amazon S3 serves as an excellent scalable storage solution for these files, it's a significant engineering challenge 🤕 to extract meaningful insights from data sitting in S3 buckets

Situation

It's critical to understand that the solution presented in this article is designed for a specific set of circumstances and constraints, and while it may apply to many similar use cases, every engineering decision should start with clearly defined functional and non-functional requirements that match your particular context. In our scenario, CSV files are being deposited into an S3 bucket at regular intervals, which creates a steady stream of data that needs to be accessible for analysis and reporting. These files come pre-structured with a consistent, well-defined schema and the data has already been cleaned and sanitized upstream, meaning we don't need to handle data quality issues, type conversions, or missing field scenarios within our solution. Most importantly, the data represents a single business entity without complex relationships or foreign key dependencies that would require maintaining referential integrity across multiple tables, allowing us to treat each file as a self-contained dataset that can be processed and queried independently.

CSV Files (Regular Intervals) → S3 Bucket Storage
     ↓
Well-structured + Pre-sanitized Data
     ↓
Single Entity (No Relational Dependencies)
     ↓
Ready for Processing (SQL)

Problem/Requirements

The core challenge lies in making the data dumped into S3 buckets accessible and actionable for stakeholders who need regular visibility into operational insights through dashboards and reports. Currently, the team handles similar data requirements by loading CSV contents into traditional SQL databases, which then serve as the data source for Power BI through standard database connections. This solution exists primarily because Power BI has robust, well-established SQL connectivity and most teams are familiar with this approach. However, this conventional method introduces several significant problems that compound over time and create unnecessary operational overhead.

Key Issues with the Current Approach:

Infrastructure Costs - Provisioning and maintaining database instances (RDS, SQL Server, etc.) for what is essentially file storage creates ongoing monthly expenses that scale with data volume and performance requirements
Unnecessary Complexity - Data gets written into SQL tables purely for Power BI access, not because relational database features like ACID transactions, foreign keys, or complex joins are actually needed for this use case
Access Management Overhead - Every new database introduces another set of credentials, connection strings, and security policies that need to be managed, rotated, and monitored
Increased System Dependencies - Adding database services means more moving parts that can fail, require updates, need backup strategies, and demand monitoring - each new service multiplies your potential points of failure
Data Refresh Constraints - Power BI's periodic refresh cycles from SQL databases create timing dependencies where dashboard data freshness becomes a configuration trade-off rather than being driven by actual data availability
Data Fragmentation - Managing multiple data types across different storage solutions (files in S3, structured data in databases) creates inconsistencies in backup strategies, access patterns, and operational procedures

Solution

The Proposed Architecture

The solution replaces the traditional database-centric approach with a serverless, event-driven architecture that treats S3 as both storage and the single source of truth for all data operations. This approach maintains data in its original CSV format while providing the SQL query interface that Power BI requires, eliminating the need for intermediate database layers entirely.

Core Requirements

Our replacement solution addresses four fundamental requirements:

Single Source of Truth - All data remains in S3 in CSV format, eliminating data duplication and synchronization issues
Intelligent Cataloging - Files are automatically cataloged with schema information and logical partitioning, similar to database table structures but without the database overhead
Event-Driven Processing - New file arrivals trigger automatic catalog updates, ensuring data is immediately available for querying
SQL Query Layer - Athena provides the SQL interface that Power BI expects, querying directly against cataloged S3 data

Architecture Components

┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│   CSV File  │───▶│  S3 Bucket  │───▶│ S3 Event    │───▶│ SQS Queue   │
└─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘
                                                                   │
┌─────────────┐    ┌─────────────┐    ┌─────────────┐    ┌─────────┴───┐
│  Power BI   │◀───│   Athena    │◀───│ Glue Catalog│◀───│   Lambda    │
└─────────────┘    └─────────────┘    └─────────────┘    └─────────────┘

Tool Breakdown

Component	Purpose	Role in Solution
S3	Primary data storage	Houses original CSV files and partitioned data
SQS	Event queue	Buffers S3 file creation events for processing
Lambda	Event processor	Reads files, creates partitions, updates Glue catalog
Glue	Data catalog	Maintains schema and partition metadata for Athena
Athena	Query engine	Provides SQL interface for Power BI connectivity

Why This Solution Works

Cost Efficiency

Athena operates on a pay-per-query model, charging approximately $5 per TB of data scanned (AWS Athena Pricing), which is dramatically cheaper than maintaining always-on database instances that can cost hundreds of dollars monthly regardless of usage patterns.

Operational Simplicity

Single Data Format - Everything stays in CSV, eliminating format conversion complexity
Unified Access Control - S3 IAM policies manage all data access, no separate database credentials
Minimal Dependencies - The entire pipeline uses managed AWS services with no servers to maintain
Direct Integration - Athena provides native Power BI connectivity through ODBC/JDBC drivers

Performance Features

Query Caching - Athena automatically caches results for repeated queries
Partition Pruning - Only scans relevant data partitions, dramatically reducing query costs
Columnar Optimization - Can easily migrate to Parquet format later for even better performance

Trade-offs and Considerations

Performance Limitations

-- Traditional DB: Sub-second response for cached data
SELECT * FROM sales WHERE date = '2024-01-15';

-- Athena: 2-10 second response depending on data size
SELECT * FROM sales WHERE year='2024' AND month='01' AND day='15';

Athena queries have higher latency than traditional databases since they scan files rather than using pre-built indexes, but this trade-off is acceptable when query frequency is moderate and cost savings are significant.

Initial Setup Complexity

Power BI + Athena Integration Requirements:

Driver Installation - ODBC/JDBC drivers on dashboard servers
Network Configuration - VPC endpoints and security group rules for federated AWS accounts
Authentication Setup - IAM roles and cross-account access policies
Gateway Configuration - On-premises data gateway driver installation

# Example driver setup
wget https://s3.amazonaws.com/athena-downloads/drivers/ODBC/SimbaAthenaODBC-1.1.17.1000-Linux64.zip
# Configure connection string in Power BI
# Server: athena.us-east-1.amazonaws.com
# Database: your_glue_database

While the initial connectivity setup between Power BI and Athena can be complex, especially in enterprise environments with federated AWS accounts, these are one-time configuration challenges that become valuable learning experiences for teams expanding their cloud-native analytics capabilities.

Suggested Implementation

This implementation guide uses TypeScript, but the concepts translate to any language. The key is building a maintainable, extensible solution that follows the Open/Closed Principle - open for extension, closed for modification.

1. Core Schema Design (Critical Prerequisite)

Schema Analysis and Configuration

Start by examining your S3 files to understand the data structure and extract partition information from filenames:

// Example filename: sales_data_2024_03_15.csv
// Partition extraction: year=2024, month=03, day=15

const filename = "sales_data_2024_03_15.csv";
const partitionPattern = /sales_data_(\d{4})_(\d{2})_(\d{2})\.csv/;
const [, year, month, day] = filename.match(partitionPattern);

Shared Configuration Structure

Create a centralized configuration that serves both your code and Infrastructure as Code (IAC):

// shared-config.json
{
  "aws": {
    "region": "us-east-1",
    "account_id": "${AWS_ACCOUNT_ID}"
  },
  "resources": {
    "source_bucket": "my-company-data-source",
    "analytics_bucket": "my-company-analytics",
    "glue_database": "analytics_db",
    "workgroup_name": "analytics_workgroup",
    "sqs_queue": "file_processing_queue"
  },
  "entities": {
    "sales": {
      "table_name": "sales_data",
      "filename_pattern": "sales_data_(\\d{4})_(\\d{2})_(\\d{2})\\.csv",
      "partition_keys": ["year", "month", "day"],
      "schema": [
        {"name": "transaction_id", "type": "string"},
        {"name": "customer_id", "type": "string"},
        {"name": "amount", "type": "decimal(10,2)"},
        {"name": "transaction_date", "type": "date"}
      ]
    }
  }
}

2. Infrastructure as Code

Account-Agnostic Infrastructure

Structure your IAC to work across any AWS account with minimal changes:

# terraform/main.tf or cloudformation template
# All names reference shared config

resource "aws_s3_bucket" "analytics_bucket" {
  bucket = var.shared_config.resources.analytics_bucket

  versioning {
    enabled = true
  }
}

resource "aws_glue_catalog_database" "analytics_db" {
  name = var.shared_config.resources.glue_database
}

resource "aws_athena_workgroup" "analytics" {
  name = var.shared_config.resources.workgroup_name

  configuration {
    result_configuration {
      output_location = "s3://${aws_s3_bucket.analytics_bucket.bucket}/query-results/"
    }

    enforce_workgroup_configuration = true
    publish_cloudwatch_metrics = true
  }
}

Complete Resource Setup

// Infrastructure components from shared config
const infraConfig = {
  s3: {
    sourceBucket: config.resources.source_bucket,
    analyticsBucket: config.resources.analytics_bucket
  },
  glue: {
    database: config.resources.glue_database,
    tables: Object.keys(config.entities)
  },
  sqs: {
    queueName: config.resources.sqs_queue,
    // Note: Standard queue only - S3 events don't support FIFO
    type: "Standard"
  },
  lambda: {
    functionName: "file-processor",
    runtime: "nodejs22.x",
    sqsTrigger: true
  }
}

3. Core Lambda Logic

File Processing Flow

// lambda/fileProcessor.ts
import { SQSEvent, SQSRecord } from 'aws-lambda';
import { S3, Glue, Athena } from 'aws-sdk';

export const handler = async (event: SQSEvent) => {
  for (const record of event.Records) {
    await processS3File(record);
  }
};

async function processS3File(sqsRecord: SQSRecord) {
  // 1. Extract S3 event from SQS message
  const s3Event = JSON.parse(sqsRecord.body);
  const bucket = s3Event.Records[0].s3.bucket.name;
  const key = s3Event.Records[0].s3.object.key;

  // 2. Get entity configuration
  const entityConfig = getEntityFromFilename(key);

  // 3. Extract partition information
  const partitions = extractPartitions(key, entityConfig.filename_pattern);

  // 4. Read and validate file
  const fileContent = await s3.getObject({ Bucket: bucket, Key: key }).promise();

  // 5. Create partitioned path
  const partitionedPath = buildPartitionPath(entityConfig, partitions);

  // 6. Write to analytics bucket
  await s3.putObject({
    Bucket: config.resources.analytics_bucket,
    Key: partitionedPath,
    Body: fileContent.Body
  }).promise();

  // 7. Update Glue catalog
  await updateGlueCatalog(entityConfig, partitions, partitionedPath);

  // 8. Optional: Validate with Athena query
  await validateCatalogUpdate(entityConfig.table_name, partitions);
}

Partition Management

function buildPartitionPath(entityConfig: any, partitions: any): string {
  // Example: sales_data/year=2024/month=03/day=15/sales_data_2024_03_15.csv
  const partitionPath = entityConfig.partition_keys
    .map(key => `${key}=${partitions[key]}`)
    .join('/');

  return `${entityConfig.table_name}/${partitionPath}/${originalFilename}`;
}

async function updateGlueCatalog(entityConfig: any, partitions: any, s3Path: string) {
  const glue = new AWS.Glue();

  await glue.createPartition({
    DatabaseName: config.resources.glue_database,
    TableName: entityConfig.table_name,
    PartitionInput: {
      Values: entityConfig.partition_keys.map(key => partitions[key]),
      StorageDescriptor: {
        Location: `s3://${config.resources.analytics_bucket}/${s3Path}`,
        InputFormat: 'org.apache.hadoop.mapred.TextInputFormat',
        OutputFormat: 'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat',
        SerdeInfo: {
          SerializationLibrary: 'org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe'
        }
      }
    }
  }).promise();
}

4. Power BI Connection Setup

Prerequisites Checklist

# 1. Local Development Setup
# Download Athena ODBC driver
wget https://s3.amazonaws.com/athena-downloads/drivers/ODBC/SimbaAthenaODBC-1.1.17.1000-Windows.msi

# 2. Gateway Installation (for enterprise distribution)
# Install driver on Power BI Gateway machine
# Configure data source in gateway admin console

Connection Configuration

# ODBC Connection String
Driver={Amazon Athena ODBC Driver};
Server=athena.us-east-1.amazonaws.com;
Port=443;
Database=analytics_db;
Workgroup=analytics_workgroup;
AuthenticationType=IAM Credentials;
UID=AKIA...;
PWD=secret_key;
S3OutputLocation=s3://my-company-analytics/query-results/;

Network Requirements

# VPC Endpoint Configuration (if using private connectivity)
vpc_endpoints:
  - service: com.amazonaws.us-east-1.athena
    route_table_ids: ["rtb-xxx"]
  - service: com.amazonaws.us-east-1.s3
    route_table_ids: ["rtb-xxx"]

# Security Group Rules
security_groups:
  athena_access:
    ingress:
      - protocol: tcp
        ports: [443]
        source: "power-bi-gateway-sg"

IAM Service Account

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "athena:*",
        "glue:GetDatabase",
        "glue:GetTable",
        "glue:GetPartitions",
        "s3:GetObject",
        "s3:ListBucket",
        "s3:PutObject"
      ],
      "Resource": [
        "arn:aws:athena:*:*:workgroup/analytics_workgroup",
        "arn:aws:glue:*:*:catalog",
        "arn:aws:glue:*:*:database/analytics_db",
        "arn:aws:s3:::my-company-analytics/*"
      ]
    }
  ]
}

Security Best Practices

Workspace Binding - Restrict report access to specific distribution lists
Row-Level Security - Implement in Power BI if data contains sensitive information
Credential Rotation - Set up automatic rotation for IAM access keys
VPC Isolation - Use VPC endpoints for private connectivity when possible

This implementation provides a solid foundation that's extensible for additional entities and maintainable across different AWS environments.

Conclusion

This serverless analytics architecture demonstrates how modern cloud services can dramatically simplify data pipelines while reducing costs and operational overhead. By treating S3 as both storage and source of truth, we've eliminated the traditional database layer that often exists purely to satisfy business intelligence tool requirements, not actual business logic needs.

The solution delivers several key advantages: astronomical cost savings through Athena's pay-per-query model versus always-on database instances, operational simplicity with fewer moving parts to manage and monitor, and architectural flexibility that scales effortlessly with data volume and query frequency. The event-driven design ensures data is immediately available for analysis without complex ETL scheduling, while the shared configuration approach makes the entire solution extensible for additional data sources and entities.

While the initial Power BI connectivity setup requires some networking and driver configuration effort, these are one-time investments that unlock significant long-term value. The slight query latency trade-off compared to traditional databases is typically negligible for most business intelligence use cases, especially when weighed against the dramatic reduction in infrastructure costs and complexity.

Key Takeaway: Before defaulting to database solutions for analytics workloads, consider whether your use case actually requires relational database features like transactions, foreign keys, or complex joins. If you're simply storing structured data for reporting and visualization, this S3-native approach can deliver the same business outcomes with fraction of the cost and complexity.

The complete implementation, including Infrastructure as Code templates and Lambda functions, provides a foundation that teams can fork, customize, and extend for their specific data analytics needs. As cloud-native architectures continue to mature, patterns like this represent the future of cost-effective, scalable data engineering.

Ready to implement this solution? Start with the shared configuration design, analyze your existing data patterns, and begin building your serverless analytics pipeline today.

Building a Secure SFTP Server on a Linode Public Subnet

JOOJO DONTOH — Fri, 05 Sep 2025 07:04:52 +0000

In the previous post, I walked through setting up a bare-ish-metal cloud environment with Linode, partitioning public and private subnets, and wiring up your own proxies, and firewalls — without handing everything off to someone else. If you haven't read it, I suggest you do!

Today, I intend to go one level deeper:
For basic learning purposes, let’s build a secure SFTP server from scratch using the node in our public subnet.

Why should you do this you ask?
Because file transfer is a foundational primitive in ops, and there’s no reason to let that knowledge slip through the cracks.

Why Build Your Own SFTP Server?

You don’t need to rely on a SaaS or managed service (lots of manual ops btw)
You control access, retention, and isolation
You understand how file access and security actually work under the hood
You can build automations and workflows around it

This is especially useful when:

You’re collaborating with partners who need to send you files securely
You want to ship logs, reports, or ETL inputs into your infra
You’re learning how SSH, chroot jails, and Linux permissions actually work

Prerequisites

A provisioned Linode instance in your public subnet
A public IP address and port 22 open to trusted IPs
Basic Linux CLI knowledge

I handled the first 2 points in this article
We'll be using Ubuntu 22.04 LTS, but this works on most distros with openssh-server.

Step-by-Step Setup

1. Install OpenSSH

Make sure SSH is installed and running:

sudo apt update && sudo apt install openssh-server -y

2. Create an SFTP-Only User

sudo groupadd sftpusers

sudo useradd -m -G sftpusers -s /sbin/nologin sftpuser1
sudo passwd sftpuser1

This prevents shell access and groups users logically.

3. Create a Secure Directory Structure

OpenSSH’s ChrootDirectory requires that the parent dir is owned by root and not writable.

sudo mkdir -p /sftp/sftpuser1/upload
sudo chown root:root /sftp/sftpuser1
sudo chmod 755 /sftp/sftpuser1

sudo chown sftpuser1:sftpusers /sftp/sftpuser1/upload

This creates a writable /upload directory while keeping the jail secure.

4. Configure `sshd_config` for SFTP Jail

Append this block to the bottom of /etc/ssh/sshd_config (sudo nano /etc/ssh/sshd_config to open):

Match Group sftpusers
  ChrootDirectory /sftp/%u
  ForceCommand internal-sftp
  X11Forwarding no
  AllowTcpForwarding no

Then reload SSH:

sudo systemctl restart ssh

5. Use SSH Key Authentication

On your local machine:

ssh-keygen -t rsa -b 4096 -f ~/.ssh/sftpuser1_key

On your server:

sudo mkdir -p /home/sftpuser1/.ssh
sudo nano /home/sftpuser1/.ssh/authorized_keys
# Paste public key here

sudo chown -R sftpuser1:sftpusers /home/sftpuser1/.ssh
sudo chmod 700 /home/sftpuser1/.ssh
sudo chmod 600 /home/sftpuser1/.ssh/authorized_keys

You can now disable password login if you wish.

6. Secure the Server

Open port 22 to only your office or VPN IP
Install fail2ban:

sudo apt install fail2ban

Consider using logrotate and basic audit logging

7. Test the Setup

From your local terminal:

sftp -i ~/.ssh/sftpuser1_key sftpuser1@<your-linode-ip>

Then:

cd /upload
put testfile.txt

Tip: SFTP isn’t a shell. You can’t run cat or echo — just put, get, ls, etc.

Why Public Subnet?

Because this server needs to be accessed from the internet. If it were in a private subnet, you’d need a bastion or VPN to reach it — useful for internal automation, but not external sharing.

Just like with your previous setup:

The public subnet gives controlled external access
Security is enforced via firewall + SSH key access

Lessons Reinforced

Chroot directories must be owned by root
SFTP can be a secure alternative to email attachments or third-party tools
You can still own your file flows in a modern, cloud-native way

Next Steps

For future improvements and personal learning growth:

Automate uploads from other services or cron jobs
Pipe incoming files into a processing queue (e.g., via inotify or systemd)
Back up uploaded files to S3
Add a DNS record if you want: sftp.yourdomain.com → your Linode IP

Final Thoughts

Owning your infra doesn't mean reinventing everything — it means understanding the tradeoffs and being able to build what you need, when you need it. This is one more building block toward that confidence.

You’ve got this. Nothing is impossible

Reclaiming Engineering Ownership: A Hands-On Guide to Bare-ish-Metal Cloud

JOOJO DONTOH — Fri, 25 Jul 2025 06:13:05 +0000

Introduction

I decided to write this after diving into the ongoing conversation around learned helplessness in software engineering. This is something David Heinemeier Hansson (creator of Ruby on Rails) has been very vocal about especially on his twitter. His points may resonate with you depending on your business needs and the layers of abstraction you are willing to take on. I've seen so many companies rack up huge cloud bills, and that can easily convince smaller teams that they need to do the same to be “serious.” But a lot of this complexity is sold to us by vendors whose business depends on making things look harder than they really are, A.K.A "Merchants of complexity"

Learned helplessness, in this context, happens when engineering teams slowly lose the ability, or even the confidence to manage and understand their own infrastructure. Over time, everything becomes someone else’s service: databases, hosting, even cron jobs. And when that happens, teams risk losing technical depth, the ability to troubleshoot under pressure, and even the curiosity that drives real innovation.

The truth is, setting up your own infrastructure isn’t always as hard or as costly as it seems. Sometimes, going hands-on—provisioning your own servers, configuring your own network—can teach you more and cost you less. This article walks through a high-level, hands-on setup of a simple app using IaaS-level tools (like VPSs, subnets, and Apache proxies), not because it’s the “right” way for every project, but because understanding the layers beneath the abstraction gives you real control—and that’s a power every engineer should have.

All of this has led me to revisit the foundational layers of cloud infrastructure, not to throw shade at modern abstractions, but to get a clearer picture of what they’re built on. In this article, I’ll walk through a high-level setup of a simple, non-production-ready web app. This focuses purely on the purpose of learning and technical understanding. It’s a hands-on journey that starts at the Infrastructure-as-a-Service (IaaS) layer, the lowest abstraction tier in cloud computing (the others being PaaS and SaaS).

Most of us have used cloud platforms in some form, whether it’s deploying serverless functions like AWS Lambda or Firebase Cloud Functions, or using tools like Heroku or Vercel that abstract away orchestration entirely. But beneath all that convenience lies real, raw infrastructure: virtual machines, subnets, proxies, and firewalls. This article is a small tutorial that dives into exactly that. I’ll also drop “nuggets” throughout. Basically pointers to deepen your understanding if you’re curious to dig further at any point in the process.

Resources Used for This Exploration

To keep things practical and grounded, I built a small full-stack notes application that serves as the foundation for the tutorial. The frontend is a Next.js application, responsible for rendering UI and communicating with the backend via API calls. The backend is a lightweight Node.js app built with Koa, handling user authentication and CRUD operations for notes. For data storage, I used a PostgreSQL database containerized and hosted within the private subnet. Everything runs on Akamai’s cloud infrastructure—specifically their VPC and Linode offerings—which provide just enough control to explore low-level networking, subnetting, and proxy setups, without overwhelming complexity.

Provisioning a VPC and Partitioning Your Network

The first step is to rent a VPS from a provider that gives you fine-grained control over networking—options include DigitalOcean, Linode, and AWS EC2. For this project, I chose Akamai’s Linode platform, which allowed me to create a Virtual Private Cloud (VPC) and define custom subnets. I partitioned the network into two subnets: a public subnet that can access the internet (ideal for hosting the frontend), and a private subnet that has no direct internet access (reserved for backend services and the database). When creating your subnets, you’ll need to allocate CIDR blocks to define the IP ranges. For example, the public subnet could use 10.0.1.0/24, while the private subnet could use 10.0.2.0/24. These ranges should be chosen with future growth and IP efficiency in mind. A good practice is to size your subnets based on how many nodes or services you expect to scale into each zone.

💡 Nugget: Take some time to explore how CIDR blocks work, how IP addresses are distributed, and why certain ranges are considered private. It’s a foundational concept for understanding modern networking.

Creating Firewalls to Enforce Subnet Isolation

With your subnets in place, the next step is to enforce network boundaries using firewall rules. Firewalls allow you to control which traffic is allowed to enter or leave a node based on IP ranges, ports, and protocols. For this tutorial, we’ll design our firewall to completely isolate the private subnet from the internet, while exposing only the necessary ports in the public subnet. Let’s break this down into inbound and outbound rules.

Inbound Rules

Inbound rules govern what kind of traffic is allowed into your nodes.

Default Deny:
By default, deny all inbound traffic. Only allow what’s explicitly needed. This is the safest baseline.
ICMP for Testing (Optional):
You may want to temporarily allow ICMP (ping) traffic to help debug connectivity.

Protocol: ICMP
Source: 0.0.0.0/0 (or your own IP)
Action: Accept

Public Subnet — Web Access (Frontend App): Your public-facing frontend must be reachable from the internet.

Ports: 80 (HTTP), 443 (HTTPS)
Protocol: TCP
Source: 0.0.0.0/0
Action: Accept

Private to Public — Forward Proxy Access: To allow your private nodes (backend/DB) to make outbound requests via the public proxy, you must enable inbound access on the proxy port in the public node.

Port: 8080 (or your chosen proxy port)
Protocol: TCP
Source: 10.0.2.0/24 (Private subnet IP range)
Action: Accept

Private Subnet — Internal Communication (Backend ↔ DB): Backend services in the private subnet need to talk to each other, especially to your database node.

Port: e.g., 5432 (PostgreSQL)
Protocol: TCP
Source: 10.0.2.0/24
Action: Accept

SSH Access for Maintenance: You’ll want to be able to SSH into your Linodes for debugging or setup.

Port: 22
Protocol: TCP
Source: Your public IP (or 0.0.0.0/0 if unrestricted, though this is not recommended)
Action: Accept
💡 Tip: For security, restrict this to your personal IP only.

💡 Nugget: SSH uses asymmetric cryptography—your private key remains on your machine, while the public key is added to the server’s ~/.ssh/authorized_keys. Understanding this is essential when managing key-based access securely.

Outbound Rules

Outbound rules control what kind of traffic your nodes are allowed to initiate.

ICMP for Testing (Optional): Allow outbound ping (ICMP) for basic connectivity tests.

Protocol: ICMP
Destination: 0.0.0.0/0
Action: Accept

Internet Access (Public Subnet Only): Allow HTTP and HTTPS requests from the public subnet.

Ports: 80, 443
Protocol: TCP
Destination: 0.0.0.0/0
Action: Accept

Private Subnet via Proxy (Handled Later): The private subnet won’t have direct internet access. Instead, outbound requests will go through the forward proxy configured on the public node. We’ll configure this in a later section.

This firewall setup ensures that your private services are protected, your frontend is accessible, and your infrastructure remains tightly controlled. Always test your rules incrementally—misconfigurations are common but easily fixed if introduced step-by-step.

An example of inbound rules:

Action	Source (IPv4)	Port	Purpose
ACCEPT	`0.0.0.0/0`	80	Public HTTP access (frontend)
ACCEPT	`0.0.0.0/0`	3000	(Optional) direct access to frontend’s internal port (e.g. for testing or bypassing Apache)
ACCEPT	`0.0.0.0/0`	443	Public HTTPS access
ACCEPT	`0.0.0.0/0`	-	ICMP for ping/testing
ACCEPT	`0.0.0.0/0`	22	SSH access (can be restricted to your personal IP)
ACCEPT	`10.0.1.2/32`	8000	Backend service from public proxy
ACCEPT	`10.0.2.0/24`	8080	Proxy communication from private subnet to public proxy
ACCEPT	`10.0.2.0/24`	5432	PostgreSQL access for backend services in the same subnet

Example of Outbound rules:

Action	Destination (IPv4)	Port	Purpose
ACCEPT	`0.0.0.0/0`	443	HTTPS (package installs, certs, APIs)
ACCEPT	`0.0.0.0/0`	80	HTTP (package installs, etc.)
ACCEPT	`0.0.0.0/0`	-	ICMP (ping)

Setting Up Your First Linode (Public Subnet)

With your VPC and firewall rules in place, it's time to spin up your actual infrastructure nodes—starting with a public-facing Linode. This Linode will act as the gateway to your application, serving your frontend (and optionally proxying to your backend), and it’s where we’ll verify that your network and firewall setup is working correctly.

To begin, provision a low-cost Linode (around \$5/month at the time of writing) and assign it to your public subnet. Make sure to also attach the firewall you previously configured, so that all the carefully crafted rules now apply to this instance.

Once deployed, you’ll need to access the machine. You can do this in two ways:

SSH into the Linode using your terminal and the public IP.
Use LISH (Linode Shell) from the Akamai console if you don’t have SSH access yet.

Inside your Linode, perform a few critical connectivity tests to ensure your networking is correctly configured:

Test Internet Access Run a simple ping to Google’s DNS to verify outbound access is allowed by your firewall:

   ping 8.8.8.8

Update the Package Index This verifies that outbound HTTPS is working and you can install packages:

   sudo apt update

If both commands succeed, your firewall and public subnet are configured correctly. You now have a functioning public node, fully capable of installing software, serving applications, and acting as a forward or reverse proxy for your private subnet. This will serve as the entry point to your application infrastructure as we move forward.

Setting Up a Private Linode (Backend Node)

Next, provision your private Linode, which will host your backend application. Just like the public node, this Linode is also very affordable (around \$5/month), but unlike the public node, this one will not be assigned a public IP address—ensuring it has no direct access to or from the internet.

When creating this Linode:

Assign it to the private subnet you created earlier.
Do not assign a public IP address. This isolation is intentional: your backend should only talk to the frontend and the database, not the public internet.
Ensure your firewall rules allow this node to communicate with:

The public node (for outbound access via proxy)
Other private nodes like the DB server

Since the private node will not have internet access, you’ll configure a forward proxy later in the article—hosted on the public Linode—to help it install packages or make outbound HTTP requests securely and indirectly.

In addition to forward proxying, you’ll also need a reverse proxy, typically using Apache, to:

Route external requests to the appropriate internal services (e.g., /api to the backend)
Handle SSL termination and clean URL routing

To configure Apache as a reverse proxy, you’ll later modify the default site config file:

sudo nano /etc/apache2/sites-available/000-default.conf

or better yet, create your own virtual host file to keep things modular and clear.

Buy and Configure a Domain

To make your app feel more “real” and not just accessible by an IP, you should purchase a cheap domain (e.g., from Namecheap or Google Domains). Once you own a domain:

Create an A Record that maps your domain (e.g., notes.online) to the public IP of your frontend Linode.
Once DNS propagation completes, install a free HTTPS certificate via Let’s Encrypt by running:

   sudo certbot --apache -d yourdomain.com

If your DNS records are correctly set up, this command will:

Validate domain ownership
Automatically install an HTTPS cert
Store it in /etc/letsencrypt/live/yourdomain.com

This setup ensures that your frontend can be accessed securely via your custom domain, and that traffic can be reverse-proxied to your backend securely—all while maintaining strict network isolation between tiers.

Setting Up a Private Linode for the Database (PostgreSQL)

The final component of your infrastructure setup is the database node, which will also reside entirely in your private subnet. This ensures that your data is not exposed to the public internet and can only be accessed by other internal services—specifically, your backend application.

Here’s how to set it up:

Provision a new Linode (again, a \$5/month plan will suffice).
Assign this Linode to the private subnet, just like your backend node.
Apply the same firewall to this node so that only traffic from the private subnet—especially your backend—can reach it.
Do not assign a public IP. Your DB should never be exposed to the internet.

Once your Linode is up and running, SSH into it using LISH (if it doesn't have internet access) or set up a jump box via your public Linode, and install PostgreSQL:

sudo apt update
sudo apt install postgresql postgresql-contrib -y

After installation, configure PostgreSQL to accept connections over the private subnet:

Step 1: Update `postgresql.conf`

This file controls PostgreSQL’s runtime behavior. You need to allow it to listen for connections beyond localhost:

sudo nano /etc/postgresql/14/main/postgresql.conf

Find the line:

#listen_addresses = 'localhost'

Replace it with:

listen_addresses = '*'

This tells PostgreSQL to listen on all network interfaces—including the private IP assigned by the subnet.

Step 2: Update `pg_hba.conf`

This file defines who can connect, from where, and how they authenticate.

sudo nano /etc/postgresql/14/main/pg_hba.conf

At the bottom, add a rule that allows incoming connections from the private subnet:

host    all             all             10.0.2.0/24            md5

This means: allow all users to connect to all databases from any machine in the private subnet using password (md5) authentication.

💡 Nugget: Spend time reading about how postgresql.conf and pg_hba.conf interact. They are the gatekeepers of your DB’s network exposure and authentication model.

Step 3: Restart PostgreSQL

Apply your configuration changes:

sudo systemctl restart postgresql

Your PostgreSQL instance is now fully isolated within the private subnet and only reachable by other nodes in the same subnet—specifically your backend node. You’ve effectively recreated a secure, cloud-style VPC networking setup, but on your own terms, for a fraction of the cost.

Enhance Communication Between Nodes and the Internet

In previous sections, we intentionally designed the infrastructure so that nodes in the private subnet do not have direct access to the internet. This is a common and recommended security posture—but it introduces a challenge: how do backend services fetch updates, install packages, or interact with external APIs?

The solution is to introduce a forward proxy in the public subnet. This allows private nodes to send outbound traffic via a trusted middleman (the public node), without exposing themselves directly to the internet.

Why Not Use Basic NAT?

While it's tempting to set up a 1:1 NAT (Basic NAT) for simplicity, this approach bypasses the layered security model we’re trying to build. It essentially grants your private nodes direct exposure, undermining the purpose of subnet separation.

🔗 This is the official documentation of the Akamai Forward Proxy

Set Up a Forward Proxy with Apache (on the Public Node)

Let’s walk through setting up a proper forward proxy using Apache.

Step 1: Access the Public Linode

SSH into your public-facing Linode (in the public subnet) or use the LISH console via Akamai’s dashboard.

ssh root@<your-public-linode-ip>

Step 2: Install & Prepare Apache

Update your packages and ensure Apache is installed:

sudo apt update -y
sudo apt install apache2 -y

Enable necessary Apache modules:

sudo a2enmod proxy proxy_http proxy_connect

Step 3: Create a Forward Proxy Configuration

Open a new config file:

sudo nano /etc/apache2/sites-available/fwd-proxy.conf

Paste the following configuration (adjust IPs as needed):

# Listen on the internal IP (public node) at port 8080.
# This sets up the Apache server to accept proxy requests from the private subnet via port 8080.
Listen 10.0.2.2:8080

<VirtualHost *:8080>
    # Admin email for server issues (not strictly required unless you're sending error reports).
    ServerAdmin webmaster@localhost

    # Root directory for served files (not used in proxying but required syntactically).
    DocumentRoot /var/www/html

    # Log errors from proxy activity here (useful for debugging)
    ErrorLog ${APACHE_LOG_DIR}/fwd-proxy-error.log

    # Log all access through the proxy
    CustomLog ${APACHE_LOG_DIR}/fwd-proxy-access.log combined

    # Enable forward proxy mode (Apache acts as a middleman for outbound traffic)
    ProxyRequests On

    # Adds headers like Via: to show request went through a proxy (useful for tracing)
    ProxyVia On

    # Restrict proxy access to only IPs from the private subnet
    <Proxy "*">
        Require ip 10.0.2.0/24
    </Proxy>
</VirtualHost>

💡 Nugget: The ProxyRequests On directive enables forward proxying. The <Proxy "*"> block restricts usage of this proxy to requests originating from your private subnet only.

Save and close the file.

Step 4: Enable the Proxy Site and Restart Apache

sudo chown root:root /etc/apache2/sites-available/fwd-proxy.conf
sudo chmod 0644 /etc/apache2/sites-available/fwd-proxy.conf
sudo a2ensite fwd-proxy
sudo systemctl restart apache2

Test the Proxy

From a private Linode, you can now route outbound traffic through the proxy:

curl -x http://10.0.2.2:8080 https://example.com

If successful, your private node is now securely communicating with the internet through your public node—without needing a public IP of its own.

This setup retains your network isolation while still enabling secure, auditable internet access.

Enhance Communication Between Nodes and the Internet (Part 2: Private Nodes)

Now that your forward proxy is configured and running on the public Linode, it’s time to set up your private nodes—specifically the backend and database Linodes—to route their outbound internet traffic through this proxy.

Backend Private Node Configuration

On your backend node in the private subnet (which should not have direct access to the internet), you’ll need to explicitly configure it to use the forward proxy you previously set up on the public Linode (e.g., 10.0.1.2:8080).

Start by configuring the proxy settings for apt, so you can perform package installations via the proxy:

echo 'Acquire::http::proxy "http://10.0.2.2:8080";' | sudo tee /etc/apt/apt.conf.d/proxy.conf

Once done, test it using:

sudo apt update

You can also test general HTTP traffic routing through the proxy with:

curl --proxy 10.0.2.2:8080 http://example.com

If both work as expected, proceed to route all HTTP/HTTPS traffic system-wide through the proxy. This ensures that any application or system utility that needs external access will use the proxy automatically.

Edit the /etc/environment file to export proxy variables globally:

sudo nano /etc/environment

Then add:

http_proxy="http://10.0.1.2:8080"
https_proxy="http://10.0.1.2:8080"
no_proxy="localhost,127.0.0.1,::1"

These environment variables will persist across sessions and reboots. For them to take full effect, reboot the node:

sudo reboot

Setup Applications and Storage

With your networking, firewall, and proxy configuration complete, the next step is to deploy your applications on the respective nodes. This section guides you through cloning, configuring, and starting both the frontend and backend apps used in this tutorial, along with their storage setup.

Public Linode – Frontend Application

Your public Linode is where the frontend (Next.js) application will live, and it’s accessible to the outside world via your domain and Apache reverse proxy.

Follow these steps:

Run an update on your packages:

   sudo apt update

Install Git:

   sudo apt install git

Clone the frontend repository made specifically for this tutorial:

   git clone https://github.com/Joojo7/notes-app-frontend
   cd notes-app-frontend

Install the required Node.js dependencies:

   npm install

There’s no need to over-engineer this with a CI/CD pipeline, since it’s a one-off learning project. You can start the app directly or with a tool like pm2 if you want to keep it alive in the background.

Private Linode – Backend Application

On your private backend Linode, follow similar steps, with additional backend-specific setup:

Update the system and install Git:

   sudo apt update
   sudo apt install git

Clone the backend repository:

   git clone https://github.com/Joojo7/notes-app-backend
   cd notes-app-backend

Ensure you have the following installed:

Node.js (v18 or higher)
npm
Docker + Docker Compose

Create a .env file at the root of the project. You can copy from .env.example and customize as needed:

   DB_USER=your_db_user
   DB_PASSWORD=your_db_password
   DB_HOST=localhost
   DB_PORT=5432
   DB_NAME=notes_db
   JWT_SECRET=your_jwt_secret
   JWT_EXPIRATION=15m
   PORT=8000

To simplify the startup process, a startup.sh script has been provided. Make it executable and run it:

   chmod +x startup.sh
   ./startup.sh

This script will handle the Docker Compose setup and the backend service bootstrapping.

More details are available in the backend repo’s README:
🔗 notes-app-backend GitHub

Setup Applications and Storage (continued)

Serve and Test the Application

Once both the frontend and backend are installed and configured, it’s time to serve the application to the internet and test its full flow. The public Linode will act as a reverse proxy, routing requests to the appropriate services via Apache.

Reverse Proxy from Domain to Frontend (and Backend API)

To serve your frontend from https://yourdomain.com without needing to append a :3000 port, we’ll configure Apache as a reverse proxy.

Steps to Enable Apache Modules

SSH into your public Linode and enable the necessary Apache modules:

sudo a2enmod proxy
sudo a2enmod proxy_http
sudo a2enmod headers
sudo a2enmod rewrite

Then restart Apache to apply the changes:

sudo systemctl restart apache2

Configure Apache Reverse Proxy for HTTP and HTTPS

Create or modify a site configuration file:

sudo nano /etc/apache2/sites-available/yourdomain.com.conf

Paste the following configuration:

HTTP (Port 80) – used for initial redirect or Certbot challenge:

<VirtualHost *:80>
    ServerAdmin webmaster@localhost
    DocumentRoot /var/www/html

    ErrorLog ${APACHE_LOG_DIR}/reverse-proxy-error.log
    CustomLog ${APACHE_LOG_DIR}/reverse-proxy-access.log combined

    ProxyRequests Off

    # Reverse proxy to backend API
    ProxyPass /api/notes/ http://10.0.2.2:8000/
    ProxyPassReverse /api/notes/ http://10.0.2.2:8000/

    <Proxy *>
        Require ip 10.0.2.0/24
    </Proxy>
</VirtualHost>

HTTPS (Port 443) – full site access and secure reverse proxy:

<IfModule mod_ssl.c>
<VirtualHost *:443>
    ServerAdmin webmaster@localhost
    ServerName yourdomain.com
    DocumentRoot /var/www/html

    ProxyRequests Off

    # Reverse proxy to backend API (must come before frontend proxy)
    ProxyPass /api/ http://10.0.2.2:8000/
    ProxyPassReverse /api/ http://10.0.2.2:8000/

    # Reverse proxy to Next.js frontend
    ProxyPass / http://localhost:3000/
    ProxyPassReverse / http://localhost:3000/

    # Allow Certbot HTTP challenge
    <Location /.well-known/acme-challenge>
        Require all granted
    </Location>

    # SSL configuration (provided by Certbot)
    SSLEngine on
    SSLCertificateFile /etc/letsencrypt/live/yourdomain.com/fullchain.pem
    SSLCertificateKeyFile /etc/letsencrypt/live/yourdomain.com/privkey.pem
    Include /etc/letsencrypt/options-ssl-apache.conf

    # Logging
    ErrorLog ${APACHE_LOG_DIR}/reverse-proxy-error.log
    CustomLog ${APACHE_LOG_DIR}/reverse-proxy-access.log combined
</VirtualHost>
</IfModule>

Final Steps

Enable the new site config:

   sudo a2ensite yourdomain.com.conf

Disable the default config (optional but recommended):

   sudo a2dissite 000-default.conf

Reload or restart Apache:

   sudo systemctl reload apache2

DNS Setup and Request Flow

Go to your domain provider (e.g., Namecheap, GoDaddy, etc.):
1. Add an A record that points your domain (e.g., yourdomain.com) to the public IP of your public Linode.
2. Wait for DNS propagation — this may take anywhere from a few minutes to a few hours depending on TTL settings.
After DNS is live, the request flow will look like this:

Client Browser
    ↓
DNS Lookup (yourdomain.com resolves to public IP)
    ↓
Firewall (allows ports 80/443 to Apache)
    ↓
Apache Web Server (reverse proxy)
    ↓
    - /api/ → Private backend service via 10.0.2.2:8000
    - /     → Local frontend served from port 3000

Now visit https://yourdomain.com — your frontend should load without the port, and API requests to /api/notes should proxy correctly to the backend on the private node.

Future Improvements and Deep Dives to Sharpen Your Understanding

First off—if you’ve made it this far, take a moment to acknowledge what you’ve accomplished. You’ve not only provisioned infrastructure at the IaaS level, but also configured firewalls, private/public subnets, secure proxies, reverse routing, and a full-stack deployment—all from scratch. That’s huge.

But this journey doesn’t end here. There’s so much more to explore, and none of it is out of reach.

Add More Services for Realistic Environments

Now that you’ve successfully deployed a frontend, backend, and database, try introducing additional Linodes to simulate multi-service architectures. For example:

Add a Redis node, a message queue, or a monitoring service like Prometheus or Grafana.
Observe how service-to-service communication happens over private IPs.
Practice managing security and performance as your ecosystem grows.

Learn About Load Balancing

Load balancing is a cornerstone of high-availability systems.

Study how Apache or Nginx can distribute requests across multiple backend servers.
Try simulating stress or high traffic to watch your load balancing strategy in action.
Experiment with sticky sessions, round-robin, and IP-hash load balancing techniques.

This is how production-scale infrastructure starts.

Rebuild the Proxy Setup in Nginx

Everything you’ve configured with Apache can be reimagined in Nginx.

Learn how to configure reverse proxies and forward proxies in Nginx.
Explore advanced modules like ngx_http_proxy_connect_module for forward proxying.
Compare the verbosity, performance, and control between Apache and Nginx.
You’ll appreciate Apache more—and gain a new appreciation for Nginx too.

You’re not starting from zero anymore. You now have mental models to guide you.

Prepare for Production-Like Workflows

Imagine if this app had users. Or stakeholders. Or deadlines. What would you automate?

Practice triggering deployments from GitHub via webhooks.
Explore CI/CD pipelines with tools like GitHub Actions, ArgoCD, or Terraform.
Think about container orchestration and start reading up on Kubernetes or Nomad.
Look into secrets management, versioned configuration, or observability tooling.

These are things you’ll naturally grow into—and now you know where to start.

Final Thoughts

This wasn’t just a tutorial. It was a walk down the forgotten path of technical self-reliance—the kind that builds confidence, clarity, and curiosity.

Yes, modern SaaS and PaaS platforms are convenient—but they abstract away the very systems we’re responsible for. Sometimes, by getting your hands dirty and walking a little closer to the metal, you reclaim something powerful: your understanding.

So, keep tinkering. Keep asking questions. Keep exploring.

You don’t need a million-dollar cloud budget to learn this stuff.
You just need a \$5 Linode, some grit, and a healthy dose of curiosity.

You’ve got this.

Building practical workflows: data observability, AI trend analysis & proactivity.

JOOJO DONTOH — Sun, 29 Jun 2025 10:15:55 +0000

Introduction

Today I want to talk about data observability and a few other things. My team and I have continuously worked on this section of our workflow I would like to share something about it. To understand data, one must first recognize that it is not merely an output, but a reflection of the logic that has been executed across systems. Every user action, system response, or triggered event leaves behind a residue in the form of recorded information. Rather than existing in abstraction, data carries the imprint of the logic that shaped it, effectively turning each row, record, or object into a timestamped decision made by code.

But working with data isn’t always smooth. Problems often arise when different systems store the same data differently, leading to confusion about which version is correct. Logic applied inconsistently across services creates more gaps. Sometimes data is left without clear ownership, making it hard to maintain. As systems grow, understanding what the data means — and how to work with it — becomes harder. And when teams rely on external tools or manual steps to combine or process data, the risk of mistakes increases.

These issues highlight why data observability matters. Without it, teams can’t easily tell where problems come from or whether their data can be trusted. Observability gives clarity. It helps teams understand how data flows, where it breaks, and how to fix it before it becomes a bigger issue.

What Is Data Observability?

Data observability is the ability to monitor the health and behavior of data as it moves through a system. At its core, it ensures that data is accurate, consistent, and reliable. This means spotting when data is missing, outdated, duplicated, or corrupted — and knowing where and why it happened.

With strong observability in place, teams can quickly detect issues and trace them back to the root cause. For example, if a report shows incorrect numbers, observability makes it easier to see whether the issue came from a failed data load, a logic error, or a stale source. Instead of guessing, teams can investigate with confidence and resolve problems faster.

Beyond fixing errors, data observability plays a key role in decision-making. When teams trust the data, they can make faster and more informed choices. From refining product strategies to debugging subscription flows or interpreting performance metrics, good data leads to better outcomes and observability is what makes that trust possible.

Understanding Data Flow

To practice data observability effectively, a team must first understand how data flows through their system. This means tracking how data is created, how it changes over time, and where it ends up. Without this awareness, it’s difficult to catch issues or explain unexpected results.

Every piece of data goes through different states. For example, a subscription might start in a PENDING_ACTIVATION state, move to ACTIVE, and eventually become EXPIRED or CANCELLED. Each of these states has a meaning tied to business logic. PENDING_ACTIVATION might signal that a user has initiated a subscription but hasn’t yet activated it. EXPIRED could mean the subscription ended naturally, while CANCELLED might indicate a user-initiated termination or a system-triggered rollback.

It’s also important to define how long data is expected to stay in each state. A record stuck in PENDING_ACTIVATION for more than 24 hours might be a red flag. Without defined time windows, teams won’t know when data is stale or whether something has failed silently.

Equally critical is understanding the transition routes — how data moves from one state to another. Tracking these transitions creates transparency and accountability. The best way to do this is through change history. A well-structured change history logs not just what changed, but also metadata around the change: who or what triggered it, when it happened, and why. The ideal structure includes:

Metadata (timestamp, source, actor),
Old Data (previous state or value),
New Data (the updated state or value).

[
  {
    "metadata": {
      "partnerSubscriptionId": "XXXXXXXXXXXXXX",
      "subscriptionEndDate": 1749200000000,
      "smc": "*********",
      "hhid": "*********",
      "salesChannel": "CHANNEL_X",
      "packId": "generic-pack-id",
      "createdAt": 1746000000000,
      "partner": "PARTNER_X",
      "assetId": "GenericAsset",
      "client": "INTERNAL_SYSTEM",
      "subscriptionId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
      "subscriptionStartDate": 1746000000000,
      "userProductSubscriptionId": "XXXXXXXXXXXXXX"
    },
    "hhid": "XXXXXXXX",
    "operation": "MODIFY",
    "puid": "anonymous",
    "loggedAt": 1749200000000,
    "accountId": "anonymous",
    "subscriptionId": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "table": "subscription_table",
    "smc": "XXXXXXXXXXXX",
    "apId": "anonymous",
    "oldData": {
      "subscriptionStatus": "ACTIVE",
      "autoRenew": true,
      "updatedAt": 1746000000000
    },
    "subscriptionStatus": "EXPIRED",
    "assetId": "GenericAsset",
    "partner": "PARTNER_X",
    "SK": "CHANGE#1749200000000",
    "newData": {
      "subscriptionStatus": "EXPIRED",
      "autoRenew": false,
      "updatedAt": 1749200000000
    },
    "id": "xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx",
    "packId": "generic-pack-monthly"
  }
]

With this level of tracking, teams gain a clear view into the life cycle of any data point, making root cause analysis, debugging, and auditing significantly easier.

Consolidation and Aggregation

Healthy data starts with the ability to see the full picture — not just fragments scattered across systems. In modern architectures, it's common for information about a single entity to live in multiple datastores, maintained by different services. Without aggregation, each piece of data remains incomplete, and insights drawn from them are at best limited, at worst misleading.

To make data useful, teams must consolidate it across both internal and external sources. This requires a clear understanding of actor profiles which in some ways are a logical grouping of all relevant data tied to a single subject, such as a user, account, or device. Without this profile view, systems remain reactive and siloed. Nobody wants silos

In our case, aggregation is performed on demand in specific contexts. For example, when we retrieve information tied to a user's smart card, we gather data from several sources at once:

Subscription data, stored internally and reflecting the user's current and historical subscriptions.
Entitlement data, calculated dynamically through CRM logic that applies partner-specific rules, eligibility criteria, and service configurations.
Partner-related data, which may be sourced externally and used to contextualize how the user interacts with third-party services.

Each of these datasets plays a role in shaping the full state of the user. Without aggregation, teams would have to manually stitch these pieces together which is too slow and fragile to support modern operations.

Additionally, system configurational data — feature flags, environment-specific settings, or service-level parameters — must be easy to access and interpret. When teams have quick visibility into the configuration that shaped a data point, debugging becomes faster and business behavior easier to explain. It’s not enough to track the data alone; we must also track the rules that govern how that data behaves.

Understanding Data Freshness and Completeness

Healthy data must not only be accurate — it must also be timely and complete. This is especially true in environments where data is sourced from multiple partners and systems. For our team, freshness refers to how recently the data was updated and how reliably it reflects the current state of CRM activity (our source of truth). When working with external integrations, it’s important to recognize that not all systems operate in real time, so decisions must be made about how often data is fetched, transformed, and loaded.

These decisions directly tie into ETL pipeline design. Striking the right balance between data consistency and performance is key. Trying to always stay perfectly in sync with every upstream system can create unnecessary load or latency. On the other hand, overly infrequent updates can make the data stale and unusable.

To address this, our ETL pipelines are designed to scale both logically and operationally:

Extraction is often handled asynchronously by standalone services or serverless functions with dedicated resources. Since this stage can be resource-intensive, especially when pulling from partner APIs or scanning large internal datasets — it’s decoupled from real-time workflows. This decoupling is important to maintain availability and durability of your real time workflows. The frequency of extraction is tuned based on the freshness requirement of each data source. Some may be pulled hourly, others daily, depending on how critical and volatile the data is.
Transformation includes various forms of computation which include simple mapping to statistical aggregations like totals, averages, and distributions. In our system, partner-specific subscription data is transformed concurrently, using context-aware processing that segments workloads by partner to avoid bottlenecks. Depending on complexity and resource cost, these transformations either happen within the extraction step or are delegated to separate transformation functions.
Load is the final stage, where the processed data is stored. For most of our needs, a single structured JSON file stored in S3 suffices, given that the data is precalculated and intended for read-heavy use cases. To improve performance, we place a read-through cache in front of this storage, allowing downstream consumers to access the latest data quickly. Whenever the load process completes, the cache is also cleared and refreshed, ensuring consistency between stored data and what consumers read.

Completeness is another pillar of data health. Often, the focus is on sanitizing data at entry points — validating input, enforcing schemas, ensuring type safety. But sanitization during retrieval is just as important, especially in systems where manual edits, migrations, or external syncs might have bypassed initial validation. We treat both entry and exit as critical points for enforcing data standards, catching missing attributes, and preserving structural integrity.

Without freshness, data becomes misleading. Without completeness, it becomes fragile. Observability into both helps teams ensure that what they’re seeing is both recent and whole.

Creating Avenues to View Data Health

Observability is only useful when data health can be inspected both broadly and in context. For a team to react quickly to issues, understand root causes, or maintain confidence in their systems, there must be clear and accessible ways to monitor how data is behaving over time.

Viewing data health can happen at two levels: system-wide or context-specific. A broad system view might highlight trends, such as an increase in failed data loads or a drop in expected event volumes. But often, the most meaningful insights come from zooming into a specific actor or data entity — seeing what happened, when, and why.

In our systems, this context-based observability takes on many forms:

User subscriptions are a core entity we track. Each subscription carries a lifecycle — from activation to renewal to expiration — and understanding the health of this data involves checking if transitions occurred as expected, if timestamps align, and if associated metadata (like auto-renew flags or entitlement links) are correct and intact. If a subscription appears stuck or missing key attributes, it may indicate a broader system issue.
Scheduled actions are another important context. These are time-driven operations like renewals, cancellations, or retries. To understand their health, we support queries across time windows and statuses — such as identifying actions that were QUEUED but never EXECUTED, or that failed unexpectedly. Being able to slice this by partner, product, or status allows teams to quickly isolate patterns and respond.

Partner events, which are signals from external systems, form another layer of contextual health. These events might indicate that a user has activated a service, consumed content, or encountered an error. We monitor if these events are received, verify that they’re parsed accurately, and ensure downstream systems respond as intended. When expected events go missing or arrive malformed, it becomes a signal that something upstream may be broken.

By building these contextual views, teams gain the ability to not just observe data — but to understand it. Investigating a single issue or analyzing long-term trends, these views into data health are what separate reactive problem-solving from proactive improvement.

Creating Avenues for Meaningful Data Transition, Extraction, and Storytelling

Raw data, no matter how accurate or complete, becomes significantly more valuable when teams can visualize, interpret, and communicate its meaning. Data storytelling transforms numbers and transitions into narratives that drive understanding, alignment, and action — especially for non-technical stakeholders.

We start with visualization, which is often the most immediate way to surface meaning. Charts help display trends, distributions, and anomalies in a digestible format — whether it's a spike in subscription failures or a dip in partner event delivery. When paired with color-coded statuses, these visualizations can immediately highlight the state of a dataset or flow — for instance, using green for COMPLETED, yellow for PENDING, and red for FAILED — without requiring users to parse detailed logs.

Beyond visuals, we invest heavily in AI-generated summaries to bridge the gap between raw data and human decision-making. Our team uses in-house agents to generate summaries at different levels of granularity:

For user actors, the agent produces insights such as subscription health, recent failures, entitlement mismatches, or eligibility violations.

For partners, another agent compiles metrics and patterns into periodic reports and strategic recommendations, covering usage, errors, and integration health.

We're actively exploring ways to enhance these summaries with memory and context. One improvement involves converting generated summaries into embeddings using NLP(Natural Language Processing) techniques and storing them as vectors. Then, on the next analysis request, the agent could convert the new prompt into an embedding, retrieve the five most similar historical summaries, and enrich the prompt with this context. This approach helps produce better, more informed summaries that evolve over time.

These generated insights are often used in non-technical decision-making, from partner relationship discussions to strategic roadmap planning. For this reason, we also support easy export options — allowing summaries to be copied directly or downloaded as PDFs for distribution in reports and presentations.

To maintain performance and availability, especially under repeated or automated usage, these agents are backed by read-through caches. This prevents overloading the AI systems, reduces latency for frequent queries, and ensures consistency in outputs for the same context.

Ultimately, storytelling is what allows technical data to influence real-world outcomes. By creating tools that present, explain, and share data meaningfully, we ensure it has the power to inform and guide decisions at every level of the organization.

Proactive Data Issue Resolution

While observability helps teams monitor and understand data, the real advantage comes when those insights are used to resolve issues before they escalate. Proactive data resolution means building systems that not only detect anomalies but also guide, automate, or trigger corrective actions across the stack.

The first step involves static logical analysis, which scans data against clearly defined rules to identify structured violations. These are issues that can be caught with deterministic checks — for example, a subscription marked ACTIVE but missing a startDate, or an entitlement with invalid configuration. These checks are currently run on demand during subscription data aggregation. In the future we will automate them to run regularly and help catch data that’s in a broken but detectable state.

More complex problems — especially those involving pattern recognition or inconsistent data relationships — require AI-driven suggestions. These AI agents help identify unstructured violations, such as unexpected spikes in cancellations, or subtle mismatches between entitlements and partner rules. These suggestions are governed by configurable guardrails to ensure they stay within bounds that are understandable and controllable by the team. On the backend, we track prompt consumption, not just for logging and debugging, but to safeguard against misuse, hallucination, or context drift that could compromise model accuracy or security.

Once a violation is detected, either through a rule or an AI-generated suggestion, resolution must be actionable. That’s why we’ve built agents that integrate directly with task management tools like Jira. When an issue is confirmed, these agents can suggest Jira tickets with full context: the data in question, violation type, metadata, and a recommended fix path. This shortens the cycle between detection and accountability, allowing issues to be tracked and resolved in standard engineering workflows.

Another key pillar of proactive resolution is maintaining synchronization between related datastores. In systems where multiple services maintain different views of the same data, desyncs are inevitable. To address this, we intend to use both manual and automated sync pipelines. Some pipelines would run on a schedule to reconcile mismatches, while others can be triggered ad hoc when a drift is manually detected. These processes ensure consistency without requiring constant developer intervention.

Proactivity in data management isn’t just about building alerts — it’s about designing flows that detect, explain, and repair issues at the right level of automation. The result is a system that doesn’t just observe its state, but works to maintain its integrity in real time.

How Data Observability Has Helped My Team

Adopting data observability has had a transformative effect on how our team operates. What once required manual digging, cross-referencing, and tribal knowledge can now be done quickly, visually, and with far more confidence. The biggest shift has been in data surveillance — we now have a clear, consistent view of how data moves and behaves throughout our systems.

One of the most immediate benefits is how easily team members can understand user data profiles. A developer debugging a flow, a QA tester verifying a fix, or a product owner validating a new rule can all inspect the full data picture for any given user without jumping across dashboards or databases. This has made behavioral patterns more traceable, allowing us to detect anomalies like missing activations, frequent subscription failures, or inconsistent entitlement states.

Data gaps such as missing fields, incomplete transitions, or failed triggers now surface visibly, making them easy to flag and investigate early. This has greatly improved our QA workflow, as testers no longer need to manually reconstruct test cases from fragmented logs. Instead, they can validate entire flows from a central point of visibility. Even during UAT, stakeholders can observe how data responds across environments with a bird’s eye view, reducing ambiguity and speeding up feedback cycles. My team is in the process of expanding viewership capabilities of the dashboard to first responders and customer service, which will greatly boost their abilities while helping customers.

Beyond day-to-day operations, observability has helped with tracking one-time activities, such as bulk email campaigns or pre-provisioning subscription entitlements. These kinds of scheduled jobs are notoriously hard to monitor without proper instrumentation. With observability in place, we can now monitor their execution, volume, and any edge-case failures without writing one-off scripts.

Finally, observability has created a direct line of visibility for leadership. High-level statistics, such as total active subscriptions, partner-triggered event rates, or renewal success ratios, are now exposed through curated summaries and dashboards. This allows management to make decisions based on data, without relying on delayed reports or engineering cycles to extract insights.

In short, observability hasn’t just improved how we handle data — it’s improved how the entire team communicates, collaborates, and aligns around it.

Future Improvements in Terms of AI

As our use of data observability matures, we’re looking to expand our AI capabilities with a sharper focus on context-awareness, scalability, and practical integration across teams. One of our main objectives is to maintain a dedicated Small Language Model (SLM) that is trained on internal systems, workflows, and vocabulary. This SLM would act as a lightweight, focused assistant — optimized for our operational context — and continuously refined by internal AI teams to stay aligned with evolving business needs.

A deeper understanding of model management will be essential. Beyond just deploying models, we’re considering a foundation for version control, prompt governance, feedback loops, and performance evaluation in real-world scenarios. The goal is to ensure that the models we rely on not only produce accurate results but also reflect the nuances of our environment and workflows.

We also plan to extend decision-making workflows through AI. This could include suggesting data fixes, detecting and prioritizing anomalies, and recommending operational actions based on historical patterns. These automations wouldn’t replace human decisions, but rather amplify the speed and quality of those decisions, especially in high-volume or high-pressure contexts.

Finally, one of the most exciting frontiers is connecting AI to team priorities and planning. We envision tools that can monitor workstreams, identify friction points, and suggest roadmap improvements based on observed data and recurring pain points. Highlighting areas ripe for automation or surfacing issues that consistently slow down delivery, AI can play a role in shaping strategy, not just supporting it.

Conclusion

Data is no longer just an output of system behavior, it is mostly the foundation on which modern decisions, automation, and user experiences are built. As many systems scale and complexity grows, it becomes crucial not only to collect data but to observe it meaningfully. Data observability ensures that data is complete, accurate, and timely, enabling teams to debug faster, monitor more effectively, and act with confidence.

But observability is only one side of the equation. The other is intelligence — and this is where AI comes in. By summarizing, interpreting, and recommending actions based on observed data, AI allows teams to move from passive awareness to proactive resolution and strategic foresight. Through summaries tailored to users and partners or workflow-integrated agents that assist with decision-making, AI transforms observability from a monitoring tool into a driver of improvement.

Together, data observability and AI form a powerful loop: observability provides the clarity needed to understand the system, while AI brings the intelligence needed to optimize it. The future lies in continuously refining both — building systems that not only see clearly, but think ahead.

Improving Deployment Velocity: How We Rebuilt for Speed and Sustainability

JOOJO DONTOH — Thu, 22 May 2025 09:53:06 +0000

Introduction

When we talk about engineering performance, deployment velocity may be one of the clearest indicators of how effectively a team delivers software. At its core, deployment velocity measures how often code changes are pushed to production. It reflects a team's ability to move fast without breaking things often, respond to change, and continuously improve. High deployment velocity means features, fixes, and improvements reach consumers more quickly, which directly benefits product delivery. For engineers, it creates a healthy rhythm of execution and feedback. It reduces the pressure of large, infrequent releases and gives developers a sense of momentum and progress. When velocity is high and sustainable, it usually points to a team that’s well-organized, technically sound, and empowered to ship confidently.

Tracking What Matters: Deployment as a Reflection of Team Growth

One of the clearest signs of progress we’ve made as a team has been the improvement in our deployment velocity—a reflection not just of speed, but of how well we’ve grown in our ability to plan, execute, and deliver. This success isn’t mine alone, it’s mostly the result of a committed, teachable, and resilient team that embraced change and moved with it. Truly grateful to them. From a measurement standpoint, we were in a good position: our team was already using Jira’s ecosystem effectively, with structured ticket creation, deployment tracking through Bitbucket, and clear release versioning. This meant that we had a consistent stream of data about our work, which gave us a solid foundation to assess progress. Having access to this kind of visibility is crucial as it sets the stage not only for identifying what’s going well, but also for spotting where things might need attention. It helps create a culture where improvement isn’t guesswork—it’s informed and intentional.

To evaluate our delivery progress, I extracted deployment data from Jira’s Deployment Panel and analyzed two distinct 9-month periods: one prior to my joining (October 2023 – July 2024), and one after (August 2024 – May 2025). The analysis focused exclusively on successful production deployments, ensuring that only one deployment per day was acknowledged to avoid overcounting batch releases or automated retries.

We measured progress by calculating the average number of production deployments per week — a clear, time-normalized metric that reflects delivery cadence. In the 9 months before I joined, there were 4 unique and major production deployments, averaging 0.09 deployments per week. In the 9 months following my onboarding, that number grew to 28, with a corresponding weekly velocity of 0.67 deployments. This represents a 0.58 increase in weekly production deployments, or a 633.33% improvement in deployment velocity — a strong signal of enhanced team autonomy, release confidence, and operational maturity.

But these numbers don’t exist in a vacuum. They represent a deeper story of teamwork, trust, and continuous learning. They reflect the changes we made together: better processes, clearer workflows, more confident code, and a shared commitment to improving how we deliver. The steep increase in velocity is also due to enabling deployments for different services while adding a lot more services to the team's portfolio. Tracking this wasn’t about proving a point—it was about understanding our pace, staying accountable, and creating space for sustainable growth.

Meeting the Team That Made It Possible

When I first joined the team, I walked into a group of individuals who, despite their different levels of experience, were deeply committed to getting things done. My manager was a key pillar—resourceful and responsive, always quick to remove blockers and bridge communication with upper management so I could focus on solving problems. Our scrum master brought structure and consistency, especially in cross-team coordination, which was critical for syncing dependencies and moving work forward. I also had the support of two highly detail-oriented QA engineers who ensured we maintained quality even under tight timelines. Then there were the engineers—young, talented, and incredibly teachable. While some were more senior and confident in their technical abilities, others were still finding their footing, but all of them shared a willingness to learn and improve. A few had an impressive grasp of the product and its edge cases, which was a huge help in my early days—they helped accelerate my understanding of the system far more than any documentation could have. Looking back, I’m reminded that transformation doesn’t start with tools—it starts with people. And I was fortunate to walk into a team that had the right mix of curiosity, humility, and heart.

Understanding the Codebase and the System We Serve

When I first met the codebase, I was stepping into a system built to solve a very specific and critical set of problems—managing user viewership access across multiple OTT partners, syncing that with a central CRM, and surfacing valuable data for the analytics team. At its core, the software ensures that when a user is granted access to a service like Prime or Viu, that entitlement is correctly handled, tracked, and communicated across platforms. The stack was familiar: JavaScript (Node.js) on the backend, DynamoDB and RDS for storage, and a broad use of AWS services to handle deployment and orchestration. What made it more interesting, though, was the fact that I joined during a pivotal architectural transition. The team was shifting from a fragmented service-per-OTT model to a unified, partner-agnostic architecture—something that not only streamlined logic, but allowed for better reusability and maintenance. We were also moving away from long-running EC2-based services toward a modular, event-driven architecture powered by AWS Lambda, which significantly reduced costs and simplified scaling.

The codebase itself was structured as a collection of discrete Lambda functions, each mapped to specific handlers and responsibilities. Shared logic and utilities were published and reused across functions using private NPM packages, allowing for cleaner separation and less duplication. The entire deployment flow was managed using the Serverless Framework, which abstracted much of the infrastructure creation. Serverless allowed us to define shared AWS resources—API Gateways, IAM roles, queues, and more—and expose them cleanly across services, making infrastructure both declarative and portable. It was clear that the building blocks were there. The challenge now was to refactor and elevate what existed, without disrupting what already worked.

The Technical Gaps That Slowed Us Down

1. Structure and Duplication

One of the first things I noticed was the lack of a clear and robust file structure. It wasn’t always obvious where functionality lived, and in some cases, versioning was misunderstood. New features were simply added as "v2" or "v3" rather than being named appropriately. More critically, logic was heavily duplicated across the codebase. Similar functions existed in multiple places, often slightly tweaked but essentially performing the same task. This made maintenance time-consuming and error-prone—changing one behavior often meant hunting down and editing several versions of the same logic.

2. Configuration Chaos

The handling of configurations posed a major challenge. Frequently changing values—such as partner IDs, environment toggles, or feature switches—were hardcoded directly in the code. This led to repeated declarations and multiple sources of truth, making even minor updates feel fragile. Without a centralized config management system, engineers had to manually trace where each variable lived and whether it was safe to change—adding unnecessary complexity to what should’ve been routine work.

3. Readability and Coupling

The code itself was often difficult to reason about. Naming conventions lacked consistency, semantics were unclear, and logic wasn’t always placed where you’d expect. This made the onboarding experience slower and raised the cost of every change. On top of that, many components were tightly coupled—meaning a small update in one area could cause unexpected issues elsewhere. Without clear boundaries or separation of concerns, engineers were sometimes forced to write new solutions for problems that had already been solved elsewhere—just because the existing ones weren’t reusable or discoverable.

4. Testing and CI/CD Gaps

Another big contributor to slow delivery was the lack of automated testing. There were no unit tests or integration tests, so regressions were common. Every change carried risk, and confidence was low. The CI/CD pipeline also wasn’t set up to support iterative development. There was no continuous delivery flow, and previous working features in production were sometimes overwritten by newer, unstable releases. These issues made velocity unpredictable, and it was clear that test coverage and release automation needed to be addressed before we could move faster.

5. Environment Bottlenecks

Finally, the absence of a local development and testing environment severely limited parallel work. Engineers were forced to deploy to shared dev or staging environments just to verify basic functionality—often waiting in line to test their code. This not only delayed releases but also introduced friction into everyday development. It was clear that having a local sandbox wasn’t just a convenience—it was a requirement for a healthy, high-velocity engineering workflow.

Operational Gaps That introduced bottlenecks

1. Gaps in Requirement Gathering and Design Planning

Before I joined, there was no dedicated architect or technical lead guiding the product-engineering process. As a result, requirement gathering was often skipped or done informally. Even after stepping in, shifting this habit took time. In the absence of structured discovery, requirements were sometimes misaligned or incomplete—leading to features being built with incorrect assumptions or missing critical edge cases. Key stakeholders were not always engaged early enough, which meant that essential business details were occasionally left out. Additionally, non-functional requirements—like performance, scalability, and maintainability—were rarely discussed, which impacted architectural decisions. There was little focus on translating requirements into thoughtful system designs, leaving modularity, reusability, and extensibility by the wayside.

2. Inefficient QA Feedback Loops

Our testing process also posed a challenge to velocity. Because there was limited automated test coverage and no structured regression suite, QA engineers had to manually retest large parts of the system—even for small changes. This led to longer feedback loops, bottlenecks in the staging environment, and delays in releases. The manual nature of testing also made it difficult to move quickly and safely, especially when features or bug fixes affected shared areas of the codebase. As a result, a lot of time was spent in the validation phase, even for otherwise minor adjustments.

3. Ambiguous or Incomplete Tickets

Many Jira tickets lacked clear acceptance criteria, which caused confusion during implementation and validation. Engineers often had to chase down clarifications or interpret the requirements on their own, which led to misalignment and rework. For QA, the absence of well-defined success criteria made it harder to validate whether a feature was complete or working as intended. This ambiguity not only slowed development—it also created uncertainty around what “done” actually meant, which is critical when working in a fast-paced environment.

Clean up and restructuring

1. Laying the Groundwork

Before diving into any cleanup or restructuring, I dedicated the first few weeks to simply understanding the system and the product. It was important to take a step back and observe—not just the code, but the broader domain we were operating in, how the existing architecture was structured, and where the boundaries between what could be changed and what needed to be worked around actually lay. This initial period was essential for building context: what the service was meant to do, how different OTT integrations functioned, and where the pain points lived—both technically and operationally. I also took time to align with stakeholders on current deliverables and expectations. One of the first pressing tasks was to lead the removal of the payment functionality from our service. This part of the system was no longer relevant as it had been marked for migration to the CRM—and its presence was adding unnecessary complexity and risk. Taking ownership of that cleanup gave me an early opportunity to untangle a critical path, work closely with the team, and begin setting a standard for the kind of change we were about to make together.

2. Reshaping the Codebase

2.1 Establishing a Consistent Foundation

The first step in cleaning up the codebase was to bring in some consistency and formatting discipline. I introduced Prettier across the entire repository and enforced a standard configuration so all contributors were working from the same baseline. This removed noise from pull requests and made the code easier to read and review. While cosmetic, this change set the tone for a more maintainable codebase and gave us a common starting point as we prepared for deeper structural changes.

2.2 Introducing Safe Refactoring Through Testing

Given how coupled and fragile parts of the system were, it wasn’t safe to dive straight into large refactors. To address this, I set up a unit testing framework and added some base tests as a starting point. I then created unit test jobs in the CI pipeline, and hosted a walkthrough with the team to align on how this would work within our development flow. To encourage meaningful adoption, I added a coverage enforcement check that allowed pull requests to pass only if test coverage increased compared to the current baseline. This ensured that every MR helped improve the safety net, bit by bit.

2.3 Reinforcing Testing Through Example and Guidance

To avoid wasting engineering effort or creating resistance, I took the lead in writing initial tests for some of the more complex or obscure sections of the code. This helped show what good tests could look like and made it easier for others to follow. I also used TODO markers within the code to flag key functions that needed coverage as they were updated during feature work. Rather than enforcing testing through policy alone, I made a habit of using code reviews as an opportunity to reinforce quality practices—encouraging things like early returns, meaningful naming, modular design, and reusability.

2.4 Cleaning Up Config and Reducing Duplication

As work progressed, one persistent pain point was the handling of configuration values. Critical settings were hardcoded in multiple places, leading to duplication and the risk of inconsistency. To solve this, I wrote utility scripts that centralized config management into a single folder, making updates easier and safer. This drastically reduced context-switching for engineers and helped eliminate a common source of friction. Together, these efforts gave the team a cleaner, more predictable development experience—making it easier to deliver with confidence and iterate quickly.

3 Integration and delivery

3.1 Reworking the Branching Strategy

When I joined, all deployments were happening directly from the dev branch, which made it hard to manage stability or separate experimental changes from production-ready features. To restore control, I took the last known stable release branch, merged it into master, and then rebased master onto dev to realign the branches. Going forward, we used dev as a long-term integration space—a place for ongoing cleanup, experimentation, and quick tests—while master served as the canonical source for release-ready code. This branching model created clear boundaries between work in progress and what was considered deployable, which was a crucial step toward predictable delivery.

3.2 Enabling Local Development

One of the biggest blockers to delivery was the lack of a local development environment. Engineers were forced to deploy to shared environments just to test their code, meaning only one person could realistically test changes at a time. Since the system was built on a serverless architecture, the team hadn’t yet figured out how to simulate AWS Lambda behavior locally. To solve this, I built a lightweight Express server that mimicked the Lambda runtime. I wired up routes to invoke the existing Lambda handlers and moved all environment variables for staging and dev into gitignored .env files, using dotenv for local support. I also refactored the handlers to support dual execution—as both Lambda functions and Express route handlers. This allowed engineers to run and test features entirely offline. I documented this setup and added README steps, making it easy for anyone to spin it up and start testing immediately.

3.3 Expanding Testing Capacity with an Additional Environment

With dev stabilized and local testing unlocked, the next bottleneck was staging. QA typically validated features in this environment, but with multiple releases in play, it often became a single point of contention. To ease the pressure, I created an additional staging-like environment that mirrored the original setup. This provided a second testing lane, allowing the QA team to test features in parallel and helping us reduce wait times during release cycles. It was a simple change with immediate impact—engineers no longer had to wait for the “main” test environment to free up before validating their work.

3.4 Building Integration Testing from the Ground Up

Integration testing was completely absent when I arrived, which meant QA had to manually retest wide portions of the system for even minor changes. To fix this, I created a dedicated integration test repository. I made the tests compact and easy to run by embedding encrypted environment variables into the repo, so engineers could decrypt and run them out of the box. The test structure mirrored the system’s endpoint layout, making them easy to navigate and extend. To drive adoption, I began pairing integration test tickets with each feature or bug fix ticket, so tests could be written alongside product work. And anytime QA uncovered an issue, we didn’t just fix it—we wrote a test for it. This wasn’t easy to automate initially, but it steadily matured into a reliable system that reduced regression risk and increased deployment confidence across the board.

3.5 Automating the Pipeline and Parallelizing Tests

To bring all the pieces together, I turned my attention to the CI/CD pipeline, which needed significant cleanup to support the environments and workflows we were building. I streamlined the pipeline configuration to properly reflect all available environments and automated critical stages of the deployment process. I integrated Jira deployments, allowing us to track releases directly from our task board. I also ensured that unit tests would run on every commit and every new merge request, creating faster feedback loops and encouraging engineers to catch issues early.

As we started integrating end-to-end tests, we noticed a drop in regression issues—but also a slowdown in build times, especially since the system handled similar functionality across multiple partners. To address this, I parallelized the test suite by running tests separately per partner, each in its own job. This was done by checking out the repo in multiple runners, tagging test files with partner-specific annotations, and using Mocha to selectively run the right set of tests for each parallel job. The result was a dramatic reduction in test execution time and an overall increase in velocity.

Finally, I added dedicated pipeline jobs for different environments, as well as manual release triggers. This gave the team an abstracted, automated delivery flow, where engineers no longer had to manually intervene or piece together build steps. They simply pushed their code, opened a merge request, and the pipeline took care of the rest—only requiring clicks when human validation or release approvals were necessary.

4 Workflow

4.1 Requirements with Architecture in Mind

A major part of increasing delivery efficiency came from getting ahead of the work with clear requirements. I made it a point to collaborate closely with stakeholders early in the process—aligning on what needed to be built, freezing requirements where possible, and translating them into system architecture diagrams. These diagrams weren’t just for me—they became a visual communication tool to bounce ideas off other engineers and architects, ensuring the design made sense before we wrote a line of code. Once confident, I broke the requirements into Jira tickets, often with partial implementations or code snippets inside to give engineers a head start and illustrate what clean, modular implementation could look like. When necessary, I would even join the implementation directly, which helped move things faster and reduced context-switching across the team.

4.2 Streamlining Workflows with Jira Automation

To reduce time spent on task management and coordination, I introduced lightweight automations in Jira that aligned with how we actually worked. Our flow moved from TODO → In Progress → Review → QA → Testing → Done, and my scrum master configured Jira so that tickets automatically moved to “Review” and were assigned to me when a merge request was opened. This meant engineers could stay focused on the task itself, without having to manually update the ticket status or chase reviewers. It also helped me stay on top of what needed to be reviewed without delay.

4.3 Handling QA Feedback with Structured Ticketing

During QA testing, we often uncovered bugs or edge cases that weren’t initially accounted for. To manage this smoothly, we established a routine: categorize the issue, assess its impact, and take immediate action. If it was a functionality break, we created a bug ticket in the current sprint. If it was a newly discovered edge case, we’d revisit the requirements, update them if needed, and either add a ticket to the sprint or backlog. For architectural improvements or design gaps, we created spike issues that I usually handled personally. This workflow ensured that feedback loops were tight and transparent—and most importantly, nothing fell through the cracks.

4.4 Evolving into Automated Release Branching

As we matured, we moved toward a release branching strategy, but I wanted to validate whether it fit the team’s workflow before enforcing it. So, for about 10–40 releases, we did it manually—tracking how the team responded and whether it introduced friction. Once I saw the team was comfortable, I automated the entire release flow using a small serverless function. This script was triggered each time I created a release in Jira and handled the branching logic end-to-end. Automating this step eliminated manual effort, removed the chance of errors, and further increased velocity by streamlining how we shipped code.

5 Enhancing the eagle's eye

5.1 The Problem: Limited Visibility and High Debugging Cost

Before we had proper observability, investigating issues in the system was a time-consuming process. Engineers often had to manually query databases or comb through log streams just to gather basic information. There was no easy way to trace a user's history, understand recent changes, or view how a partner integration behaved at a specific point in time. Even something as essential as subscription change history didn’t exist, which made debugging regressions or investigating edge cases particularly difficult. This lack of visibility slowed us down in moments when speed and clarity were most needed.

5.2 The Solution: Observability Dashboard and Data Aggregation

To solve this, I built a custom observability backend and an internal only dashboard that consolidated the most critical system and user data in one place. At a high level, it provided an overview of all partner-related activity, including:

Total subscriptions per partner
Sales channel distribution
Monthly subscription trends and breakdowns

It also gave stakeholders powerful tools for user-level investigation. By searching a user, they could instantly access:

Identity (minus sensitive info)
Device and eligibility details
Subscription status and full change history
All push notifications triggered by the CRM

This dramatically reduced the turnaround time for debugging and helped teams get to the root of issues without needing backend support or deep system access.

5.3 Impact: Faster Resolution and Strategic Insight

In addition to reducing debugging overhead, the dashboard became a valuable source of insight for product managers and upper leadership. It helped them monitor subscription growth across partners, identify patterns in user behavior, and assess the effectiveness of CRM events and entitlements. The real-time event tracking view made it easier to confirm whether user actions had triggered expected flows—or pinpoint where something had silently failed. What started as a tool for engineering observability quickly became a shared knowledge surface across teams, enabling faster collaboration, smarter decisions, and a stronger sense of control over a complex ecosystem.

6 Alerting System

6.1 The Challenge: No Central Alerting System

At the time I joined, the system had no unified alerting mechanism, and critical issues often went unnoticed until they became user-facing or required manual inspection. There was no structured way to monitor key system failures or event anomalies, which made it difficult to respond quickly in moments that required urgent action. The absence of real-time visibility into failures not only delayed incident resolution but also made the system feel opaque for both engineers and stakeholders.

6.2 The Solution: Centralized, Reusable Notification System

To address this, I built a centralized error logging and alerting mechanism, powered by AWS SNS. Critical system errors and high-priority events were published to a single topic with filtered subscribers—allowing me to fan out alerts to various consumers (emails, logs, dashboards, etc.) without duplicating logic or tightly coupling components. This architecture ensured the system remained modular and reusable, enabling new subscribers to plug into alert streams effortlessly. More importantly, it gave key stakeholders real-time visibility into what was happening, so they could respond to incidents faster and with context.

6.3 User-Facing Notification View

To make alerts even more accessible, I also built a notifications view directly into the dashboard, giving users the option to opt in or out of in-app alerts. This view allowed team members and stakeholders to see critical system activity and messages without relying solely on email, creating a more intuitive and centralized experience. By surfacing this information in a user-friendly way, we gave everyone—engineers, QA, product leads—a shared awareness of system health, directly within the tools they already used day-to-day.

7 The scheduler

7.1 The Problem: Scattered, Rigid Async Logic

When I joined, there were already some solutions in place for handling asynchronous activity, but they were tightly coupled to specific actions—like subscription renewals or email notifications. While these worked in isolation, they weren’t scalable. If a new async task needed to be introduced—say, for downgrading a subscription or sending reminders—a brand new solution had to be built from scratch. For a middleware team expected to handle a wide range of integrations and business workflows, this wasn’t sustainable. We needed something that could adapt with us.

7.2 The Solution: Designing a Scalable, Generic Scheduler

To solve this, I took the initiative to design and implement a reusable, extensible scheduling service—a topic I cover in more detail in another article. This new scheduler was built to be action-agnostic. Any asynchronous activity could be represented as a scheduled "action" with its own configuration: execution time, repeat logic, and stop condition. It now handles everything from subscription terminations and downgrades to reminders and notifications—all in a single system. On top of that, I integrated it with our dashboard so we could get hourly visibility into scheduled activity, giving us a strong signal on system health and operational progress. This wasn’t just a technical upgrade—it gave us clarity and control.

7.3 The Impact: A Reusable Core That Keeps Improving

This scheduler became a central piece of our architecture, dramatically reducing the time needed to implement new async flows. Instead of reinventing the wheel, all we needed to do was schedule a new action or extend the fulfillment logic. It became a flexible engine that directly improved our delivery speed, because it removed the need for repetitive, boilerplate async infrastructure. That said, the journey wasn’t without challenges. We faced (and still refine) issues around load handling, concurrency, rate limiting, and deduplication. But the difference now is that we’re improving a single, unified system—not stitching together new ones with every requirement. The scheduler turned a scattered pattern into a strategic capability.

The Challenges

Understanding a Complex System Without a Map

One of the toughest parts of stepping into this role was grasping the entire system end-to-end—not because the business model was deeply complex, but because the supporting documentation was sparse or outdated. There were gaps between what the system was supposed to do and what the code actually did. That disconnect made onboarding harder than it needed to be. I’ve always believed that the best documentation is often the code itself, but that only works when the code is readable, modular, and semantically meaningful. In this case, I was dealing with a codebase that had accumulated poor naming conventions, logic sprawl, and limited structure, which made it feel like I was reverse-engineering behavior instead of working with an intentionally designed system. It took me a while to mentally map how each function connected to a business workflow, and often I had to rely on multiple sources—logs, QA inputs, and even trial-and-error debugging—to fully understand the purpose of certain components. That cognitive overhead slowed me down initially and made early decisions riskier than I liked.

Balancing Leadership, Reviews, and Individual Contribution

Another major challenge was balancing technical leadership with hands-on contribution. To move the transformation forward at a sustainable pace, I had to go beyond guiding the work—I had to get involved in the work. I love writing code, and during the early stages of cleanup, I was actively implementing changes, setting up tools, fixing tests, and writing automation. But that came with a cost. As a lead, I was also pulled into multiple meetings—syncs with stakeholders, platform discussions, issue triage, and architecture planning. Add to that the weight of code reviews, planning sessions, and mentoring, and it became increasingly difficult to manage my time. While it was rewarding to stay hands-on, it required constant context switching and discipline to ensure I wasn’t bottlenecking others or burning out myself. I had to build boundaries around deep work time and become more intentional about prioritizing leadership tasks without losing my engineering rhythm.

Driving Cultural Change Through Code Reviews

Introducing cultural change is never instant—especially when it comes to engineering discipline and code quality expectations. Early on, many of our review sessions turned into mini workshops, where I’d explain why we needed early returns, how to name things clearly, or why separating concerns was critical for reusability. While the team was wonderfully teachable and open-minded, these sessions often made reviews longer and more involved. It wasn’t just about green checks—it was about transferring thinking patterns and reshaping habits. I didn’t want to enforce standards through silence or bureaucracy; I wanted to help the team see the “why” behind each change. Over time, this started to stick—engineers began reflecting those practices in their pull requests, asking better questions, and thinking more critically about structure. But the emotional and cognitive load of being both a gatekeeper and a teacher was something I had to carry consistently, and it’s one of the less visible but most persistent challenges in trying to build a better culture.

Conclusion: From Foundation to Flow

Looking back, this journey wasn’t just about speeding up deployments or cleaning up a codebase—it was about building clarity, culture, and confidence into a system and a team that were already doing their best with what they had. When I joined, the signs of potential were everywhere: a team that cared, a product with purpose, and a system that—despite its complexity—had survived real-world pressure. But to go from surviving to thriving, we had to be intentional. We had to understand what was slowing us down, challenge it at its roots, and rebuild with scale and sustainability in mind.

The improvements didn’t happen overnight. From setting up local development environments and refactoring legacy code, to establishing CI/CD pipelines and writing test coverage policies, every step required focus, patience, and a willingness to collaborate. We untangled infrastructure, designed reusable patterns, centralized configurations, and introduced observability tools that brought transparency to everyone—from engineers to product leads. We shifted away from reactive firefighting to proactive design, and began using data to guide our decisions, track our growth, and prove our value.

But perhaps the most meaningful transformation wasn’t in the code—it was in the team. We evolved how we work together. Engineers became more confident, more consistent, and more aware of their impact. QA gained tools to test smarter and faster. Stakeholders got visibility into what's really happening. And as for me, I got to witness the kind of change that can only happen when people trust the process and commit to the long haul.

There’s still more to do. There always will be. But the foundation is solid now, and the flow has begun. We’re no longer just delivering—we’re delivering well, and that’s the kind of velocity that matters most.

Understanding My Relationship with AI Through the Lens of TAM

JOOJO DONTOH — Sat, 10 May 2025 06:33:29 +0000

Introduction

Today I want to talk about something a little different. During my master’s degree, I wrote a short paper on technology acceptance. Seeing parts of that thinking show up in how people are adopting AI today led me to write this piece.

We all have our own way of deciding whether a new piece of technology is worth using. Sometimes it just clicks (PlayStation, anyone?). Other times, we hesitate, question it, or drop it altogether. That’s where models come in—to help explain how and why people adopt new tech.

There are plenty of these models out there. One of the most well-known is the Technology Acceptance Model (TAM). Another is the Innovation Diffusion Theory (IDT). A few years ago, researchers combined them to better understand how people were responding to blockchain technology. Their model, still in the hypothesis stage as of 2017, laid out a bunch of helpful ideas that feel just as relevant today—especially when it comes to AI.

This article doesn’t try to prove the model’s hypotheses, but I’ll use them to explain how certain ideas from the model have shown up in my personal adoption of AI.

The model looks at how useful and easy a technology feels, how well it fits into our existing habits (compatibility), how much better it is than what came before (relative advantage), and how complex it seems (complexity). Together, these things shape our attitude, our intention, and ultimately whether we actually use the technology.

Here’s a quick summary of what the model suggests:

If something feels useful and easy, we’re more likely to use it.
If it fits our needs and seems better than what we had, that boosts our interest.
If it feels too complex, that can get in the way—even if it’s powerful.

Even though the original model focused on blockchain, I’ve found that it reflects a lot of how I’ve come to use AI in my own life and work. So in this article, I want to walk through how these ideas apply to my relationship with AI—what pulled me in, what slowed me down, and what finally made it feel worth using.

Deeper dive

Before diving into how this applies to AI, let’s take a closer look at the model. The Technology Acceptance Model (TAM)—especially when extended with ideas from the Innovation Diffusion Theory—tries to explain why we accept or reject new technologies. It starts with three foundational factors:

Compatibility
This refers to how well the technology fits into your existing habits, tools, or lifestyle. If something aligns with how you already think or work, you're far more likely to adopt it. For example, if you're already using cloud tools, switching to a new cloud-based AI service feels natural—it’s compatible.
Relative Advantage
This is about whether the new technology clearly offers something better than what came before. It could be faster, cheaper, more accurate, or just more convenient. If the benefits are obvious and meaningful, people are more motivated to try it.
Complexity
This looks at how difficult the technology feels to understand or use. The more confusing or overwhelming it is, the less likely people are to embrace it—no matter how powerful it is. Simplicity matters, especially early on.

These three factors shape how we experience the technology in two major ways:

Perceived Usefulness – Does this actually help me get things done better or faster?
Perceived Ease of Use – Is this simple enough for me to pick up without frustration?

Together, these perceptions form the backbone of our attitude toward using the technology. If that attitude is positive, it leads to behavioral intention (the decision to try or continue using it), which then leads to actual use.

In short, TAM lays out a kind of emotional and cognitive journey:
Fit → Benefits → Simplicity → Attitude → Intention → Action.

It’s not just about what the tech can do—it’s about how it feels to use it.

Here’s a paragraph that fits into your current section and captures how compatibility applies to you personally, using your structure and tone:

How do these factors apply to me

1. Compatibility (How well it fits)

To make sense of how compatibility plays out in my relationship with AI, I’m going to break it down into entity pairs—essentially, how well two things work together. This approach mirrors the basic idea behind compatibility: does this new thing align with what already exists? So I’ll be looking at how AI fits with me as a person, how it fits with the tools and devices I use, and how I, in turn, relate to those tools. These pairings help unpack not just whether AI works, but whether it fits—with my mindset, my habits, and my environment.

First, there’s no inner conflict between me and the concept of AI. I’ve been in software engineering for years, so the idea of feeding structured/semi-structured data into a system and getting meaningful output doesn’t feel foreign—it feels normal. In that sense, AI and I are a good match.
Then there’s the compatibility between AI and the devices I rely on. Whether it’s my phone, MacBook, iPad, or smart speakers, AI already integrates with many of them, often invisibly, through personal assistants and smart apps.
Finally, there’s the link between me and those devices themselves. I spend a lot of time working with technology and virtual assistants, which makes the transition to AI-enhanced workflows almost frictionless. In short, AI fits well into both my mindset and my digital environment—it didn’t need to force its way in.

2. Relative Advantage (How it’s better than before)

Relative advantage is all about whether AI gives me a noticeable improvement over the way I used to work, think, or solve problems. It’s not just about what AI can do—it’s about what it can do better, faster, or more intelligently compared to my previous tools or workflows. In this section, I’ll explore this idea by looking at how AI has helped me in both professional and personal contexts, using a few practical examples to show where the advantage really shows up.

At work, AI has become more than a novelty tbh it’s a strategic companion. I use it to break down complex technical problems into more manageable parts. It gives me access to a curated knowledge base that spans across domains and industries—something no single search or documentation site can offer. When I'm stuck on a design decision, I can use AI as a sounding board to validate assumptions or explore alternatives. It has helped me spot edge cases I hadn’t considered in architectural discussions, and has become a go-to tool for code generation, implementation strategies, and even migration planning. In moments when I need to move fast without losing depth, AI offers clarity and speed.

Outside of work, the relative advantage shows up in small but meaningful ways. I often use AI to answer quick questions or explore unfamiliar topics—replacing the old process of jumping through multiple tabs or forums. It helps me refine emails (honestly this is a big one), especially when I want to strike the right tone or simplify a message. And sometimes, when I need dinner ideas, it becomes a personalized recipe engine that saves me time and mental energy. My current banana cake recipe is AI generated.

In all of these cases, AI doesn’t just replicate what I used to do—it elevates it and I think this is one of the most important bits (enhancement and not replacement). That’s what makes it relatively advantageous: it extends my capability, accelerates my thinking, and fills in knowledge gaps I didn’t even know I had.

3. Complexity (How hard it is to understand or use)

AI is, by nature, incredibly complex. Behind the scenes, there are machine learning models, massive datasets, and layers of statistical reasoning that all work together to produce something that feels intelligent. It involves everything from neural networks to natural language processing, and these systems don’t just respond—they learn, adapt, and generate context-aware answers. When you think about the engineering and math that goes into building these systems, it’s no surprise that AI can seem intimidating.

But here’s the thing: all that complexity has been beautifully abstracted. Though I have a fair understanding of how some of the items like transformer models, embeddings, vectors or search algorithms work under the hood, I really appreciate how these apps and platforms I use don’t actually require me to understand how embeddings represent meaning in high-dimensional space. Instead, I interact with AI through simple input boxes, I get help building context naturally through prompts and clarifications, and I receive surprisingly meaningful outputs—whether I’m asking for architectural advice or rewriting a sentence. Pure gold

This abstraction removes the complexity from my day-to-day experience almost entirely. As a user, I don’t feel overwhelmed or buried in technical detail. I just use it—and it works. In many ways, it’s like driving a high-performance car without needing to know how the engine is tuned.

All roads lead to actual use

When I reflect on why AI has become part of my life, it’s because of how well it fits for certain tasks, how much better it makes things, and how effortlessly I’m able to interact with it. AI feels like a seamless extension of the tools I already use. I don’t have to jump through hoops to access it or figure out how it works. This is all too familiar in a way that the input-prompt model mirrors how I’ve always used search engines. This effectively presents a low barrier of adoption for me. Beyond that, AI actually helps me get where I want to go. In my work, it allows me to reach my goals efficiently and effectively. Outside of work, it’s just as valuable—whether I’m refining an email, learning something new, or figuring out what to cook. The tools I use abstract away all the complexity happening under the hood, so I never feel intimidated or overwhelmed. It doesn’t feel like I’m using advanced machine learning models, it feels like I’m just having a conversation. And thanks to my small background in machine learning, I have an idea of how complex these systems really are, which only deepens my appreciation for how simple they’ve made the experience.

All of that naturally shapes how I feel about using AI. I have a positive attitude toward it because my experience has consistently been smooth, helpful, and efficient. That attitude makes me more likely to keep using it, and more open to exploring new ways it could support my work and life. It’s not a tool I’m testing anymore—it’s a tool I intend to use safely and regularly. And that intention shows up in my behavior. The line between trying it out and actually using it has quietly disappeared. Heck I pay for it 😂

The Spoof, the bad and the fugly

The journey through the TAM model hasn’t always been smooth. One thing that’s easy to overlook is that TAM isn’t a one-time path you walk through and complete. It’s not a straight line from "this is useful" to "I use it forever." Instead, it’s dynamic—your perceptions can shift, and in some cases, the very things that once made a tool feel useful and easy can start to erode. Over time, the good aspects need to be reinforced, or else the bad ones begin to grow, especially as the product evolves—or fails to.

In my experience with AI, there have been clear breaches in that positive flow. I have to admit, there are moments when the system just doesn’t deliver. During research, I’ve encountered hallucinations—responses that sound confident but are completely made up. When I push deeper with more nuanced or technical questions, I start to see the limits in reasoning or coherence. Sometimes, AI forgets the context I’ve already provided, or applies it incorrectly, forcing me to rephrase, reframe, or start again. And in those moments, my perception of ease and usefulness takes a hit. In moments like these, its feels more brittle than smooth.

When that happens, I find myself taking a step back. I double-check everything. I slow down. I cross-reference with my own knowledge or other sources. That extra cognitive load—the need to proofread, verify, and interpret—interrupts the simplicity I had come to expect. Thankfully, things have improved over time. The models have gotten better, and so have the interfaces that help manage these limitations. But those setbacks are still reminds me that that usefulness is fragile, and ease of use is conditional. Two of the key things to not in my experience with this mode

There’s also something more personal I’ve noticed. I’ve come to believe that keeping a healthy distance from AI is important—not just for accuracy, but for maintaining my own cognitive edge. It’s easy to slip into a mode of over-reliance, especially when a tool is so capable and responsive. But leaning on AI for things I already know—or could figure out with a bit of effort—can dull that edge over time. I think there’s value in doing things the “manual” way sometimes. Recalling knowledge. Struggling a bit. Solving things on my own. That’s not a rejection of AI; it’s a reminder that human ability still matters, and that AI is a tool—not a crutch.

So yes, the TAM can work in reverse. If a system starts to feel less reliable, or too controlling, or too limiting, it can cause attitudes to shift. Intention drops. Actual use fades. And unless those gaps are acknowledged and improved, even the most impressive tech can lose its place in your workflow. For me, staying conscious of these cracks helps me use AI wisely—not blindly. It keeps the relationship

Ok, Technology Acceptance Model (TAM) Matters, Why should we care???

1. Startups and Product Design

For startups, TAM isn’t just a theoretical model—it’s a tactical advantage. In the early stages of building a product, every decision counts. TAM provides a structured way to think about how real users will perceive and interact with the product. Startups can intentionally design their tools to be more compatible with users’ existing habits and workflows, reducing the barrier to entry. They can also focus on delivering a clear relative advantage over existing solutions—whether it's faster execution, better insights, or increased convenience. At the same time, simplifying onboarding and usage helps reduce perceived complexity, allowing users to quickly experience value without feeling overwhelmed.

By keeping these factors in mind—compatibility, advantage, and simplicity—startups are better positioned to shape user perceptions of usefulness and ease. These are critical to early adoption. Additionally, TAM helps startups test whether they’re truly solving a meaningful problem in a way that users will embrace. If users don’t find it useful or easy to adopt, that’s a signal for iteration. TAM trends can also be used to forecast whether the product will gain traction in a competitive market or fall into the trap of being too niche, too hard to use, or just not compelling enough.

2. Understanding Technology Adoption Over Time

TAM, along with the models that have extended or evolved from it, offers a window into how people have historically accepted (or resisted) new technology. What makes TAM especially interesting is that it doesn't isolate adoption to just features or technical specs—it looks at human perception, which is often influenced by much more than just functionality. Over time, researchers have built on TAM to incorporate new dimensions like trust, user experience, hype, and social influence.

These models reveal a lot about how technology acceptance isn’t always driven by individual logic. In many cases, people use new tools because they’re socially pressured to, or because someone they trust has validated the tool. This is especially true in workplace environments or tight-knit communities. Influencers, managers, or even teammates can significantly affect whether a tool is picked up or pushed aside. TAM captures this broader narrative—that adoption is both individual and collective, and shaped by shifting social, technical, and emotional factors. Understanding these patterns helps us predict not only if a tool will be adopted, but why and under what conditions.

3. Enterprise Software Adoption

In larger organizations, where buying and rolling out software affects hundreds or thousands of users, TAM becomes a critical decision-making tool. It allows companies to think beyond cost and features, and instead ask: Will our employees actually use this? Companies can apply TAM to assess internal attitudes toward new tools—either through surveys, small pilots, or feedback loops—and use that insight to design better onboarding, training, and internal communication strategies. For example, if employees perceive a tool as too complex or irrelevant, adoption rates will be low, no matter how powerful the tool is on paper. A clear example of this for me was a software in one of my earlier companies used for time sheets.

To counteract this, some companies introduce gamification—reward systems, challenges, or social features—to increase engagement and shift employee attitudes toward the tool. Over time, these tactics can improve perceived ease of use and usefulness, which in turn boosts intention and actual use. These days, businesses don’t have to start from scratch. They can look at how other companies in their industry have applied TAM principles to software adoption, learning from their successes or failures. In this way, TAM becomes not just a research tool, but a practical playbook for driving internal tech adoption at scale.

Conclusion

My journey with AI has mirrored many of the ideas in the Technology Acceptance Model—how something feels useful, how easy it is to use, how well it fits into my life, and how much better it makes things compared to what came before. But just like any relationship with technology, it hasn’t been perfect. There have been moments of friction, doubt, and necessary distance. What this model reminds me is that adoption isn’t a single decision—it’s a continuous process. As AI continues to evolve, so will my experience with it. And by staying aware of what makes it valuable—and where it falls short—I can keep using it in a way that supports me, not replaces me.

Building a Scalable, Reliable, and Cost-Effective Event Scheduler for Asynchronous Jobs

JOOJO DONTOH — Mon, 20 Jan 2025 09:32:04 +0000

Introduction

Welcome back to my blog! 😁 This is where I talk to myself—and hopefully, to you—about the engineering problems I solve at work. I do this mainly because finding solutions excites me. My journey of identifying inefficiencies, bottlenecks, and challenges has led me to tackle a common yet critical problem in software engineering.

That problem is the need to execute actions asynchronously—often with precise timing and sometimes on a recurring basis. Following my core approach to problem-solving (across space and time), I decided to build a solution that wasn’t just tailored to a single action but was extendable to various use cases. Whether it's sending notifications, processing transactions, or triggering system workflows, many tasks require scheduled execution. Without a robust scheduling mechanism, handling these jobs efficiently can quickly become complex, unreliable, and costly.

To address this, I set out to build a scalable, reliable, and cost-effective event scheduler—one that could manage delayed, immediate and recurring actions seamlessly.

In this article, I’ll walk you through:

The problem that led to the need for an event scheduler
The functional and non-functional requirements for an ideal solution
The system design and architecture decisions behind the implementation

By the end, you’ll have a clear understanding of how to build a serverless scheduled actions system that ensures accuracy, durability, and scalability while keeping costs in check. Let’s dive in!

The Problem: Managing Subscription Changes Across a Calendar Cycle

Subscription management comes with unique challenges, especially when handling cancellations or downgrades 😭. Users can request these changes at any time during their billing cycle, but due to the prepaid nature of subscriptions, such modifications can only take effect at the end of the cycle. This delay introduces a need for asynchronous execution—a system that can record these requests immediately but defer their execution until the appropriate time.

The Solution: A proper scheduling mechanism

Without a proper scheduling mechanism, managing these deferred actions efficiently becomes complex. The system must ensure that every request is executed at the right time while preventing missed or duplicate actions. Furthermore, frequent executions—such as batch processing of multiple scheduled changes—must be handled without overwhelming the system. To address this, we needed a reliable, scalable, and cost-effective scheduler capable of handling delayed and recurring execution seamlessly.

Functional Requirements: Defining the Core Capabilities

A robust and scalable scheduled actions system must be able to efficiently schedule, execute, update, monitor, and retry actions while ensuring reliability and flexibility.

1. Scheduling and Creating Actions

The system must allow users to schedule actions with:

Action type, execution time, execution data, and metadata as required fields.
Optional fields like repeat, frequency, and execution remainder.
Early validation to ensure actions conform to their fulfillment requirements.

2. Updating or Deleting an Action

Users can update or delete an action before it is locked (2 minutes before execution). Once locked, no external changes are allowed.

3. Action Status Management

Each action must have an internally managed status that reflects its execution progress. Status transitions and results must be logged in metadata for tracking.

4. Action Fulfillment Mapping

Every action must map to a specific fulfillment service responsible for its execution. Actions without a matching fulfillment service must be flagged to prevent execution errors.

5. Retrying Failed Actions

Failed actions must retry using exponential backoff to handle temporary failures. Actions that exceed the maximum retry limit must be flagged for manual intervention.

6. Handling Immediate vs. Delayed Actions vs. Repeated Actions

The system must distinguish between immediate and delayed actions to ensure timely execution:

Immediate actions (execution within 2 minutes) must be processed in real-time without scheduling delays.
Delayed actions (execution after 2 minutes) must be scheduled and processed at the correct time.
Repeated actions must be proceed for the required number of times at the required frequency

Non-Functional Requirements (NFR): Ensuring a Reliable and Scalable System

A scheduled actions system must meet key NFRs to guarantee reliability, scalability, security, and maintainability.

1. Reliability and Durability

Actions must execute correctly and on time (±2 minutes).
Repeating actions must execute exactly as scheduled with the correct frequency.
Failed actions must retry with exponential backoff, and non-repeating actions must execute only once.

2. Scalability

The system must scale dynamically to handle high request loads.
A serverless architecture ensures cost-efficiency and flexibility.
A queue-based approach (e.g., AWS SQS) must regulate execution frequency to prevent overloading downstream services.

3. Availability

The system must be always available with no cold starts, ensuring immediate execution when needed. A serverless architecture supports this with reasonable cost

4. Security

Signature-based validation must secure requests and prevent unauthorized execution.

5. Maintainability

The system must be modular, encapsulated, and organized within a single repository.
Infrastructure and database indexing rules must be codified.
A typed language must be used for better reliability.
Local testing must be enabled with encrypted environment variables.
A startup script can automate package installation and environment setup.
Comprehensive tests must ensure safe changes and integration.

6. Observability

API endpoints must expose:
- All scheduled actions.
- Actions filtered by status.
- Retry functionality for failed actions.
A centralized logging system must track execution issues consistently.

Tools: Powering the Scheduled Actions System

AWS Lambda: Serverless Compute for Execution

Enables event-driven execution without managing servers.
Handles action scheduling and validation.
Processes immediate actions using real-time event streams.
Executes delayed actions at the scheduled time.
Manages fulfillment tasks based on the action type.

Amazon EventBridge: Managing Scheduled Execution

Acts as a scheduler for delayed actions.
Polls for due pending actions every 5 minutes and enqueues them for processing.
Ensures execution happens within ±2 minutes of the scheduled time.

Amazon SQS: Queueing Actions for Scalability

Decouples execution workloads by handling scheduled actions asynchronously.
Controls fulfillment request frequency to prevent system overload.
Uses FIFO (First-In-First-Out) processing to maintain execution order and prevent duplicate executions.

Amazon DynamoDB: Storing Scheduled Actions

Serves as the primary database for storing scheduled actions.
Provides fast read/write operations for handling high workloads.
Stores metadata for tracking execution status, retries, and results.
Uses DynamoDB Streams to trigger immediate executions.

Amazon API Gateway: Exposing Endpoints for Management

Provides HTTP endpoints for creating, updating, and deleting scheduled actions.
Exposes monitoring endpoints to retrieve actions by status and retry failed actions.
Ensures secure access with authentication and authorization mechanisms.

System Design: Database Schema for Scheduled Actions

Field	Description
`id`	Unique identifier for each scheduled action.
`data`	Stores execution-specific details.
`action`	Defines the type of action to execute.
`executionTime`	Specifies when the action should run.
`repeat`	Indicates if the action should repeat.
`frequency`	Defines the interval for recurring actions.
`executionRemainder`	Tracks the remaining number of executions.
`status`	Execution state ("PENDING", "IN_PROGRESS", "COMPLETED", "FAILED").
`createdAt`	Timestamp when the action was created.
`updatedAt`	Last modified timestamp.
`retryCount`	Counts failed execution retries.
`metadata`	Stores logs and additional execution details.

Example: Scheduled Notification Action

{
    "data": {
        "mobile": "60123456789",
        "subject": "Test",
        "name": "Joojo",
        "templateType": "USER_LATE_PAYMENT_NOTIFICATION",
        "notificationType": "SMS"
    },
    "repeat": true,
    "frequency": "DAILY",
    "executionRemainder": 5,
    "action": "SEND_NOTIFICATION",
    "executionTime": 1736930117120
}

Project structure

I used a layered-modular approach for maintainability, scalability, and ease of change. Many times, different teams may want to extend changes in a service without introducing unintended side effects. I tried to achieve this by organizing components into distinct modules. Let's dive deeper below

1. Single Application with a Modular Design

The entire system is built as a single application, but with a modular structure that separates concerns. Each module is responsible for a specific aspect of the system, making the codebase easier to navigate and modify.

./src
├── app.ts
├── clients
├── config
├── controllers
├── handlers
├── helpers
├── middleware
├── models
├── routes
├── service
├── types
└── utils

2. Serverless Handlers for Distributed Execution

The project is designed around AWS Lambda, with different handlers exported and structured to allow seamless execution of scheduled actions. These handlers ensure that various tasks are processed independently, improving fault tolerance and scalability.

Action Handlers: Manage creating, scheduling, retrieving, updating, deleting, and processing scheduled actions. This keeps all action-related logic centralized, making it easy to modify without affecting other parts of the system.
Delayed Action Handlers: Specifically handle actions that need to be initiated at a later time. This separation ensures that delayed actions are efficiently scheduled and processed without interfering with real-time execution.
Immediate Action Handlers: Trigger execution for actions that must start within 2 minutes, using DynamoDB Streams to detect changes and initiate execution instantly. This ensures timely processing of urgent tasks.
Fulfillment Handlers: Ensure that scheduled actions are executed properly by interacting with the appropriate fulfillment services. This design allows fulfillment logic to evolve independently of action scheduling.

├── handlers
│   ├── fulfillment.ts
│   ├── initiate-scheduled-actions.ts
│   ├── initiate-stream-actions.ts
│   └── process.ts
       ├── http-apis.ts

3. Maintainability through Separation of Concerns

Each module in the project is self-contained, meaning changes to one component do not directly impact others. This reduces the risk of breaking existing functionality and simplifies debugging.

Controllers handle request routing and execution logic.
Services manage business logic and data interactions.
Clients interact with external services like databases, queues, and APIs.
Models define the data structures used across the system.
Middleware ensures that requests pass through validation and authentication layers.
Utilities provide reusable helper functions for logging, error handling, and retries.

./src
├── clients
├── controllers
├── middleware
├── models
├── service
└── utils

4. Ease of Extensibility

With a modular design, new features can be added without modifying core components. For example:

A new type of scheduled action can be introduced by adding a new action in the fulfillment service without modifying the existing scheduling or queuing logic.
A new external service integration can be implemented by extending the clients module, ensuring seamless communication with third-party systems.

Delayed Execution: Ensuring Timely Execution

The system efficiently processes scheduled actions through periodic execution, ensuring that all pending actions are executed at the right time without delays.

Periodic Execution for Scheduled Actions

A Lambda function periodically scans the database for actions with PENDING status and an executionTime that is due.
Amazon EventBridge acts as a scheduler, triggering this Lambda function every 5 minutes to ensure that actions are picked up on time.
The function enqueues these pending actions into Amazon SQS, ensuring a reliable and scalable execution pipeline.

Why This Approach Works

Efficient batch processing ensures that multiple actions can be picked up at once.
Scalability is maintained by decoupling execution with SQS, preventing system overload. Queues are extremely critical to handling load towards downstream systems.
State Management: Actions follow a lifecycle (PENDING → IN_PROGRESS → COMPLETED/FAILED/NO_ACTION), with each state persisted in the database for tracking and recovery.
Execution Handling: Successful executions are marked COMPLETED, failures are marked FAILED, and recurring actions update their execution remainder before resetting to PENDING.
Automatic Retries: Failed actions use exponential backoff for retries. If retries exceed the limit, the action remains FAILED until manually reset.
Idempotency & Data Integrity: Execution remainders prevent duplicate executions, and invalid operations (e.g., negative remainders) are blocked.
Observability: Metadata stores logs, execution timestamps, API responses, and failure reasons for easy debugging.

Immediate Execution: Handling Time-Sensitive Actions Using DynamoDB Streams

Some scheduled actions require immediate execution if their execution time is within 2 minutes of creation. To handle these efficiently, the system leverages DynamoDB Streams and AWS Lambda for real-time processing.

1. How Immediate Execution Works

DynamoDB Streams detect changes in the database when a new action is inserted or modified.
A Lambda function listens to these changes, processes new actions, and determines whether they require immediate execution.
If an action is scheduled to execute within 2 minutes, the Lambda function enqueues it into Amazon SQS for execution.

2. Breakdown of the Processing Logic

Listening to DynamoDB Stream Events

The function initiateProcessFromDynamoStream is triggered whenever a new record is INSERTED or MODIFIED in DynamoDB.

export const initiateProcessFromDynamoStream = async (
  event: DynamoDBStreamEvent,
): Promise<void> => {
  try {
    const { Records } = event;
    if (!Records || Records.length === 0) {
      console.log("No records to process in DynamoDB stream event.");
      return;
    }
    console.log(`${Records.length} records received.`);

The function checks if there are new records in the event.
If no records exist, the function exits early.

Processing Each Record

The function loops through each record, extracts its details, and determines whether it needs to be processed.

const processingPromises = Records.map(async (record: DynamoDBRecord) => {
  const { eventName, dynamodb } = record;

  if (!dynamodb?.NewImage) {
    console.log("Skipping record: Missing NewImage.");
    return Promise.resolve();
  }

  const cleanedImage = unmarshall(dynamodb.NewImage as Record<string, any>);
  console.log(
    "Cleaned NewImage object:",
    JSON.stringify(cleanedImage, null, 2),
  );

  if (!["INSERT", "MODIFY"].includes(eventName || "")) {
    console.log(`Skipping record with eventName ${eventName}.`);
    return Promise.resolve();
  }
  console.log(`Processing record with eventName: ${eventName}`);

The function extracts newly inserted or modified data from the DynamoDB stream.
It filters out irrelevant records (i.e., records that don’t have a NewImage or are not newly inserted/modified).

Checking for Immediate Execution

The function then checks whether the action needs immediate execution by calculating the time difference.

const { status, retryCount, id, executionTime } = cleanedImage;

// Check buffer time logic
const currentTime = Date.now();
const timeUntilExecution = executionTime - currentTime;

if (timeUntilExecution > TWO_MINUTES_IN_MS) {
  console.log(
    `Skipping record with id ${id}: Execution time is outside the 2-minute buffer window.`,
  );
  return Promise.resolve();
}

The current time is compared with the action’s executionTime.
If the action is more than 2 minutes away, it is skipped (it will be picked up later by the periodic execution).
If the action needs immediate execution, it continues processing.

Ensuring Valid Status and Retry Limits

if (!status || ![STATUSES.PENDING].includes(status)) {
  console.log("Skipping record: Missing or invalid status.");
  return Promise.resolve();
}

if (retryCount !== undefined && retryCount > CONSTANTS.MAX_RETRY) {
  console.log(
    `Skipping record with retryCount exceeding limit: ${retryCount}`,
  );
  return Promise.resolve();
}

if (!id) {
  console.log("Skipping record: Missing id.");
  return Promise.resolve();
}

Ensures the action has a valid PENDING status before processing.
Checks whether the retry limit has been exceeded to prevent infinite retries.
Ensures the action has a valid ID before sending it to the queue.

Sending the Action to SQS for Execution

try {
  await sendMessage(cleanedImage);
} catch (error: any) {
  await fail(id, `Failed to add action to the queue: ${error.message}`);
}

If the action qualifies for immediate execution, it is sent to SQS, where it will be processed by the fulfillment service.
If SQS fails, the action is marked as FAILED and logged for debugging.

3. Why This Approach is Reliable

Real-Time Execution: Actions scheduled within 2 minutes execute immediately instead of waiting for periodic polling.
Automatic Filtering: Actions scheduled for later execution are skipped and processed by EventBridge at the right time.
Error Handling: If an action fails to enqueue in SQS, it is marked as FAILED instead of being lost.
Scalability: The Lambda function can process multiple events concurrently, ensuring no action is delayed.

Repeated Execution: Managing Recurring Actions

Some scheduled actions need to execute multiple times at fixed intervals. The system handles repeated execution using three key fields:

repeat – Indicates whether the action should run multiple times.
executionRemainder – Tracks how many more times the action should execute.
frequency – Defines the time interval between executions.

1. Handling Repeated Execution

The function complete(id, notes) is responsible for managing the completion of actions. If an action is set to repeat, it updates the execution time and tracks how many executions remain.

Deducting Execution Remainder

const newExecutionRemainder = repeat && executionRemainder > 0 ? executionRemainder - 1 : 0;
if (newExecutionRemainder < 0) {
  throw new Error("Execution remainder cannot be negative.");
}

If the action is repeating, the execution remainder decreases by 1.
If the remainder is less than 0, an error is thrown to prevent unintended behavior.

2. Completing the Final Execution

if (repeat && newExecutionRemainder === 0) {
  await update(id, {
    status: STATUSES.COMPLETED,
    ttl: calculateTTL(),
    executionRemainder: 0,
    metadata: {
      ...metadata,
      executionResponses: [...(metadata?.executionResponses || []), notes],
    },
  });

  console.log("Final execution completed successfully:", id);
  return;
}

If no executions remain, the action is marked as COMPLETED.
The TTL (Time-To-Live) is set to delete the record after two weeks.
Execution metadata is updated for tracking and observability.

3. Scheduling the Next Execution

If the action still has remaining executions, the function schedules the next execution.

if (repeat && newExecutionRemainder > 0 && frequency) {
  const frequencyInMs = getFrequencyInMilliseconds(frequency);
  if (!frequencyInMs) {
    throw new Error(`Invalid frequency: ${frequency}`);
  }

  await update(id, {
    status: STATUSES.PENDING,
    executionTime: executionTime + frequencyInMs,
    executionRemainder: newExecutionRemainder,
    metadata: {
      ...metadata,
      executionResponses: [...(metadata?.executionResponses || []), notes],
    },
  });

  console.log("Recurring action updated successfully:", id);
  return;
}

The execution time is updated by adding the interval from the frequency field.
The action status is set to PENDING so it can be picked up again.
The metadata is updated to log execution history.

4. Frequency Conversion

The system converts predefined frequencies into milliseconds to update the execution time.

const frequencyDurations: Record<string, number> = {
  TEN_MINS: 10 * 60 * 1000,
  HOURLY: 60 * 60 * 1000,
  DAILY: 24 * 60 * 60 * 1000,
  WEEKLY: 7 * 24 * 60 * 60 * 1000,
  MONTHLY: 30 * 24 * 60 * 60 * 1000, //Not accurate and for demonstration purposes
};

This allows flexible scheduling based on predefined intervals.

5. Ensuring Idempotency and Data Integrity

For non-repeating actions, the system ensures that execution happens only once.

await update(id, {
  status: STATUSES.COMPLETED,
  ttl: calculateTTL(),
  metadata: { ...metadata, notes },
});
console.log("Action completed successfully with TTL:", id);

Actions that do not repeat are marked COMPLETED immediately.
The TTL ensures that data is retained for a limited time before deletion.

Why This Approach Works

Automated Rescheduling – The system automatically sets the next execution time.
Preventing Overexecution – Execution stops when the remainder reaches zero.
Efficient Tracking – Each execution updates metadata for debugging and observability.
Data Integrity – Ensures that frequency values are valid and that the execution remainder is correctly decremented.

Action Processing: Ensuring Reliable Execution with Deduplication

The system processes scheduled actions using Amazon SQS FIFO Queues or Redis-based deduplication to ensure each action is executed only once, preventing duplicate processing.

1. Handling Action Processing with SQS FIFO

Messages are sent to an Amazon SQS FIFO queue, ensuring actions are processed in first-in-first-out order.
FIFO queues guarantee deduplication, preventing the same message from being processed multiple times.
This approach is ideal for strict ordering and exactly-once processing.

2. Alternative Deduplication Using Redis

If a FIFO queue is not used, the system leverages Redis to manage deduplication before sending messages to a standard SQS queue.

How Redis Deduplication Works

Each message is assigned a deduplication ID based on:
- The action’s unique ID
- The status of the action
- The retry count (if applicable)

const deduplicationId = `${messageBody.id}-${messageBody.status}-${messageBody.retryCount || 0}`;
const redisKey = `sqs-deduplication:${deduplicationId}`;

Before sending a message to SQS, Redis checks if the deduplication ID exists.

const redisCheck = await RedisClient.get(redisKey);
if (redisCheck.success && redisCheck.data) {
  console.log(
    `Duplicate message detected. Skipping send for ID: ${deduplicationId}`,
  );
  return;
}

If a duplicate is detected, the message is not sent, avoiding redundant processing.

3. Sending Messages to SQS

If the action is not a duplicate, it is sent to the SQS queue for processing.

const command = new SendMessageCommand({
  QueueUrl: queueUrl,
  MessageBody: JSON.stringify(messageBody),
});
await executeSQSCommand(command);
console.log(`Message sent successfully to ${queueUrl}`);

The system ensures messages are delivered without unnecessary duplicates.
Actions proceed to the fulfillment stage after entering the queue.

4. Storing Deduplication Data in Redis

After sending a message, the deduplication ID is stored in Redis with a TTL of 5 minutes to ensure temporary deduplication.

await RedisClient.set(redisKey, true, 300); // 300 seconds = 5 minutes

The short expiration time ensures that retried actions are still processed if needed.
Redis helps manage temporary deduplication without affecting long-term action execution.

5. Handling Errors Gracefully

If sending a message to SQS fails, errors are handled based on the failure type:

if (error.name === "TimeoutError") {
  throw new AppError({
    ...CommonErrors.REQUEST_TIMEOUT,
    message: "Timeout occurred while sending the message to SQS.",
    metadata: { queueUrl, messageBody, error: error.message },
  });
}
throw new AppError({
  ...CommonErrors.INTERNAL_SERVER_ERROR,
  message: "Failed to send message to SQS.",
  metadata: { queueUrl, messageBody, error: error.message },
});

Timeouts trigger a specific retry strategy.
Other failures log metadata to help diagnose issues.

Why This Approach Works

FIFO queues ensure strict ordering and deduplication for time-sensitive tasks.
Redis deduplication prevents unnecessary duplicate processing when a FIFO queue is not available.
Error handling mechanisms ensure messages are retried when necessary.

Action Fulfillment: Processing Scheduled Actions from SQS

Once scheduled actions reach their execution time, they are processed by a Lambda function that reads messages from Amazon SQS. The function ensures that actions are executed correctly, updates their status accordingly, and handles errors or retries when needed.

1. Processing Actions from SQS

The fulfill function listens for SQS events, where each record represents a scheduled action that needs execution.

export const fulfill = async (event: { Records: SQSRecord[] }): Promise<void> => {
  const { Records } = event;

  if (!Records || Records.length === 0) {
    console.log("No records to process in SQS event.");
    return;
  }

The function checks if there are new records to process.
If no records exist, it exits early.

2. Ensuring Actions are Executed Correctly

For each action in the queue:

If the action has exceeded the maximum retry attempts, it is marked as FAILED.

if (retryCount >= CONSTANTS.MAX_RETRY) {
  await handleFailure(
    id,
    receiptHandle,
    metadata?.retryReason || "Exceeded maximum retry attempts",
  );
  continue;
}

If the action is being executed for the first time, its status is set to IN_PROGRESS.

if (retryCount === 0) {
  await start(id);
}

3. Handling Different Types of Actions

Even though the scheduler does not allow an action to be scheduled without a valid fulfillment service, there is a defensive mechanism (NO_ACTION) to handle cases where an action is manually altered or corrupted in the database.

If the action type is not recognized, it is marked as NO_ACTION and removed from the queue.

if (!Object.values(ACTIONS).includes(scheduledAction?.action as Actions)) {
  await noAction(id);
  await deleteMessage(receiptHandle);
  continue;
}

If the action is valid, it is processed based on its type.

Processing a General Task

case ACTIONS.EXECUTE_TASK:
  result = await taskExecutionService.performTask(scheduledAction.data);
  await complete(id, result);
  break;

Calls an external service to execute a generic task (e.g., processing a user request).
Marks the action as COMPLETED once finished.

Processing a Notification

case ACTIONS.SEND_ALERT: {
  const { recipient, messageType, ...messageData } = scheduledAction.data;

  const processedMessageData = processMessage(messageData);

  result = await notificationService.send(
    recipient,
    messageType,
    processedMessageData,
  );

  await complete(id, result);
  break;
}

Sends an alert or notification using a notification service.
Marks the action as COMPLETED after execution.

4. Handling Failures and Retries

If an action fails, the system applies exponential backoff and retries the execution before marking it as permanently failed.

Marking an Action as Failed

const handleFailure = async (id: string, receiptHandle: string, reason: string): Promise<void> => {
  console.error(`Action with id: ${id} failed after maximum retries. Reason: ${reason}`);
  await fail(id, reason);
  await deleteMessage(receiptHandle);
};

If an action exceeds retry limits, it is marked as FAILED.
The message is removed from the queue to prevent further processing.

Retrying an Action with Backoff

const handleProcessingError = async (
  id: string,
  receiptHandle: string,
  retryCount: number,
  error: any,
): Promise<void> => {
  console.error(`Error processing message with id: ${id}. Error: ${error.message || error}`);

  await applyExponentialBackoff(retryCount, id);
  const actionMarkedForRetry = await retry(
    id,
    error instanceof AppError
      ? JSON.stringify(error?.metadata) || error?.message || error?.code || error?.name
      : `Processing error : ${error}`,
  );
  await sendMessage(actionMarkedForRetry);
  await deleteMessage(receiptHandle);
};

If an error occurs, the system applies exponential backoff and increments the retry count.
The failed action is re-enqueued into SQS for a retry.

5. Finalizing Execution

Once an action completes successfully, it is removed from SQS.

await deleteMessage(receiptHandle);
console.log(`Message with id: ${id} processed successfully.`);

If all records from the SQS event are processed, a summary log is printed.

console.log(`${Records.length} records from the SQS event have been processed.`);

Why This Approach Works

Ensures Actions are Always Executed – Each action is retried with backoff before failing permanently.
Handles Different Action Types – Supports notifications, tasks, subscription updates, and other scheduled jobs.
Prevents Duplicate Execution – Uses SQS FIFO or Redis deduplication to avoid duplicate processing.
Reliable State Management – Updates the database with IN_PROGRESS, COMPLETED, FAILED, or NO_ACTION statuses.
Defensive Handling – NO_ACTION is a safeguard in case an action is altered manually in the database.

Presentation Layer: Exposing Endpoints for Observability

The presentation layer consists of a single Lambda function that serves as an API, exposing HTTP endpoints through Amazon API Gateway. These endpoints allow users to observe, manage, and interact with scheduled actions, ensuring real-time monitoring and control.

1. Exposing HTTP Endpoints via API Gateway

A serverless API is built using AWS Lambda and API Gateway, providing access to key functionalities related to scheduled actions.

Get Actions by Status and Counts
- Fetches actions grouped by their current status (e.g., pending, completed, failed).
- Provides count summaries to track execution trends.
Initiate Failed Actions
- Allows users to retry failed actions manually.
- Ensures failed jobs can be reprocessed without waiting for an automated retry cycle.
Delete Actions
- Provides an endpoint to remove old or unnecessary actions.
- Helps maintain a clean database by managing expired records.

2. Integrating with a Monitoring Dashboard

The data exposed by these endpoints can be visualized on a dashboard for real-time observability.

Displays action counts by status to track performance.
Allows users to manually retry or delete actions via an interface.
Provides insights into system health and execution reliability.

Challenges in Building the Scheduled Actions System

Developing a reliable and scalable scheduled actions system comes with several challenges that need to be carefully addressed.

Race Conditions
- When multiple processes attempt to update or execute the same action simultaneously, inconsistencies can occur.
- Proper locking mechanisms, deduplication, and FIFO queues help prevent duplicate execution.
Load Testing at Every Point of the Cycle
- The system must be tested under high loads to ensure that scheduling, execution, retries, and fulfillment scale properly.
- Testing includes database performance, SQS message handling, Lambda execution limits, and API response times.
Jobs Being Picked Up by Both Streams and the Scheduler
- Actions scheduled within 2 minutes of execution are processed by DynamoDB Streams, while others rely on EventBridge.
- Without proper coordination, duplicate executions may occur. Ensuring actions transition correctly between pending, in-progress, and completed states prevents this issue.
Understanding Limitations of Tools
- Concurrency Handling: AWS Lambda scales automatically, but high concurrency can lead to throttling and delays in processing.
- Lambda Runtime Limits: Since Lambda has a max execution time, long-running tasks must be broken into smaller executions or offloaded to a worker service.

Future Improvements

Webhooks for Real-Time Notifications
- Implementing webhooks would allow external services that schedule actions to receive real-time updates when an action executes, fails, or retries.
- This reduces the need for polling and improves system responsiveness.
Handler for Missed Actions
- A dedicated handler to detect and process actions that are still in a PENDING state but have an execution time in the past.
- This ensures no scheduled action is permanently missed due to system failures, delays, or scaling issues.

These improvements would enhance reliability, observability, and integration with external systems, making the scheduling system even more robust.

Conclusion: Scheduling, Scaling, and Sanity 🚀

Building a reliable, scalable, and fault-tolerant scheduled actions system isn’t just about setting up a cron job and hoping for the best—it’s about engineering resilience into every step of the process. From scheduling and execution to retries and observability, every component must work together to ensure that no action is lost, no notification is forgotten, and no subscription goes unmanaged.

Through this journey, we've tackled:

✅ Dynamic scheduling with DynamoDB Streams, EventBridge, and SQS.

✅ Precise execution using a mix of immediate and delayed actions.

✅ Reliable processing with deduplication, retries, and exponential backoff.

✅ Scalability by leveraging serverless architecture to handle high loads.

✅ Observability with APIs to monitor, retry, and delete scheduled actions.

Of course, every system has its quirks and challenges—race conditions, tool limitations, and unexpected failures—but with the right design patterns, defensive coding, and future improvements (like webhooks and missed action recovery), this system can evolve into an even more powerful, intelligent, and autonomous scheduler.

At the end of the day, automation is about making life easier—whether it’s managing user subscriptions, sending notifications, or processing time-sensitive transactions. And while computers never sleep, we certainly need to, which is why designing a system that can handle its own problems before waking us up at 3 AM is always worth the effort.

So here’s to building systems that work while we don’t! 🎉

Building a Robust and Cost effective Token Verification Service: A Guide to Secure API Integrations

JOOJO DONTOH — Sun, 10 Nov 2024 11:56:26 +0000

Introduction

In today’s interconnected systems, secure communication between services is very important 🤔. APIs are increasingly used to bridge services across organizational boundaries, often involving sensitive data exchanges such as top secret pancake recipes😅. This has amplified the need for robust mechanisms to verify and authenticate requests between systems securely. Imagine a token verification service that not only signs requests but also automates key rotation, providing security and convenience for API integrations.

In this article, we’ll walk through a solution architecture to build a scalable and secure token verification service using AWS tools like Lambda, Secrets Manager, and API Gateway 🎉. This service is designed to enable partners and internal teams to verify requests originating from your service while maintaining high availability and resilience. The solution balances functional requirements with performance and cost-effectiveness, making it an ideal addition to your organization’s security toolkit. Now let's look at the problem 😢

Problem
Imagine needing to authenticate and authorize requests from internal or external services, sign requests, or implement role-based access control. A straightforward solution could be to integrate symmetric verification endpoints into an existing service and expose them to other services that require these capabilities.
This approach provides a quick and accessible way to add secure, verifiable authentication without extensive restructuring.

But wait! While this quick fix might address the immediate problem, it may not scale well or handle similar challenges beyond the current scope. Here’s why: if the verification is symmetric, meaning the same key is used for both signing and verifying requests, it can create inefficiencies. Depending on the architecture of the host service, this could quickly turn into a bottleneck, especially under high load or across multiple services needing validation.
So, what’s the alternative? Imagine solving authentication and authorization issues “across space and time.” 🧐

Solving problems "across space" means designing a solution that addresses the current issue not just for a single team or service, but for any team or service that might face the same challenge. Meanwhile, solving "across time" involves creating a solution that can address similar problems that may have existed in the past or could arise in the future—a versatile solution that can be applied in various scenarios or extended to tackle new issues.

So, how do we devise a solution that achieves this?

We’ll create a service with the following capabilities:

Exposing public keys for token verification
Signing payloads with private keys on a secure, protected endpoint
Automating key rotation to avoid manual intervention
Ensuring high availability, security, and cost-effectiveness

Solution Overview

Our proposed solution leverages AWS’s serverless offerings—such as Secrets Manager, Lambda, and API Gateway—to handle token signing, verification, and key rotation efficiently, ensuring security, scalability, and maintainability. Here’s a breakdown of the tools and technologies we’ll use:

API Gateway: Manages HTTP requests, allowing secure access to endpoints.
AWS Lambda: Facilitates secure, on-demand execution for signing, public key retrieval, and key rotation.
AWS S3: Stores public keys for easy access through the public endpoint.
AWS Secrets Manager: Manages private keys securely and supports automated rotation.
Serverless Framework: Simplifies deployment and infrastructure management.

This combination allows us to build a service that is secure, highly available, and optimized for fast public key retrieval, making it both lightweight and cost-effective.

Let's scope out the requirements

Functional Requirements

Public Key Retrieval
- A public endpoint will serve public keys for verifying tokens issued by the service, allowing clients to authenticate tokens securely.
Private Key Management
- The service will securely store private keys in Secrets Manager, which can be accessible through a protected endpoint or directly from AWS Secrets Manager via authorized roles.
Authorization Control for Signing
- Only authorized clients will access the signing endpoint, and each client must have explicit permissions.
Automated Key Rotation
- Keys will be rotated automatically to ensure that they remain secure, with an immediate rotation option for emergency cases.

Non-Functional Requirements

High Availability
- The service should be designed with minimal downtime to guarantee constant availability.
Robustness and Maintainability
- Easy-to-maintain code that other teams can access, use, and adapt as needed.
Security and Performance
- Strong security practices, including efficient public key retrieval and endpoint protection, without impacting speed or availability.

Service Architecture and Flow

Key Components

🔐 Token Signing

Clients authorized for signing can request token generation by calling a protected API endpoint. The process works as follows:
- Permissions Check: Only clients with explicit permissions can access this endpoint.
- Private Key Retrieval: The service retrieves the private key securely from AWS Secrets Manager.
- Token Generation: The payload is signed using the private key, and the resulting token is returned to the client.
🔑 Token Verification

Token verification is facilitated by a JWKS (JSON Web Key Set) endpoint, allowing clients to retrieve public keys for verifying tokens. The process is:
- Public Key Access: The JWKS endpoint exposes public keys that match the key ID in signed tokens.
- Verification: Clients use the retrieved public keys to validate the authenticity and integrity of the token.
🔄 Key Rotation

Key rotation is handled automatically by AWS Secrets Manager:
- Periodic Rotation: Secrets Manager rotates keys at set intervals.
- Lambda Handling: An AWS Lambda function manages the updates to public and private keys in Secrets Manager, ensuring smooth transitions.
☁️ Serverless Framework and AWS Integration
- Resource Provisioning: Serverless automates AWS Lambda and API Gateway setups, delivering responsive, load-adaptive infrastructure.
- Environment Configuration: Enables secure, isolated environment configurations (e.g., dev, staging, production) with controlled access to AWS Secrets Manager and S3.
- CI/CD Pipeline: Integrates with Pipelines for automated, reliable deployments without manual intervention.
- Cost Efficiency and Scalability: Combines AWS Lambda’s pay-as-you-go model with Serverless’s efficient setup, reducing idle costs and scaling resources on demand.

High-Level Flow

1. Token Signing Process

Step 1: A client with the necessary authorization initiates a request to sign a payload.
Step 2: The signing endpoint verifies the client’s permissions, retrieves the private key from Secrets Manager, signs the payload, and returns a token with a unique key ID (kid) in the header for easy verification. If the private key is missing, the service automatically rotates keys to ensure availability before signing the payload.

2. Token Verification Process

Step 1: A client receives a signed token, which includes a key ID in its header.
Step 2: The client fetches the matching public key from the JWKS endpoint and verifies the token.

3. Key Rotation Process

Generates two RSA key pairs (2048 and 4096 bits) with unique key IDs.
Retrieves existing public keys, appends the new public keys, and saves the updated set of keys.
Updates or ensures the existence of a private key in AWS Secrets Manager.
Returns metadata upon successful rotation, with error handling for issues during the key generation or save process.

🎉 Advantages of the Solution

🔒 Security: Private keys are securely managed in Secrets Manager, and only authorized clients can access sensitive endpoints.
🌐 Availability: AWS’s serverless infrastructure ensures high availability, even under heavy traffic.
💰 Cost-Effectiveness: Serverless architecture allows on-demand scaling, making it a highly cost-effective solution.
🔄 Resilience: Automated key rotation, with the option for immediate rotation, boosts both security and resilience.
⚙️ Rotation Atomicity: Key rotation is atomic, ensuring public keys update first. If public key storage fails, the process stops, keeping the current key pair intact.If private key storage fails, the failure isn't harmful, because the current key pair is intact. However a configured alert will call for manual intervention.
🔧 Key Signing Dynamism: The sign endpoint supports flexible payload signing, allowing dynamic expiry, key length, and algorithm selection.
📜 Key Signing Availability: If the private key is missing, the sign endpoint triggers a key rotation, ensuring keys are always available for signing.
📉 Pay-as-You-Go Model: AWS Lambda charges only for compute time, making this solution ideal for short-lived signing and verification tasks.
🚫 No Dedicated Infrastructure: Serverless architecture eliminates the need for infrastructure management, as AWS handles scaling and security patches.
🔐 Efficient Secret Management: AWS Secrets Manager securely stores and rotates private keys, reducing the risks and costs of manual management.
📦 Scalable Storage with Amazon S3: Public keys stored on Amazon S3 enjoy durable, scalable storage. Paired with CloudFront, this minimizes data transfer costs.
⚖️ Dynamic Scaling & No Downtime Costs: AWS’s serverless infrastructure scales automatically, eliminating downtime costs and avoiding idle capacity.

🛠️ Use Cases

🔑 Access and Refresh Token Management
- Securely issues and manages access and refresh tokens, allowing clients to verify token validity with ease.
📜 API Request Signing
- Enables clients to securely sign and validate API requests, ensuring the integrity and authenticity of requests.
👥 Role-Based Authentication
- Supports role-specific authentication for actions, restricting access to authorized roles only.
🚨 Emergency Key Rotation and Compliance
- Allows for immediate key rotation in case of a security incident, ensuring quick compliance with security best practices.

Conclusion

Creating a token verification service with automated key rotation and secure signing endpoints is essential in modern API ecosystems. Our solution not only provides a robust token verification framework but also ensures security, availability, and cost-effectiveness. Leveraging AWS’s serverless and managed services makes it easy to scale, maintain, and secure the service, giving partners and teams the confidence to rely on the service for request verification.

This service blueprint provides a powerful framework for implementing secure API integrations. By focusing on both functional and non-functional requirements, you can ensure that your token verification process is resilient, fast, and secure, setting the stage for a more connected and protected API ecosystem.

PS: Though the idea solution and implementation are mine, this article was reviewed and enhanced by my Uncle: Mr Chat GPT.😁