DEV Community: Esa Data

Building a Real-Time E-Commerce Support Platform with an AI Agent Team

Esa Data — Sun, 25 Oct 2026 14:55:55 +0000

Ever abandoned a shopping cart because you couldn't quickly ask, "When will my package arrive?" Industry averages put cart abandonment at around 70%, and slow answers are a major cause: contact forms, ticket portals, and emails that get a reply hours later.

That problem led me to build Aura Gear, a real-time customer support platform for e-commerce. It's also an experiment in how an engineer and a team of AI agents can build a complete system together, from specification to operations.

ketut-garjita / aidevtools

⚡ Aura Gear: Real-Time E-Commerce Customer Support Platform

An asynchronous, ultra-low latency real-time customer support platform built for modern e-commerce storefronts. Features an embeddable customer concierge widget with instant order tracking and a multi-pane support agent command center.

🛍️ Storefront & Customer Chat

📋 Evaluation Rubric Alignment Matrix (12 Criteria / 24 Points)

No	Criterion	Target Score	Evidence / Deliverable File	Summary
1	Problem Description	2 / 2	README.md, product-spec.md	Clearly describes problem, shopper/agent personas, and expected behavior.
2	Product Specification	2 / 2	product-spec.md, openapi.yaml	Full specifications with OpenAPI contract and public task tracker with acceptance criteria.
3	Agent Team Specification	2 / 2	.agents/agents.md, README.md	Defined specialized agent team (`@pm`, `@engineer`, `@qa`, `@devops`, `@specialist`, `@oncall-responder`).
4	Context Management	2 / 2	AGENTS.md, docs/	Master governance file pointing to modular docs (architecture, extension pack, permissions, runbook).
5	Application

…

View on GitHub

The Problem

Online shoppers usually need fast answers on three things:

Parcel tracking and carrier timelines
Return and exchange policies
Sizing, specifications, and inventory availability

Aura Gear addresses this with two components:

Customer concierge widget: an embeddable chat for online stores with instant order tracking.
Agent command center: a multi-pane dashboard for the support team.

Persona	Needs
Shopper	Instant answers with zero friction, order lookup without leaving the page, and the option to talk to a human agent
Support agent	Many concurrent chats, customers' typing previews, one-click canned macros, and instant order context

Architecture at a Glance

Frontend: a lightweight glassmorphic UI (under 150KB) with no build step
Backend: Python 3.10+ with FastAPI and Uvicorn, using native asynchronous WebSockets
Persistence: a thread-safe asynchronous chat store for conversations, messages, and agent assignments, with session recovery
Deployment: multi-stage Docker image on Google Cloud Run

Run it locally with one command:

python3 app_build/server.py

Or with containers:

docker compose up -d --build

Built by an AI Agent Team

The project was designed, built, and verified by six agent personas, each with a specific role:

Agent	Role	Deliverables
`@pm`	Product Manager	`product-spec.md`, `openapi.yaml` API contract
`@engineer`	Full-stack engineer	FastAPI backend, WebSocket manager, UI
`@qa`	QA engineer	Unit and integration tests
`@devops`	DevOps	Docker, CI/CD, Cloud Run deployment
`@specialist`	Agent Ops	MCP server, hooks, agent extension pack
`@oncall-responder`	Incident responder	Automated response when alerts fire

Each agent has its own skill (an instruction file) under .agents/skills/, and a single startcycle.md workflow orchestrates the whole cycle with a human approval gate. Humans keep the final say.

Context Management: Keeping Agents on Track

AI agents perform well when their context is clear, so the repo uses modular documentation:

AGENTS.md: the master governance file
docs/architecture.md: architecture and sequence diagrams
docs/permissions.md: tool execution boundaries, filesystem safety, credential isolation
security/ai-data-policy.md: telemetry policy (no PII or credentials)
ops/runbook.md: diagnosis and alerting guide

Well-organized documentation is the agent's working memory. The clearer the boundaries, the more dependable the results.

Testing and CI/CD

One command runs all tests:

python3 -m unittest discover tests

Unit tests: order ID normalization (#ORD-1001, ord-1001, 1001), the order catalog, and the chat store
Integration tests: health endpoint, order lookup, 404 handling, and the macro endpoint

The GitHub Actions pipeline has three stages:

Push / Pull Request
        |
        v
 Stage 1: CI     -> flake8 + unit/integration tests
        |
        v
 Stage 2: CD Dev -> build container, deploy & verify dev
        |
        v  (manual approval)
 Stage 3: CD Prod -> workflow_dispatch, promote to production
                     + probe Cloud Run health URL

Production deployment sits behind a manual approval gate. Automation can be aggressive in dev, but production calls for a conscious decision.

Alerting on Real User Impact

Instead of generic CPU or RAM graphs, alerts target what users actually feel:

WebSocket drop rate above 5% within 3 minutes (shoppers losing connection)
p95 latency above 500ms (chat feels laggy)
5xx error rate above 2% on /api/orders (order tracking failing)

Every WebSocket connection carries a trace_id so logs can be correlated easily, and telemetry never includes personal data, payment details, or credentials.

The On-Call Agent

When an alert fires, @oncall-responder wakes up on its own:

Parses the alert webhook
Correlates error logs via trace_id
Cleans up memory from orphaned sessions
Restarts ASGI workers if they become unresponsive
Verifies recovery with ops/health-check.sh

This is where platform engineering meets AI engineering: a well-written runbook can become the "brain" of an operational agent.

Try It Yourself

The demo is not kept running permanently. You can deploy it yourself using the instructions in the repo.

Lessons Learned

Spec first, code second. An OpenAPI contract and a task tracker with acceptance criteria keep agents focused.
Clear roles make agents more reliable. One agent, one responsibility, each with its own skill.
Keep humans in the loop. Approval gates in the workflow and in production deploys preserve control.
Alert on user impact, not just infrastructure numbers.
Runbooks are AI assets. Documented procedures can be executed directly by agents.

Closing Thoughts

Aura Gear is more than a chat app. It's an exercise in building a complete software delivery lifecycle with AI agents: specification, code, tests, containers, CI/CD, observability, and incident response.

If you're working at the intersection of data/platform engineering and AI engineering, I'd love to hear how you structure agent roles and guardrails in your own projects. Share your approach in the comments, and a ⭐ on the repo is always appreciated.

From Telemetry to Automated Incident Response: My DevOps & Observability Journey

Esa Data — Sat, 03 Oct 2026 03:49:54 +0000

Description: A hands-on learning journey building observability and automated incident response into a FastAPI order-tracking application using OpenTelemetry, Prometheus, Loki, Tempo, Grafana, and Codex CLI.

From Telemetry to Automated Incident Response

As part of the DataTalksClub AI Dev Tools Zoomcamp, I worked on a small but practical challenge: turning a simple FastAPI application into an observable system with automated incident response.

The project started as a basic Order Tracker application backed by SQLite.

By the end, it had evolved into a system that could:

collect metrics, logs, and traces
visualize telemetry in Grafana
detect application errors through alerting
send incidents to an incident-response service
provide evidence for investigation
trigger a coding agent to investigate and remediate an issue

The most valuable lesson for me was that observability is not just about collecting data.

It is about connecting detection → investigation → remediation → verification.

The Problem

A basic application can work perfectly during development and still fail in production.

For an order-tracking API, a failure might look simple:

GET /api/orders/express-1002

returns a server error.

Without observability, the workflow often becomes:

Something is broken
        ↓
Check application logs
        ↓
Search for the error
        ↓
Try to reproduce it
        ↓
Find the problematic code
        ↓
Apply a fix
        ↓
Restart the application
        ↓
Test again

This works, but it becomes increasingly difficult as the system grows.

I wanted to build a workflow where telemetry could help connect these steps.

The Solution

The final architecture introduced several components:

This architecture separates telemetry collection, storage, visualization, alerting, and incident response.

Instrumenting the Application

The first step was adding OpenTelemetry instrumentation to the FastAPI application.

I wanted each order lookup to produce useful telemetry, including:

HTTP route
HTTP status code
request information
logs
traces
request metrics

For example:

GET /api/orders/standard-1001
→ 200

and:

GET /api/orders/standard-1002
→ 404

The important part was not simply knowing that a request happened.

The telemetry needed enough context to answer:

What endpoint failed, what was the HTTP status, and what happened during the request?

Adding the Observability Stack

The next step was connecting the application to an observability stack:

OpenTelemetry Collector — receives and routes telemetry
Prometheus — metrics
Loki — logs
Tempo — traces
Grafana — visualization and alerting

The OpenTelemetry Collector became the central telemetry pipeline.

Conceptually:

Application
    │
    │ OTLP
    ▼
OpenTelemetry Collector
    │
    ├── Metrics → Prometheus
    │
    ├── Logs    → Loki
    │
    └── Traces  → Tempo

Grafana then provides a single interface for exploring these signals.

This was one of the important lessons from the project:

Metrics tell you that something is happening.
Logs help explain what happened.
Traces help show where it happened.

Using the three together provides much more useful operational context than relying on a single signal.

Grafana Alerting

After telemetry was available, I configured Grafana alerting for server-side failures.

The goal was to detect 5xx responses.

A 404 response such as:

GET /api/orders/standard-1002
→ 404

should not trigger the 5xx incident workflow.

This distinction was useful because it forced me to think about alert conditions rather than simply alerting on every failed request.

The alert also needed to provide enough information for the next stage of the workflow.

Automated Incident Response

The next step was adding a dedicated incident-response service.

The service exposes:

POST /alerts

When Grafana sends an alert, the responder stores the incident information and gathers useful evidence.

The idea is to transform:

Grafana Alert

into:

Incident
├── endpoint
├── status
├── logs
├── traces
└── investigation context

This creates a bridge between observability and automated remediation.

Connecting an AI Coding Agent

The most interesting part of the exercise was connecting the incident workflow to a coding agent.

Instead of simply notifying a developer:

🚨 500 error detected

the system can provide the coding agent with context about the incident.

The intended workflow becomes:

5xx detected
     ↓
Grafana alert
     ↓
Incident-response service
     ↓
Collect evidence
     ↓
Coding agent investigates
     ↓
Identify root cause
     ↓
Apply fix
     ↓
Restart application
     ↓
Verify behavior

This changes the role of AI from simply generating code to participating in an operational workflow.

Finding the Bug

The final exercise intentionally exposed a bug in the Express order lookup.

The problematic implementation calculated the estimated delivery date using:

estimated_at = placed_at.replace(day=placed_at.day + 2)

This looks reasonable at first glance.

However, it assumes that the resulting day exists in the same month.

For an order created near the end of a month, that assumption can fail.

The fix was to use date arithmetic instead:

estimated_at = placed_at + timedelta(days=2)

This allows Python's datetime implementation to correctly handle month boundaries.

The important part was not only finding the line of code.

The observability and incident-response workflow provided the path from:

HTTP failure
    ↓
Telemetry
    ↓
Alert
    ↓
Incident evidence
    ↓
Root-cause investigation
    ↓
Code change
    ↓
Verification

What I Learned

1. Observability is more than logging

Before this project, it was easy to think of observability as:

"Add some logs."

The project changed that perspective.

A useful observability system combines multiple signals and makes them actionable.

2. Alerts need context

An alert saying:

Something went wrong

is not particularly useful.

An actionable incident should provide enough context to start an investigation.

For example:

Endpoint
HTTP status
Logs
Trace
Time window
Dashboard context

3. Automation should be connected to verification

Automatically changing code is not enough.

A remediation workflow should also verify that the application works after the change.

That makes the workflow closer to:

Detect → Investigate → Fix → Verify

rather than simply:

Detect → Fix

4. AI agents need operational context

A coding agent becomes much more useful when it receives evidence from the system instead of being asked to investigate blindly.

Telemetry can provide the context needed for the agent to understand what actually happened.

Technology Stack

The project brought several tools together:

Area	Technology
API	FastAPI
Runtime	Python
Database	SQLite
Telemetry	OpenTelemetry
Telemetry Pipeline	OpenTelemetry Collector
Metrics	Prometheus
Logs	Loki
Traces	Tempo
Visualization	Grafana
Containers	Docker / Docker Compose
AI-assisted remediation	Codex CLI

Verification

The completed workflow was validated through the homework tasks.

Step	Scenario	Result
Q1	Application health check	`200`
Q2	Existing order lookup	`200`
Q3	Missing order lookup	`404`
Q4	5xx alert behavior	Normal for 404
Q5	Incident-response workflow	Completed
Q6	Express order incident	Root cause identified and fixed

The important outcome was not just that each individual component worked.

The complete workflow worked as a chain.

Final Takeaway

The biggest lesson from this project was that observability becomes much more valuable when it is connected to action.

A mature workflow can look like:

Application
     ↓
Telemetry
     ↓
Observability
     ↓
Alerting
     ↓
Incident Response
     ↓
AI-assisted Investigation
     ↓
Remediation
     ↓
Verification

This project gave me hands-on experience connecting these pieces together rather than learning them as isolated technologies.

For me, that's the real value of Learning in Public: documenting not only what I built, but also the problems I encountered, the root causes I found, and what changed in my understanding along the way.

Acknowledgments

This project was completed as part of the DataTalksClub AI Dev Tools Zoomcamp — Homework 4: DevOps and Observability for AI-Built Apps.

Thanks to the DataTalksClub community for creating practical exercises that connect software development, DevOps, observability, and AI-assisted engineering.

Github Repository: order-tracker-v2

Agent Relay v2

Esa Data — Mon, 21 Sep 2026 10:44:14 +0000

From SQLite to Kubernetes: Building a Test-Gated CI/CD Pipeline for a Task Relay Service

I recently completed an engineering exercise that started as a small FastAPI + SQLite application and gradually evolved into a containerized, Kubernetes-deployed service with a full CI/CD pipeline.
The project is Agent Relay — a task relay service for registering agents, delivering tasks, tracking claims, and recording results.

The evolution

SQLite → PostgreSQL → Docker → Kubernetes (Kind) → GitHub Actions CI/CD

What made this interesting wasn't just "getting it running" — it was watching each layer expose a new class of problem that the previous layer didn't have to deal with.

What I implemented and verified along the way

• Agent registration and authentication
• At-least-once task delivery
• Lease-based task claims
• Worker heartbeats
• Task recovery after worker failure
• Idempotent terminal requests
• PostgreSQL persistence
• Docker containerization
• Kubernetes Deployment, Service, Secret, and PVC
• Automated pytest verification
• Git SHA-based Docker image tagging
• Test-gated CI/CD
• End-to-end deployment verification

The lesson that mattered most
A successful Kubernetes rollout does not mean an application is actually working. kubectl rollout status can go green while the app underneath is silently broken — a bad config, a missing secret, a dependency that never connected.
So the pipeline doesn't stop at "the pods are Running." After deployment, it port-forwards the service and runs an application-level check against the dashboard — the same kind of check a human would do by opening the page and looking at it.

The pipeline, accurately
My first pass at diagramming this pipeline drew pytest, Docker build, and Kubernetes deployment as three parallel branches. That's not what actually happens. The real pipeline is sequential and test-gated: nothing gets built or deployed unless the tests pass first.

Developer
↓
Git Repository (push)
↓
GitHub Actions
↓
job: test
├── setup Python 3.11 + uv
├── Postgres 16-alpine (service container)
└── uv run pytest -q
↓ (needs: test → only proceeds if tests pass)
job: build-and-deploy
├── Docker build (image tagged with Git SHA)
└── deploy to Kind cluster
├── kubectl apply -f k8s/
├── kubectl set image
└── rollout status
↓
Agent Relay (FastAPI service, incl. dashboard)
↓
Verification: port-forward + curl → dashboard check

Why this matters beyond one project
The instinct to treat "deployed" as "done" is exactly the instinct that breaks pipelines at scale. Whether it's a Kubernetes rollout or a data/ML pipeline, the pattern is the same: infrastructure-level success (pod is Running, DAG task is green) is not application-level success (the thing actually does what it's supposed to do). Closing that gap with a real, purpose-built check — not just a health probe — is what turns "it deployed" into "it works."

I'm currently exploring how these same engineering principles — test gating, staged evolution, verifying at the application level rather than the infrastructure level — apply to larger Data Engineering and AI/LLM systems, where the failure modes are often even less visible than a crashed pod.

The repository contains the implementation, Kubernetes manifests, Docker configuration, automated tests, and the CI/CD workflow.

GitHub repo: https://github.com/ketut-garjita/agent-relay