DEV Community: ibraheembello

Building a real-time anomaly detection engine for a self-hosted Nextcloud (HNG Stage 3)

ibraheembello — Sun, 26 Apr 2026 02:05:12 +0000

TL;DR - I built a Python daemon that watches every HTTP request hitting an Nginx reverse proxy, learns what normal traffic looks like in real time, and automatically bans IPs that misbehave by inserting iptables rules at the kernel level. It alerts to Slack and ships its live metrics on a public dashboard. This post explains every piece in beginner-friendly terms.

Live demo: http://cloud-ng-anomaly.duckdns.org
Source code: https://github.com/ibraheembello/hng-stage3-anomaly-detector

What the project does and why it matters

Imagine you run a public website, let's say, a self-hosted Google-Drive-style cloud storage. Random people visit it all day. Most are real users. Some are bots. A few are attackers trying to flood your server with fake requests until it falls over. That last category is called a DDoS attack - Distributed Denial of Service.

The traditional way to deal with this is reactive: when something breaks, an engineer wakes up at 3 a.m., diagnoses the issue, and manually blocks the offender. That's slow and painful.

The better way is proactive detection: watch the traffic continuously, learn what "normal" looks like, and automatically block the abnormal stuff before it brings the site down.

That's what this project does - for a Nextcloud cluster running behind Nginx.

It builds its own definition of "normal" from the last 30 minutes of real traffic, not a hardcoded number. It works whether you get 2 requests per second at 3 a.m. or 200 at peak. And when an attacker sends a burst, the detector reacts within a couple of seconds - adds a kernel-level firewall rule, pings me on Slack, and shows the ban on a dashboard.

The architecture in one picture

Four moving parts:

Component	What it does
Nginx	The doorman. Every request hits Nginx first; Nginx forwards it to Nextcloud. Crucially, it also writes a JSON line per request into a log file.
Nextcloud	The actual application. Doesn't know any of this is happening.
The detector daemon	A Python program that reads the log file as it grows. The brain of the operation.
iptables	The Linux kernel's firewall. The detector tells it "drop every packet from this IP on ports 80 and 443" and the kernel does the rest.

Everything runs in Docker containers, orchestrated by Docker Compose, on one Ubuntu VPS.

Step 1 - How the sliding window works

This is the heart of the detector. Skip to step 2 if data structures bore you, but trust me - this part is clever and simple.

A "rate" is a question: "how many requests has this IP made in the last 60 seconds?" If you naively count requests per minute by tallying them in 60-second buckets, you get the wrong answer when traffic falls right between two buckets - and an attacker can exploit that gap.

A sliding window answers the same question, but the 60-second period slides forward continuously with the clock. So at 12:00:30, "the last 60 seconds" means 11:59:30 to 12:00:30. At 12:00:31, it means 11:59:31 to 12:00:31. Always exactly the past 60 seconds, never a fixed bucket.

How do you implement that without re-counting every time? Use a deque - a "double-ended queue".

from collections import deque

# Per-IP and global windows: each one stores timestamps of recent requests
window = deque()
WINDOW_SECONDS = 60

def record(now: float):
    # 1. Drop everything older than 60s from the front
    while window and window[0] <= now - WINDOW_SECONDS:
        window.popleft()
    # 2. Append the new request to the back
    window.append(now)

def current_rate() -> float:
    return len(window) / WINDOW_SECONDS

That's it. Two operations per request: pop old entries off the front (O(1)), append the new one to the back (O(1)). The current rate is just len(window) / 60.

The detector keeps one such window per source IP and one global window across all traffic.

⚠️ The Stage 3 brief explicitly forbids "faking the sliding window with a per-minute counter." This deque pattern is the cheapest way to satisfy that requirement honestly.

Step 2 - How the baseline learns from traffic

The detector now knows the current request rate. But it has no idea whether 5 requests per second is a lot or a little for this site. To answer that, it needs a baseline - a model of normal traffic.

The baseline keeps a 30-minute history of per-second request counts. Every second, the count of requests in that second is appended to a rolling list. The list is bounded so anything older than 30 minutes is automatically forgotten.

Every 60 seconds, the baseline thread does this:

mean = statistics.fmean(per_second_counts)
stddev = statistics.stdev(per_second_counts)

That gives us two numbers describing the baseline:

mean - on average, how many requests does this site get per second?
stddev ("standard deviation") - how much does that number bounce around?

Per-hour-of-day slots

Here's a subtle wrinkle: traffic at 3 a.m. is naturally lower than at 3 p.m. If you used one universal baseline, the detector would think 3 p.m. traffic was an attack just because it's higher than the all-day average. Or it would miss a real attack at 3 a.m. because it's still below the all-day average.

The fix: keep 24 separate slots, one per hour of day. Once the slot for the current hour has at least 5 minutes of data, use that slot instead of the global rolling baseline. So 3 a.m. traffic is judged against other 3 a.m. traffic, and 3 p.m. against 3 p.m.

Floors

If traffic is genuinely zero for a while, mean and stddev both crash toward zero. The very next request would then have an infinite z-score (you can't divide by zero) and trigger a false alarm. Two cheap "floor" values prevent this:

mean_floor: 1.0
stddev_floor: 0.5

The mean is never allowed below 1.0; the stddev never below 0.5. Boring, but necessary.

Step 3 - How detection makes a decision

For every request that comes in, the detector asks two simultaneous questions:

Is this rate too many sigmas above the mean?
The "z-score" answers that: z = (rate - mean) / stddev.
If z > 3.0, the rate is more than three standard deviations away from normal - which statistically happens in about 0.3% of normal traffic. Almost certainly an anomaly.
Is this rate flat-out higher than 5× the mean?
This is a hard ceiling that catches steady, sustained floods even when stddev is wide. If the mean is 2 req/s, anything above 10 req/s is suspicious regardless of statistics.

If either fires, the IP (or the global stream) is anomalous. Whichever fires first wins.

z = (rate - mean) / max(stddev, 1e-9)
if z > 3.0 or rate > mean * 5.0:
    return "anomaly!"

There's one more wrinkle: an error surge. If an IP's 4xx/5xx response rate is at least 3× the baseline error rate, the detector tightens that IP's thresholds (multiplies them by 0.7). The intuition: an IP generating mostly errors is probing or scanning, and we want to ban it sooner.

Step 4 - How iptables is used to block an IP

Once the detector decides an IP is bad, the question is: how do you actually block it?

Linux kernels include a packet-filtering subsystem called netfilter. The user-space tool to talk to it is iptables. When you run

sudo iptables -I INPUT -p tcp -s 1.2.3.4 --dport 80 -j DROP

…the kernel installs a rule that says: "any TCP packet coming in on port 80 from 1.2.3.4 - drop it silently." The packet never even reaches Nginx. It's the lowest-level, fastest way to block traffic.

The blocker module shells out to iptables directly:

subprocess.run(["iptables", "-I", "INPUT", "-p", "tcp", "-s", ip,
                "--dport", str(port), "-j", "DROP"], check=True)

I scope the rule to TCP ports 80 and 443 only, not the whole IP. Why? So that when an admin's workstation accidentally trips the detector, the admin doesn't lose SSH at the same time. (Yes, I learned this the painful way.)

The ban isn't permanent - it's tiered:

schedule_seconds: [600, 1800, 7200]   # 10 min, 30 min, 2 h

The first ban lasts 10 minutes. If the same IP misbehaves again later, the second ban lasts 30 minutes. The third lasts 2 hours. The fourth is permanent. This back-off pattern keeps you from permanently banning a real user who had one bad minute, but punishes repeat offenders harder each time.

A small "unbanner" thread polls every 5 seconds for expired bans, removes the corresponding iptables rule, and pings Slack: "unbanned 1.2.3.4 (ban #1)".

Putting it all together

Here's a real audit log line generated during testing:

[2026-04-26T00:31:22Z] BAN 102.88.55.19 | z-score 3.03 > 3.00 | rate=2.52 | baseline=1.00 | duration=600s
[2026-04-26T00:41:27Z] UNBAN 102.88.55.19 | … | duration=released | reason=scheduled-release

I sent a burst of 400 HTTP requests from my home IP. The detector took ~30 seconds to process them, computed z-score = 3.03, banned my IP for 10 minutes, and notified Slack. Exactly 600 seconds later, the unbanner removed the rule and notified Slack again. Meanwhile, the dashboard was updating every 3 seconds, the baseline was learning from the burst (its stddev climbed to 6.6 and decayed back over the next 30 minutes), and a synthetic multi-IP attack later that hour fired the global detection path with rate 11.28 > mean 2.26 × 5.00.

Everything I described above ran without me intervening once.

What I would change next time

Production WSGI - Flask's built-in dev server runs the dashboard. For a real deployment I'd put it behind gunicorn.
TLS - I left HTTP-only because the brief didn't mandate it; in real life I'd terminate TLS at Nginx via certbot.
Persistence of the baseline - currently the baseline restarts cold on a daemon restart. Snapshotting the recent history to disk would let it warm up faster.
Adaptive backoff - the schedule is static. A repeat-offender weighting that takes the gap between offences into account would feel more humane.

But those are polish. The core mechanism - sliding windows + rolling baseline + z-score + iptables - works.

Try it yourself

git clone https://github.com/ibraheembello/hng-stage3-anomaly-detector.git
cd hng-stage3-anomaly-detector
cp .env.example _private/.env
$EDITOR _private/.env
docker compose --env-file _private/.env up -d --build

The README has a full from-scratch runbook, including how to install Docker on a fresh Ubuntu 24.04 box, how to point a free DuckDNS domain at the EC2 IP, and how to set up the Slack webhook.

If you build something on top of this, ping me - I'd love to see what you change.

Deploying NestJS Microservices to Azure Container Apps

ibraheembello — Wed, 12 Nov 2025 04:29:41 +0000

Introduction

After spending countless hours (and a few late nights) deploying a distributed notification system to Azure, I wanted to share my complete journey - including all the challenges, solutions, and lessons learned. This isn't just a tutorial; it's a real-world deployment story with all the gotchas you'll actually encounter.

What we'll cover:

🏗️ Setting up a microservices architecture with NestJS and Prisma
🐳 Building production-ready Docker images
☁️ Deploying to Azure Container Apps
🐛 Troubleshooting common deployment issues
✅ Testing and monitoring live services

Live Demo:

🔗 User Service API Docs
🔗 Template Service API Docs

📊 The Architecture

We're building a distributed notification system with the following components:

Services

User Service - Handles user authentication, authorization, and profile management
Template Service - Manages notification templates with variable substitution
(Coming soon: Notification Service, Worker Service, Analytics Service)

Tech Stack

Backend: NestJS (Node.js 20)
ORM: Prisma 5.22
Database: PostgreSQL 15 (Azure Flexible Server)
Container Platform: Azure Container Apps
Container Registry: Azure Container Registry
Region: UK South (London)

Why Azure Container Apps?

Azure Container Apps hit the sweet spot for microservices:

✅ Fully managed - No Kubernetes complexity
✅ Auto-scaling - Scale to zero or scale out based on load
✅ Built-in HTTPS - Automatic SSL certificates
✅ Simple deployments - Deploy from container registries
✅ Cost-effective - Pay only for what you use

🏗️ Project Structure

distributed-notification-system/
├── services/
│   ├── user-service/
│   │   ├── src/
│   │   ├── prisma/
│   │   │   └── schema.prisma
│   │   ├── Dockerfile
│   │   ├── package.json
│   │   └── .env
│   └── template-service/
│       ├── src/
│       ├── prisma/
│       │   └── schema.prisma
│       ├── Dockerfile
│       ├── package.json
│       └── .env
└── infrastructure/
    └── azure/

🚦 Step 1: Building the Services

User Service - Key Features

Authentication & Authorization:

// src/auth/auth.service.ts
@Injectable()
export class AuthService {
  constructor(
    private prisma: PrismaService,
    private jwtService: JwtService,
  ) {}

  async register(dto: RegisterDto) {
    const hashedPassword = await bcrypt.hash(dto.password, 10);

    const user = await this.prisma.user.create({
      data: {
        email: dto.email,
        password: hashedPassword,
        first_name: dto.first_name,
        last_name: dto.last_name,
      },
    });

    const tokens = await this.generateTokens(user);
    return { user, ...tokens };
  }

  async login(dto: LoginDto) {
    const user = await this.prisma.user.findUnique({
      where: { email: dto.email },
    });

    if (!user || !(await bcrypt.compare(dto.password, user.password))) {
      throw new UnauthorizedException('Invalid credentials');
    }

    return this.generateTokens(user);
  }

  private async generateTokens(user: User) {
    const payload = { sub: user.id, email: user.email };

    return {
      access_token: await this.jwtService.signAsync(payload, {
        expiresIn: '15m',
      }),
      refresh_token: await this.jwtService.signAsync(payload, {
        expiresIn: '7d',
      }),
    };
  }
}

Prisma Schema:

// prisma/schema.prisma
generator client {
  provider = "prisma-client-js"
}

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

model User {
  id              String           @id @default(uuid())
  email           String           @unique
  password        String
  first_name      String
  last_name       String
  is_active       Boolean          @default(true)
  created_at      DateTime         @default(now())
  updated_at      DateTime         @updatedAt
  preferences     UserPreference?
  push_tokens     PushToken[]

  @@map("users")
}

model UserPreference {
  id                    String   @id @default(uuid())
  user_id               String   @unique
  user                  User     @relation(fields: [user_id], references: [id], onDelete: Cascade)
  email_enabled         Boolean  @default(true)
  sms_enabled           Boolean  @default(false)
  push_enabled          Boolean  @default(true)
  timezone              String   @default("UTC")
  language              String   @default("en")
  created_at            DateTime @default(now())
  updated_at            DateTime @updatedAt

  @@map("user_preferences")
}

Template Service - Key Features

Template Rendering with Variables:

// src/templates/templates.service.ts
@Injectable()
export class TemplatesService {
  constructor(private prisma: PrismaService) {}

  async renderTemplate(id: string, variables: Record<string, any>) {
    const template = await this.prisma.template.findUnique({
      where: { id },
    });

    if (!template) {
      throw new NotFoundException('Template not found');
    }

    // Simple variable substitution
    let renderedSubject = template.subject;
    let renderedBody = template.body;

    Object.keys(variables).forEach((key) => {
      const regex = new RegExp(`{{\\s*${key}\\s*}}`, 'g');
      renderedSubject = renderedSubject.replace(regex, variables[key]);
      renderedBody = renderedBody.replace(regex, variables[key]);
    });

    return {
      subject: renderedSubject,
      body: renderedBody,
      template_id: template.id,
      template_name: template.name,
    };
  }
}

Usage Example:

// Create a template
POST /api/v1/templates
{
  "name": "welcome-email",
  "type": "email",
  "subject": "Welcome {{user_name}}!",
  "body": "Hello {{user_name}}, thanks for joining {{app_name}}!",
  "language": "en"
}

// Render with variables
POST /api/v1/templates/:id/render
{
  "variables": {
    "user_name": "John Doe",
    "app_name": "NotificationApp"
  }
}

// Result:
{
  "subject": "Welcome John Doe!",
  "body": "Hello John Doe, thanks for joining NotificationApp!"
}

🐳 Step 2: Dockerizing the Services

The Challenge: Alpine Linux vs Debian

First Attempt (Failed):

# ❌ This DOESN'T work with Prisma!
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npx prisma generate
RUN npm run build

FROM node:20-alpine
WORKDIR /app
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/dist ./dist
CMD ["node", "dist/main.js"]

The Error:

PrismaClientInitializationError: Unable to require(`libquery_engine-linux-musl.so.node`)
Error loading shared library libssl.so.1.1: No such file or directory

The Problem:

Alpine Linux uses musl instead of glibc
Latest Alpine (3.22) removed openssl1.1-compat package
Prisma's query engine needs OpenSSL 1.1 or 3.x

The Solution: Use Debian Slim

# ✅ This works perfectly!
FROM node:20-slim AS builder

# Install OpenSSL and other dependencies
RUN apt-get update -y && \
    apt-get install -y openssl ca-certificates && \
    rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY package*.json ./
COPY prisma ./prisma/

RUN npm ci

COPY . .

RUN npx prisma generate

RUN npm run build

# Production stage
FROM node:20-slim

RUN apt-get update -y && \
    apt-get install -y openssl ca-certificates && \
    rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/prisma ./prisma
COPY --from=builder /app/package*.json ./

EXPOSE 3001

CMD ["node", "dist/main.js"]

Why Debian Slim?

✅ OpenSSL 3.x included by default
✅ Compatible with Prisma
✅ Still relatively small (~200MB vs Alpine's ~50MB)
✅ Most stable for production

☁️ Step 3: Azure Infrastructure Setup

Creating Azure Resources

#!/bin/bash

# Variables
RESOURCE_GROUP="notification-system-rg"
LOCATION="uksouth"
ACR_NAME="notificationsystemacr"
ENV_NAME="notification-env"

# 1. Create Resource Group
az group create \
  --name $RESOURCE_GROUP \
  --location $LOCATION

# 2. Create Azure Container Registry
az acr create \
  --resource-group $RESOURCE_GROUP \
  --name $ACR_NAME \
  --sku Basic \
  --location $LOCATION \
  --admin-enabled true

# 3. Create PostgreSQL for User Service
az postgres flexible-server create \
  --name user-service-db \
  --resource-group $RESOURCE_GROUP \
  --location $LOCATION \
  --admin-user dbadmin \
  --admin-password "YourSecurePassword123" \
  --sku-name Standard_B1ms \
  --tier Burstable \
  --version 15 \
  --storage-size 32 \
  --public-access 0.0.0.0

# 4. Create database
az postgres flexible-server db create \
  --resource-group $RESOURCE_GROUP \
  --server-name user-service-db \
  --database-name user_service_db

# 5. Create PostgreSQL for Template Service
az postgres flexible-server create \
  --name template-service-db \
  --resource-group $RESOURCE_GROUP \
  --location $LOCATION \
  --admin-user dbadmin \
  --admin-password "YourSecurePassword123" \
  --sku-name Standard_B1ms \
  --tier Burstable \
  --version 15 \
  --storage-size 32 \
  --public-access 0.0.0.0

az postgres flexible-server db create \
  --resource-group $RESOURCE_GROUP \
  --server-name template-service-db \
  --database-name template_service_db

# 6. Configure firewall rules
az postgres flexible-server firewall-rule create \
  --resource-group $RESOURCE_GROUP \
  --name user-service-db \
  --rule-name AllowAzureServices \
  --start-ip-address 0.0.0.0 \
  --end-ip-address 0.0.0.0

az postgres flexible-server firewall-rule create \
  --resource-group $RESOURCE_GROUP \
  --name template-service-db \
  --rule-name AllowAzureServices \
  --start-ip-address 0.0.0.0 \
  --end-ip-address 0.0.0.0

# 7. Create Container Apps Environment
az containerapp env create \
  --name $ENV_NAME \
  --resource-group $RESOURCE_GROUP \
  --location $LOCATION

echo "✅ Infrastructure setup complete!"

🚀 Step 4: Database Migrations

Important: Run migrations BEFORE deploying containers!

# User Service migrations
cd services/user-service
DATABASE_URL="postgresql://dbadmin:YourSecurePassword123@user-service-db.postgres.database.azure.com:5432/user_service_db?sslmode=require" \
  npx prisma migrate deploy

# Template Service migrations
cd services/template-service
DATABASE_URL="postgresql://dbadmin:YourSecurePassword123@template-service-db.postgres.database.azure.com:5432/template_service_db?sslmode=require" \
  npx prisma migrate deploy

📦 Step 5: Building and Pushing Docker Images

# Login to ACR
az acr login --name notificationsystemacr

# Build and push User Service
cd services/user-service
docker build -t notificationsystemacr.azurecr.io/user-service:latest .
docker push notificationsystemacr.azurecr.io/user-service:latest

# Build and push Template Service
cd services/template-service
docker build -t notificationsystemacr.azurecr.io/template-service:latest .
docker push notificationsystemacr.azurecr.io/template-service:latest

🎯 Step 6: Deploying Container Apps

# Get ACR credentials
ACR_PASSWORD=$(az acr credential show \
  --name notificationsystemacr \
  --query "passwords[0].value" -o tsv)

# Generate JWT secrets
JWT_SECRET=$(openssl rand -base64 32)
JWT_REFRESH_SECRET=$(openssl rand -base64 32)

# Deploy User Service
az containerapp create \
  --name user-service-app \
  --resource-group notification-system-rg \
  --environment notification-env \
  --image notificationsystemacr.azurecr.io/user-service:latest \
  --registry-server notificationsystemacr.azurecr.io \
  --registry-username notificationsystemacr \
  --registry-password "$ACR_PASSWORD" \
  --target-port 3001 \
  --ingress external \
  --min-replicas 1 \
  --max-replicas 3 \
  --cpu 0.5 \
  --memory 1Gi \
  --env-vars \
    "DATABASE_URL=secretref:database-url" \
    "JWT_SECRET=secretref:jwt-secret" \
    "JWT_REFRESH_SECRET=secretref:jwt-refresh-secret" \
    "PORT=3001" \
    "NODE_ENV=production" \
  --secrets \
    "database-url=postgresql://dbadmin:YourSecurePassword123@user-service-db.postgres.database.azure.com:5432/user_service_db?sslmode=require" \
    "jwt-secret=$JWT_SECRET" \
    "jwt-refresh-secret=$JWT_REFRESH_SECRET"

# Deploy Template Service
az containerapp create \
  --name template-service-app \
  --resource-group notification-system-rg \
  --environment notification-env \
  --image notificationsystemacr.azurecr.io/template-service:latest \
  --registry-server notificationsystemacr.azurecr.io \
  --registry-username notificationsystemacr \
  --registry-password "$ACR_PASSWORD" \
  --target-port 3004 \
  --ingress external \
  --min-replicas 1 \
  --max-replicas 3 \
  --cpu 0.5 \
  --memory 1Gi \
  --env-vars \
    "DATABASE_URL=secretref:database-url" \
    "PORT=3004" \
    "NODE_ENV=production" \
  --secrets \
    "database-url=postgresql://dbadmin:YourSecurePassword123@template-service-db.postgres.database.azure.com:5432/template_service_db?sslmode=require"

🔥 Common Issues & Solutions

Issue 1: Containers Keep Crashing with OpenSSL Error

Symptoms:

PrismaClientInitializationError
Error loading shared library libssl.so.1.1

Root Cause: Alpine Linux 3.22+ doesn't have openssl1.1-compat

Solution: Switch to Debian Slim (see Dockerfile above)

Issue 2: Container Apps Not Using New Images

Symptoms: Updated code, pushed new images, but containers still running old code

Root Cause: Azure cached the :latest tag

Solution: Force new revision with unique suffix

TIMESTAMP=$(date +%s)

az containerapp update \
  --name user-service-app \
  --resource-group notification-system-rg \
  --image notificationsystemacr.azurecr.io/user-service:latest \
  --revision-suffix "v2-${TIMESTAMP}"

Issue 3: Database Connection Timeout

Symptoms:

Error: P1001: Can't reach database server

Solution:

Check firewall rules allow Azure services (0.0.0.0)
Verify connection string format includes ?sslmode=require
Wait 2-3 minutes after creating firewall rules

Issue 4: Rate Limiter Warning

Warning:

ValidationError: The 'X-Forwarded-For' header is set but Express 'trust proxy' is false

Solution: Add to main.ts:

async function bootstrap() {
  const app = await NestFactory.create(AppModule);

  // Trust Azure's proxy
  app.set('trust proxy', 1);

  await app.listen(3001);
}

✅ Verification & Testing

Health Checks

Both services expose health endpoints:

# User Service
curl https://user-service-app.blacksky-6bcbe9ee.uksouth.azurecontainerapps.io/api/v1/health

# Response:
{
  "status": "healthy",
  "timestamp": "2025-11-12T02:33:03.054Z",
  "service": "user-service",
  "version": "1.0.0",
  "uptime": 371.52,
  "database": {
    "status": "connected",
    "response_time_ms": 88
  },
  "memory": {
    "used_mb": 19,
    "total_mb": 21
  }
}

API Testing

1. Register a User:

curl -X POST \
  https://user-service-app.blacksky-6bcbe9ee.uksouth.azurecontainerapps.io/api/v1/auth/register \
  -H "Content-Type: application/json" \
  -d '{
    "email": "john@example.com",
    "password": "SecurePass123!",
    "first_name": "John",
    "last_name": "Doe"
  }'

2. Login:

curl -X POST \
  https://user-service-app.blacksky-6bcbe9ee.uksouth.azurecontainerapps.io/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{
    "email": "john@example.com",
    "password": "SecurePass123!"
  }'

3. Access Protected Endpoint:

curl -X GET \
  https://user-service-app.blacksky-6bcbe9ee.uksouth.azurecontainerapps.io/api/v1/users \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN"

4. Create a Template:

curl -X POST \
  https://template-service-app.blacksky-6bcbe9ee.uksouth.azurecontainerapps.io/api/v1/templates \
  -H "Content-Type: application/json" \
  -d '{
    "name": "welcome-email",
    "type": "email",
    "subject": "Welcome {{user_name}}!",
    "body": "Hello {{user_name}}, thanks for joining!",
    "language": "en"
  }'

📊 Monitoring & Metrics

Azure Portal Monitoring

Navigate to your Container App in Azure Portal
Click Metrics in the left menu
Add metrics:
- Requests - HTTP request count
- Replica Count - Number of running instances
- CPU Usage - Container CPU utilization
- Memory Usage - Container memory utilization

Application Insights (Optional)

Add Application Insights for detailed telemetry:

// src/main.ts
import { INestApplication } from '@nestjs/common';
import * as appInsights from 'applicationinsights';

async function bootstrap() {
  // Initialize Application Insights
  if (process.env.APPLICATIONINSIGHTS_CONNECTION_STRING) {
    appInsights.setup()
      .setAutoCollectRequests(true)
      .setAutoCollectPerformance(true)
      .setAutoCollectExceptions(true)
      .start();
  }

  const app = await NestFactory.create(AppModule);
  await app.listen(3001);
}

💰 Cost Optimization

Current Setup Costs (Approximate)

Resource	Tier	Monthly Cost
Container Apps Environment	Consumption	$0 (free tier)
User Service Container	0.5 vCPU, 1GB RAM	~$15-25
Template Service Container	0.5 vCPU, 1GB RAM	~$15-25
PostgreSQL User DB	Burstable B1ms	~$13
PostgreSQL Template DB	Burstable B1ms	~$13
Container Registry	Basic	~$5
Total		~$60-80/month

Optimization Tips

Scale to Zero for dev/staging:

az containerapp update \
  --name user-service-app \
  --resource-group notification-system-rg \
  --min-replicas 0 \
  --max-replicas 3

Use Spot Instances for non-critical workloads
Share PostgreSQL Server across multiple databases
Use Azure Reserved Instances for 30-40% savings

🎯 Results & Performance

Deployment Metrics

Build Time: ~3-4 minutes per service
Deployment Time: ~2-3 minutes per service
Cold Start: ~3-5 seconds
Response Time (P95): <100ms
Database Latency: ~80-120ms

Load Test Results

Using Apache Bench for basic load testing:

ab -n 10000 -c 100 \
  https://user-service-app.blacksky-6bcbe9ee.uksouth.azurecontainerapps.io/api/v1/health

Results:

Requests per second: ~450
Mean response time: 220ms
95th percentile: 380ms
Failed requests: 0

🔐 Security Best Practices

1. Secrets Management

❌ DON'T hardcode secrets:

const jwtSecret = 'my-secret-key'; // BAD!

✅ DO use environment variables:

const jwtSecret = process.env.JWT_SECRET;

✅ BETTER - Use Azure Key Vault:

az keyvault create \
  --name notification-kv \
  --resource-group notification-system-rg

az keyvault secret set \
  --vault-name notification-kv \
  --name jwt-secret \
  --value "your-secret"

2. Network Security

✅ Use private networking for databases
✅ Restrict database firewall to specific IP ranges
✅ Enable SSL/TLS for all connections
✅ Use managed identities instead of passwords

3. Container Security

# Run as non-root user
FROM node:20-slim
RUN groupadd -r nodejs && useradd -r -g nodejs nodejs
USER nodejs

# Use specific versions
FROM node:20.10.0-slim

# Scan for vulnerabilities
RUN npm audit fix

📚 Key Learnings

What Went Right ✅

Debian Slim over Alpine - Saved hours of OpenSSL troubleshooting
Migrations before deployment - Databases ready when containers start
Health checks - Essential for monitoring and debugging
Swagger documentation - Makes API testing effortless
Auto-scaling - Handles traffic spikes automatically

What I'd Do Differently 🔄

Use Infrastructure as Code (Terraform/Bicep) from day one
Set up CI/CD pipeline before manual deployments
Implement centralized logging (ELK/Datadog) earlier
Add integration tests for database connectivity
Use Azure Key Vault for secrets management

Challenges Overcome 💪

Alpine OpenSSL compatibility → Switched to Debian
Container caching issues → Used unique revision suffixes
Database connection timeouts → Configured firewall rules properly
Rate limiter warnings → Added Express trust proxy configuration

🚀 Next Steps

Phase 2: Additional Services

[ ] Notification Service - Email, SMS, and push notification delivery
[ ] Notification Worker - Queue processing with Bull/BullMQ
[ ] Analytics Service - Track delivery rates and engagement
[ ] Admin Dashboard - Service management UI

Phase 3: Enhancements

[ ] CI/CD Pipeline - GitHub Actions or Azure DevOps
[ ] Infrastructure as Code - Terraform or Bicep
[ ] API Gateway - Centralized routing and authentication
[ ] Message Queue - Redis or Azure Service Bus
[ ] Caching Layer - Redis for frequently accessed data
[ ] Monitoring - Application Insights, Prometheus, Grafana

🔗 Live Demo & Resources

Try It Yourself!

User Service:

🔗 API Documentation
🔗 Health Check

Template Service:

🔗 API Documentation
🔗 Health Check

Test User:

Email: test@example.com
Password: TestPass123!

Additional Resources

📖 Azure Container Apps Documentation
📖 NestJS Documentation
📖 Prisma Documentation
🐙 Source Code (Add your GitHub repo link)

💭 Final Thoughts

Deploying microservices to production is never as straightforward as tutorials make it seem. Between Docker compatibility issues, database connectivity problems, and platform-specific quirks, there's always something unexpected.

But that's also what makes it rewarding! Every issue solved is a lesson learned, and every deployment gets smoother.

Key Takeaways:

✅ Start with a solid local development setup
✅ Test Docker images thoroughly before deploying
✅ Use managed services (databases, registries) to reduce complexity
✅ Implement comprehensive health checks
✅ Monitor everything from day one
✅ Document your deployment process

If you found this helpful, give it a ❤️ and feel free to ask questions in the comments!

🤝 Connect With Me

💼 http://www.linkedin.com/in/ibraheem-bello-049b34287
🐙 https://github.com/ibraheembello
🐦 https://x.com/Officialibrosky
📧 belloibrahimolawale@gmail.com

Tags: #azure #nestjs #microservices #docker #prisma #typescript #devops #cloudcomputing #containerization #tutorial

Have questions or suggestions? Drop a comment below! 👇

GitHub Issue Monitor AI Agent with Mastra

ibraheembello — Sun, 02 Nov 2025 19:19:58 +0000

Building a GitHub Issue Monitor AI Agent with Mastra and Telex.im

How I built an intelligent AI agent to track repository issues and integrated it with Telex using the A2A protocol

🎯 The Challenge

As a developer, staying on top of GitHub issues across multiple repositories can be overwhelming. Whether you're maintaining open-source projects, contributing to others, or just keeping track of important discussions, manually checking each repository is time-consuming and inefficient.

I wanted to solve this by building an AI agent that could:

Monitor GitHub repositories for new and updated issues
Provide intelligent summaries of changes
Integrate seamlessly with communication platforms
Run automatically in the background

This is the story of how I built the GitHub Issue Monitor using Mastra and integrated it with Telex.im.

🛠️ Tech Stack

Here's what I used to build this project:

Mastra: AI agent framework for building and orchestrating intelligent agents
Telex.im: Communication platform with A2A (Agent-to-Agent) protocol support
Azure App Service: Cloud hosting for the agent
Node.js + TypeScript: Backend runtime and type safety
Octokit: Official GitHub API client
OpenAI GPT-4o-mini: Language model for intelligence

🏗️ Architecture Overview

The GitHub Issue Monitor consists of several key components working together seamlessly.

System Design

User Message (Telex) 
    ↓
JSON-RPC 2.0 Request
    ↓
Azure App Service (Node.js Server)
    ↓
Message Parser & Router
    ↓
Simple GitHub Agent
    ↓
GitHub API (Octokit)
    ↓
Format & Return
    ↓
A2A Response
    ↓
Telex.im Display

💻 Building the Agent

Step 1: Setting Up Mastra

I started by creating a Mastra agent with clear instructions:

import { Agent } from '@mastra/core/agent';
import { openai } from '@ai-sdk/openai';

export const githubIssueMonitorAgent = new Agent({
  name: 'GitHub Issue Monitor',
  instructions: `
    You are a GitHub Issue Monitor that tracks repository issues 
    and provides concise summaries with proper formatting.
  `,
  model: openai('gpt-4o-mini'),
});

Step 2: GitHub API Integration

Using Octokit, I built the core functionality:

import { Octokit } from 'octokit';

const octokit = new Octokit({
  auth: process.env.GITHUB_TOKEN,
});

// Fetch repository issues
const { data: issues } = await octokit.rest.issues.listForRepo({
  owner: 'facebook',
  repo: 'react',
  state: 'open',
  per_page: 10,
  sort: 'created',
  direction: 'desc',
});

Step 3: Direct Formatting for Reliability

After experimenting with LLM-based formatting, I found that direct JavaScript formatting was more reliable:

let summary = `📊 **GitHub Repository: ${owner}/${repo}**\n\n`;
summary += `Total open issues: **${issues.length}**\n`;
summary += `Showing 5 most recent:\n\n`;

recentIssues.forEach((issue, index) => {
  summary += `${index + 1}. **#${issue.number}**: ${issue.title}\n`;
  summary += `   👤 @${issue.user?.login}`;
  if (labels) {
    summary += ` | 🏷️ ${labels}`;
  }
  summary += `\n   🔗 ${issue.html_url}\n\n`;
});

Why direct formatting?

✅ Faster (no LLM API call)
✅ More reliable (predictable output)
✅ Cost-effective (no extra tokens)
✅ Easier to debug

Step 4: Implementing A2A Protocol

The key challenge was supporting Telex's JSON-RPC 2.0 format:

// Parse JSON-RPC request
if (data.jsonrpc === '2.0' && data.params) {
  const textPart = data.params.message?.parts?.find(
    (p: any) => p.kind === 'text'
  );
  userMessage = textPart?.text;
}

// Return A2A-compliant response
const a2aResponse = {
  kind: 'message',
  role: 'assistant',
  parts: [{
    kind: 'text',
    text: formattedSummary
  }]
};

// Wrap in JSON-RPC format
res.end(JSON.stringify({
  jsonrpc: '2.0',
  id: data.id,
  result: a2aResponse
}));

Step 5: Deploying to Azure

Deployment was streamlined with GitHub Actions:

Push code to GitHub
Azure detects changes
Runs npm install and npm run build
Deploys to App Service
Restarts server automatically

Environment variables (configured in Azure):

OPENAI_API_KEY=sk-proj-...
GITHUB_TOKEN=ghp_...
GITHUB_OWNER=facebook
GITHUB_REPO=react
CHECK_INTERVAL_MINUTES=30
PORT=8080

🎨 Features & Output

Example Interaction

User: Check facebook/react

Agent:

📊 **GitHub Repository: facebook/react**

Total open issues: **847**
Showing 5 most recent:

1. **#35033**: Typo in comment: "untill" → "until" in ReactDOMRoot.js
   👤 @Ashish-coder-gif | 🏷️ Status: Unconfirmed
   🔗 https://github.com/facebook/react/issues/35033

2. **#35032**: chore(react-reconciler/README.md): replace older Medium links
   👤 @onstash | 🏷️ CLA Signed
   🔗 https://github.com/facebook/react/pull/35032

3. **#35029**: [Compiler Bug]: set-state-in-effect false negative
   👤 @mdwyer6 | 🏷️ Type: Bug, Status: Unconfirmed
   🔗 https://github.com/facebook/react/issues/35029

4. **#35028**: Bug: state changes inside forwardRef change props object
   👤 @Hypnosphi | 🏷️ Status: Unconfirmed
   🔗 https://github.com/facebook/react/issues/35028

5. **#35027**: test: add coverage for zero-length style values
   👤 @Biki-das | 🏷️ CLA Signed
   🔗 https://github.com/facebook/react/pull/35027

💡 *Ask for specific labels or date ranges to filter results*

🚧 Challenges & Solutions

Challenge 1: Empty LLM Responses

Problem: Initial implementation using Mastra tools returned empty responses.

Root Cause: The tool wasn't being invoked properly, and the LLM sometimes returned empty text.

Solution:

Added toolChoice: 'required' to force tool usage
Eventually replaced with direct formatting (more reliable)

Challenge 2: A2A Protocol Format

Problem: Telex uses JSON-RPC 2.0 with nested structures:

{
  "jsonrpc": "2.0",
  "params": {
    "message": {
      "parts": [{"kind": "text", "text": "..."}]
    }
  }
}

Solution: Created flexible message parser supporting multiple formats:

// Support JSON-RPC
if (data.jsonrpc === '2.0') {
  messageData = data.params;
}

// Support standard messages
if (data.messages) {
  userMessage = data.messages[data.messages.length - 1];
}

// Support parts array
if (data.parts) {
  const textPart = data.parts.find(p => p.kind === 'text');
  userMessage = textPart?.text;
}

Challenge 3: ES Module Imports

Problem: TypeScript compiles to .js but imports need explicit extensions:

// This fails at runtime
import { tool } from './tools/github-tool';

// This works
import { tool } from './tools/github-tool.js';

Solution: Added .js extensions to all relative imports in TypeScript files.

Challenge 4: Build Errors on Azure

Problem: Azure deployment failed due to TypeScript errors in test files.

Solution: Updated tsconfig.json to exclude test files:

{
  "exclude": ["node_modules", "dist", "src/**/*.test.ts", "src/**/test-*.ts"]
}

📊 Performance & Metrics

Response Times:

GitHub API call: ~500ms
Formatting: <50ms
Total response: ~2-3 seconds

Reliability:

Uptime: 99.9%
Success rate: 100% (with proper error handling)
Average requests/day: ~50 (testing phase)

Cost:

Azure App Service B1: $13/month
GitHub API: Free (5,000 requests/hour)
OpenAI (minimal): ~$0.10/month (mostly unused now)

🎓 Key Learnings

1. Simplicity Over Complexity

My first instinct was to use LLMs for everything. But sometimes plain JavaScript is better:

More predictable
Faster
Cheaper
Easier to debug

2. Protocol Compliance Matters

Understanding the A2A protocol and JSON-RPC 2.0 was crucial. Reading the spec saved hours of debugging.

3. Azure is Developer-Friendly

The GitHub integration and automatic deployments made iteration fast. Environment variables through the portal are convenient.

4. Error Handling is Critical

Production agents need robust error handling:

try {
  const issues = await fetchIssues();
  return formatResponse(issues);
} catch (error) {
  console.error('[Agent] Error:', error.message);
  return `Error: ${error.message}`;
}

5. Logging is Your Friend

Detailed logs at every step made debugging on Azure much easier:

console.log('[Simple Agent] Fetching issues for', owner, repo);
console.log('[Simple Agent] Fetched', issues.length, 'issues');
console.log('[Simple Agent] Formatted summary (', summary.length, 'chars)');

🚀 Future Enhancements

Near-term:

Advanced filtering by labels, milestones, assignees
Custom formatting options (compact, detailed, etc.)
Multi-repository monitoring in single query
Caching to reduce API calls

Long-term:

Real-time notifications for important changes
Analytics dashboard with trends and insights
Smart summaries using LLM for complex analysis
Integration with other Git platforms (GitLab, Bitbucket)

📚 Technical Stack Details

Backend:

Node.js 20 LTS
TypeScript 5.x
Express-style HTTP server
ES Modules

APIs & SDKs:

@mastra/core: ^0.1.x
octokit: ^3.x
@ai-sdk/openai: ^0.x
dotenv: ^16.x

Deployment:

Platform: Azure App Service
OS: Linux
Region: West Europe
Tier: Basic B1
Auto-deploy: GitHub Actions

💡 Tips for Building Your Own

If you want to build something similar:

Start with curl - Test your agent locally before deploying
Read the A2A spec - Understanding the protocol saves time
Use environment variables - Never hardcode API keys
Test edge cases - Empty responses, API errors, invalid inputs
Monitor logs - Set up proper logging from day one
Keep it simple - Start basic, add complexity when needed

🤝 Open Source & Community

This project is open source! You can:

View the code
Report issues
Contribute features
Fork and customize

GitHub Repository: [https://github.com/ibraheembello/GitHub-Issue-Monitor.git]

Contributions welcome for:

Bug fixes
New features
Documentation improvements
Test coverage
Performance optimizations

🎯 Conclusion

Building this GitHub Issue Monitor taught me valuable lessons about AI agent development, protocol integration, and production deployment. The combination of Mastra's framework and Telex's A2A protocol makes it surprisingly easy to build powerful, communicative AI agents.

Whether you're tracking open-source projects, managing team workflows, or experimenting with AI agents, I hope this post inspires you to build something cool!

Try the agent in Telex.im and let me know what you think!

📞 Connect & Follow

Twitter/X: [https://x.com/Officialibrosky]
GitHub: [https://github.com/ibraheembello]
LinkedIn: [www.linkedin.com/in/ibraheem-bello-049b34287]

Built with ❤️ using Mastra, Telex.im, and Azure

#AI #Mastra #Telex #GitHub #AgentDevelopment #A2A #Azure #NodeJS #TypeScript #OpenSource

Found this helpful? ⭐ Star the repo and share with your network!
: A Complete A2A Integration Guide