ibraheembello

Posted on Nov 12

Deploying NestJS Microservices to Azure Container Apps

#azure #nestjs #microservices #docker

Introduction

After spending countless hours (and a few late nights) deploying a distributed notification system to Azure, I wanted to share my complete journey - including all the challenges, solutions, and lessons learned. This isn't just a tutorial; it's a real-world deployment story with all the gotchas you'll actually encounter.

What we'll cover:

🏗️ Setting up a microservices architecture with NestJS and Prisma
🐳 Building production-ready Docker images
☁️ Deploying to Azure Container Apps
🐛 Troubleshooting common deployment issues
✅ Testing and monitoring live services

Live Demo:

🔗 User Service API Docs
🔗 Template Service API Docs

📊 The Architecture

We're building a distributed notification system with the following components:

Services

User Service - Handles user authentication, authorization, and profile management
Template Service - Manages notification templates with variable substitution
(Coming soon: Notification Service, Worker Service, Analytics Service)

Tech Stack

Backend: NestJS (Node.js 20)
ORM: Prisma 5.22
Database: PostgreSQL 15 (Azure Flexible Server)
Container Platform: Azure Container Apps
Container Registry: Azure Container Registry
Region: UK South (London)

Why Azure Container Apps?

Azure Container Apps hit the sweet spot for microservices:

✅ Fully managed - No Kubernetes complexity
✅ Auto-scaling - Scale to zero or scale out based on load
✅ Built-in HTTPS - Automatic SSL certificates
✅ Simple deployments - Deploy from container registries
✅ Cost-effective - Pay only for what you use

🏗️ Project Structure

distributed-notification-system/
├── services/
│   ├── user-service/
│   │   ├── src/
│   │   ├── prisma/
│   │   │   └── schema.prisma
│   │   ├── Dockerfile
│   │   ├── package.json
│   │   └── .env
│   └── template-service/
│       ├── src/
│       ├── prisma/
│       │   └── schema.prisma
│       ├── Dockerfile
│       ├── package.json
│       └── .env
└── infrastructure/
    └── azure/

🚦 Step 1: Building the Services

User Service - Key Features

Authentication & Authorization:

// src/auth/auth.service.ts
@Injectable()
export class AuthService {
  constructor(
    private prisma: PrismaService,
    private jwtService: JwtService,
  ) {}

  async register(dto: RegisterDto) {
    const hashedPassword = await bcrypt.hash(dto.password, 10);

    const user = await this.prisma.user.create({
      data: {
        email: dto.email,
        password: hashedPassword,
        first_name: dto.first_name,
        last_name: dto.last_name,
      },
    });

    const tokens = await this.generateTokens(user);
    return { user, ...tokens };
  }

  async login(dto: LoginDto) {
    const user = await this.prisma.user.findUnique({
      where: { email: dto.email },
    });

    if (!user || !(await bcrypt.compare(dto.password, user.password))) {
      throw new UnauthorizedException('Invalid credentials');
    }

    return this.generateTokens(user);
  }

  private async generateTokens(user: User) {
    const payload = { sub: user.id, email: user.email };

    return {
      access_token: await this.jwtService.signAsync(payload, {
        expiresIn: '15m',
      }),
      refresh_token: await this.jwtService.signAsync(payload, {
        expiresIn: '7d',
      }),
    };
  }
}

Prisma Schema:

// prisma/schema.prisma
generator client {
  provider = "prisma-client-js"
}

datasource db {
  provider = "postgresql"
  url      = env("DATABASE_URL")
}

model User {
  id              String           @id @default(uuid())
  email           String           @unique
  password        String
  first_name      String
  last_name       String
  is_active       Boolean          @default(true)
  created_at      DateTime         @default(now())
  updated_at      DateTime         @updatedAt
  preferences     UserPreference?
  push_tokens     PushToken[]

  @@map("users")
}

model UserPreference {
  id                    String   @id @default(uuid())
  user_id               String   @unique
  user                  User     @relation(fields: [user_id], references: [id], onDelete: Cascade)
  email_enabled         Boolean  @default(true)
  sms_enabled           Boolean  @default(false)
  push_enabled          Boolean  @default(true)
  timezone              String   @default("UTC")
  language              String   @default("en")
  created_at            DateTime @default(now())
  updated_at            DateTime @updatedAt

  @@map("user_preferences")
}

Template Service - Key Features

Template Rendering with Variables:

// src/templates/templates.service.ts
@Injectable()
export class TemplatesService {
  constructor(private prisma: PrismaService) {}

  async renderTemplate(id: string, variables: Record<string, any>) {
    const template = await this.prisma.template.findUnique({
      where: { id },
    });

    if (!template) {
      throw new NotFoundException('Template not found');
    }

    // Simple variable substitution
    let renderedSubject = template.subject;
    let renderedBody = template.body;

    Object.keys(variables).forEach((key) => {
      const regex = new RegExp(`{{\\s*${key}\\s*}}`, 'g');
      renderedSubject = renderedSubject.replace(regex, variables[key]);
      renderedBody = renderedBody.replace(regex, variables[key]);
    });

    return {
      subject: renderedSubject,
      body: renderedBody,
      template_id: template.id,
      template_name: template.name,
    };
  }
}

Usage Example:

// Create a template
POST /api/v1/templates
{
  "name": "welcome-email",
  "type": "email",
  "subject": "Welcome {{user_name}}!",
  "body": "Hello {{user_name}}, thanks for joining {{app_name}}!",
  "language": "en"
}

// Render with variables
POST /api/v1/templates/:id/render
{
  "variables": {
    "user_name": "John Doe",
    "app_name": "NotificationApp"
  }
}

// Result:
{
  "subject": "Welcome John Doe!",
  "body": "Hello John Doe, thanks for joining NotificationApp!"
}

🐳 Step 2: Dockerizing the Services

The Challenge: Alpine Linux vs Debian

First Attempt (Failed):

# ❌ This DOESN'T work with Prisma!
FROM node:20-alpine AS builder
WORKDIR /app
COPY package*.json ./
RUN npm ci
COPY . .
RUN npx prisma generate
RUN npm run build

FROM node:20-alpine
WORKDIR /app
COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/dist ./dist
CMD ["node", "dist/main.js"]

The Error:

PrismaClientInitializationError: Unable to require(`libquery_engine-linux-musl.so.node`)
Error loading shared library libssl.so.1.1: No such file or directory

The Problem:

Alpine Linux uses musl instead of glibc
Latest Alpine (3.22) removed openssl1.1-compat package
Prisma's query engine needs OpenSSL 1.1 or 3.x

The Solution: Use Debian Slim

# ✅ This works perfectly!
FROM node:20-slim AS builder

# Install OpenSSL and other dependencies
RUN apt-get update -y && \
    apt-get install -y openssl ca-certificates && \
    rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY package*.json ./
COPY prisma ./prisma/

RUN npm ci

COPY . .

RUN npx prisma generate

RUN npm run build

# Production stage
FROM node:20-slim

RUN apt-get update -y && \
    apt-get install -y openssl ca-certificates && \
    rm -rf /var/lib/apt/lists/*

WORKDIR /app

COPY --from=builder /app/node_modules ./node_modules
COPY --from=builder /app/dist ./dist
COPY --from=builder /app/prisma ./prisma
COPY --from=builder /app/package*.json ./

EXPOSE 3001

CMD ["node", "dist/main.js"]

Why Debian Slim?

✅ OpenSSL 3.x included by default
✅ Compatible with Prisma
✅ Still relatively small (~200MB vs Alpine's ~50MB)
✅ Most stable for production

☁️ Step 3: Azure Infrastructure Setup

Creating Azure Resources

#!/bin/bash

# Variables
RESOURCE_GROUP="notification-system-rg"
LOCATION="uksouth"
ACR_NAME="notificationsystemacr"
ENV_NAME="notification-env"

# 1. Create Resource Group
az group create \
  --name $RESOURCE_GROUP \
  --location $LOCATION

# 2. Create Azure Container Registry
az acr create \
  --resource-group $RESOURCE_GROUP \
  --name $ACR_NAME \
  --sku Basic \
  --location $LOCATION \
  --admin-enabled true

# 3. Create PostgreSQL for User Service
az postgres flexible-server create \
  --name user-service-db \
  --resource-group $RESOURCE_GROUP \
  --location $LOCATION \
  --admin-user dbadmin \
  --admin-password "YourSecurePassword123" \
  --sku-name Standard_B1ms \
  --tier Burstable \
  --version 15 \
  --storage-size 32 \
  --public-access 0.0.0.0

# 4. Create database
az postgres flexible-server db create \
  --resource-group $RESOURCE_GROUP \
  --server-name user-service-db \
  --database-name user_service_db

# 5. Create PostgreSQL for Template Service
az postgres flexible-server create \
  --name template-service-db \
  --resource-group $RESOURCE_GROUP \
  --location $LOCATION \
  --admin-user dbadmin \
  --admin-password "YourSecurePassword123" \
  --sku-name Standard_B1ms \
  --tier Burstable \
  --version 15 \
  --storage-size 32 \
  --public-access 0.0.0.0

az postgres flexible-server db create \
  --resource-group $RESOURCE_GROUP \
  --server-name template-service-db \
  --database-name template_service_db

# 6. Configure firewall rules
az postgres flexible-server firewall-rule create \
  --resource-group $RESOURCE_GROUP \
  --name user-service-db \
  --rule-name AllowAzureServices \
  --start-ip-address 0.0.0.0 \
  --end-ip-address 0.0.0.0

az postgres flexible-server firewall-rule create \
  --resource-group $RESOURCE_GROUP \
  --name template-service-db \
  --rule-name AllowAzureServices \
  --start-ip-address 0.0.0.0 \
  --end-ip-address 0.0.0.0

# 7. Create Container Apps Environment
az containerapp env create \
  --name $ENV_NAME \
  --resource-group $RESOURCE_GROUP \
  --location $LOCATION

echo "✅ Infrastructure setup complete!"

🚀 Step 4: Database Migrations

Important: Run migrations BEFORE deploying containers!

# User Service migrations
cd services/user-service
DATABASE_URL="postgresql://dbadmin:YourSecurePassword123@user-service-db.postgres.database.azure.com:5432/user_service_db?sslmode=require" \
  npx prisma migrate deploy

# Template Service migrations
cd services/template-service
DATABASE_URL="postgresql://dbadmin:YourSecurePassword123@template-service-db.postgres.database.azure.com:5432/template_service_db?sslmode=require" \
  npx prisma migrate deploy

📦 Step 5: Building and Pushing Docker Images

# Login to ACR
az acr login --name notificationsystemacr

# Build and push User Service
cd services/user-service
docker build -t notificationsystemacr.azurecr.io/user-service:latest .
docker push notificationsystemacr.azurecr.io/user-service:latest

# Build and push Template Service
cd services/template-service
docker build -t notificationsystemacr.azurecr.io/template-service:latest .
docker push notificationsystemacr.azurecr.io/template-service:latest

🎯 Step 6: Deploying Container Apps

# Get ACR credentials
ACR_PASSWORD=$(az acr credential show \
  --name notificationsystemacr \
  --query "passwords[0].value" -o tsv)

# Generate JWT secrets
JWT_SECRET=$(openssl rand -base64 32)
JWT_REFRESH_SECRET=$(openssl rand -base64 32)

# Deploy User Service
az containerapp create \
  --name user-service-app \
  --resource-group notification-system-rg \
  --environment notification-env \
  --image notificationsystemacr.azurecr.io/user-service:latest \
  --registry-server notificationsystemacr.azurecr.io \
  --registry-username notificationsystemacr \
  --registry-password "$ACR_PASSWORD" \
  --target-port 3001 \
  --ingress external \
  --min-replicas 1 \
  --max-replicas 3 \
  --cpu 0.5 \
  --memory 1Gi \
  --env-vars \
    "DATABASE_URL=secretref:database-url" \
    "JWT_SECRET=secretref:jwt-secret" \
    "JWT_REFRESH_SECRET=secretref:jwt-refresh-secret" \
    "PORT=3001" \
    "NODE_ENV=production" \
  --secrets \
    "database-url=postgresql://dbadmin:YourSecurePassword123@user-service-db.postgres.database.azure.com:5432/user_service_db?sslmode=require" \
    "jwt-secret=$JWT_SECRET" \
    "jwt-refresh-secret=$JWT_REFRESH_SECRET"

# Deploy Template Service
az containerapp create \
  --name template-service-app \
  --resource-group notification-system-rg \
  --environment notification-env \
  --image notificationsystemacr.azurecr.io/template-service:latest \
  --registry-server notificationsystemacr.azurecr.io \
  --registry-username notificationsystemacr \
  --registry-password "$ACR_PASSWORD" \
  --target-port 3004 \
  --ingress external \
  --min-replicas 1 \
  --max-replicas 3 \
  --cpu 0.5 \
  --memory 1Gi \
  --env-vars \
    "DATABASE_URL=secretref:database-url" \
    "PORT=3004" \
    "NODE_ENV=production" \
  --secrets \
    "database-url=postgresql://dbadmin:YourSecurePassword123@template-service-db.postgres.database.azure.com:5432/template_service_db?sslmode=require"

🔥 Common Issues & Solutions

Issue 1: Containers Keep Crashing with OpenSSL Error

Symptoms:

PrismaClientInitializationError
Error loading shared library libssl.so.1.1

Root Cause: Alpine Linux 3.22+ doesn't have openssl1.1-compat

Solution: Switch to Debian Slim (see Dockerfile above)

Issue 2: Container Apps Not Using New Images

Symptoms: Updated code, pushed new images, but containers still running old code

Root Cause: Azure cached the :latest tag

Solution: Force new revision with unique suffix

TIMESTAMP=$(date +%s)

az containerapp update \
  --name user-service-app \
  --resource-group notification-system-rg \
  --image notificationsystemacr.azurecr.io/user-service:latest \
  --revision-suffix "v2-${TIMESTAMP}"

Issue 3: Database Connection Timeout

Symptoms:

Error: P1001: Can't reach database server

Solution:

Check firewall rules allow Azure services (0.0.0.0)
Verify connection string format includes ?sslmode=require
Wait 2-3 minutes after creating firewall rules

Issue 4: Rate Limiter Warning

Warning:

ValidationError: The 'X-Forwarded-For' header is set but Express 'trust proxy' is false

Solution: Add to main.ts:

async function bootstrap() {
  const app = await NestFactory.create(AppModule);

  // Trust Azure's proxy
  app.set('trust proxy', 1);

  await app.listen(3001);
}

✅ Verification & Testing

Health Checks

Both services expose health endpoints:

# User Service
curl https://user-service-app.blacksky-6bcbe9ee.uksouth.azurecontainerapps.io/api/v1/health

# Response:
{
  "status": "healthy",
  "timestamp": "2025-11-12T02:33:03.054Z",
  "service": "user-service",
  "version": "1.0.0",
  "uptime": 371.52,
  "database": {
    "status": "connected",
    "response_time_ms": 88
  },
  "memory": {
    "used_mb": 19,
    "total_mb": 21
  }
}

API Testing

1. Register a User:

curl -X POST \
  https://user-service-app.blacksky-6bcbe9ee.uksouth.azurecontainerapps.io/api/v1/auth/register \
  -H "Content-Type: application/json" \
  -d '{
    "email": "john@example.com",
    "password": "SecurePass123!",
    "first_name": "John",
    "last_name": "Doe"
  }'

2. Login:

curl -X POST \
  https://user-service-app.blacksky-6bcbe9ee.uksouth.azurecontainerapps.io/api/v1/auth/login \
  -H "Content-Type: application/json" \
  -d '{
    "email": "john@example.com",
    "password": "SecurePass123!"
  }'

3. Access Protected Endpoint:

curl -X GET \
  https://user-service-app.blacksky-6bcbe9ee.uksouth.azurecontainerapps.io/api/v1/users \
  -H "Authorization: Bearer YOUR_ACCESS_TOKEN"

4. Create a Template:

curl -X POST \
  https://template-service-app.blacksky-6bcbe9ee.uksouth.azurecontainerapps.io/api/v1/templates \
  -H "Content-Type: application/json" \
  -d '{
    "name": "welcome-email",
    "type": "email",
    "subject": "Welcome {{user_name}}!",
    "body": "Hello {{user_name}}, thanks for joining!",
    "language": "en"
  }'

📊 Monitoring & Metrics

Azure Portal Monitoring

Navigate to your Container App in Azure Portal
Click Metrics in the left menu
Add metrics:
- Requests - HTTP request count
- Replica Count - Number of running instances
- CPU Usage - Container CPU utilization
- Memory Usage - Container memory utilization

Application Insights (Optional)

Add Application Insights for detailed telemetry:

// src/main.ts
import { INestApplication } from '@nestjs/common';
import * as appInsights from 'applicationinsights';

async function bootstrap() {
  // Initialize Application Insights
  if (process.env.APPLICATIONINSIGHTS_CONNECTION_STRING) {
    appInsights.setup()
      .setAutoCollectRequests(true)
      .setAutoCollectPerformance(true)
      .setAutoCollectExceptions(true)
      .start();
  }

  const app = await NestFactory.create(AppModule);
  await app.listen(3001);
}

💰 Cost Optimization

Current Setup Costs (Approximate)

Resource	Tier	Monthly Cost
Container Apps Environment	Consumption	$0 (free tier)
User Service Container	0.5 vCPU, 1GB RAM	~$15-25
Template Service Container	0.5 vCPU, 1GB RAM	~$15-25
PostgreSQL User DB	Burstable B1ms	~$13
PostgreSQL Template DB	Burstable B1ms	~$13
Container Registry	Basic	~$5
Total		~$60-80/month

Optimization Tips

Scale to Zero for dev/staging:

az containerapp update \
  --name user-service-app \
  --resource-group notification-system-rg \
  --min-replicas 0 \
  --max-replicas 3

Use Spot Instances for non-critical workloads
Share PostgreSQL Server across multiple databases
Use Azure Reserved Instances for 30-40% savings

🎯 Results & Performance

Deployment Metrics

Build Time: ~3-4 minutes per service
Deployment Time: ~2-3 minutes per service
Cold Start: ~3-5 seconds
Response Time (P95): <100ms
Database Latency: ~80-120ms

Load Test Results

Using Apache Bench for basic load testing:

ab -n 10000 -c 100 \
  https://user-service-app.blacksky-6bcbe9ee.uksouth.azurecontainerapps.io/api/v1/health

Results:

Requests per second: ~450
Mean response time: 220ms
95th percentile: 380ms
Failed requests: 0

🔐 Security Best Practices

1. Secrets Management

❌ DON'T hardcode secrets:

const jwtSecret = 'my-secret-key'; // BAD!

✅ DO use environment variables:

const jwtSecret = process.env.JWT_SECRET;

✅ BETTER - Use Azure Key Vault:

az keyvault create \
  --name notification-kv \
  --resource-group notification-system-rg

az keyvault secret set \
  --vault-name notification-kv \
  --name jwt-secret \
  --value "your-secret"

2. Network Security

✅ Use private networking for databases
✅ Restrict database firewall to specific IP ranges
✅ Enable SSL/TLS for all connections
✅ Use managed identities instead of passwords

3. Container Security

# Run as non-root user
FROM node:20-slim
RUN groupadd -r nodejs && useradd -r -g nodejs nodejs
USER nodejs

# Use specific versions
FROM node:20.10.0-slim

# Scan for vulnerabilities
RUN npm audit fix

📚 Key Learnings

What Went Right ✅

Debian Slim over Alpine - Saved hours of OpenSSL troubleshooting
Migrations before deployment - Databases ready when containers start
Health checks - Essential for monitoring and debugging
Swagger documentation - Makes API testing effortless
Auto-scaling - Handles traffic spikes automatically

What I'd Do Differently 🔄

Use Infrastructure as Code (Terraform/Bicep) from day one
Set up CI/CD pipeline before manual deployments
Implement centralized logging (ELK/Datadog) earlier
Add integration tests for database connectivity
Use Azure Key Vault for secrets management

Challenges Overcome 💪

Alpine OpenSSL compatibility → Switched to Debian
Container caching issues → Used unique revision suffixes
Database connection timeouts → Configured firewall rules properly
Rate limiter warnings → Added Express trust proxy configuration

🚀 Next Steps

Phase 2: Additional Services

[ ] Notification Service - Email, SMS, and push notification delivery
[ ] Notification Worker - Queue processing with Bull/BullMQ
[ ] Analytics Service - Track delivery rates and engagement
[ ] Admin Dashboard - Service management UI

Phase 3: Enhancements

[ ] CI/CD Pipeline - GitHub Actions or Azure DevOps
[ ] Infrastructure as Code - Terraform or Bicep
[ ] API Gateway - Centralized routing and authentication
[ ] Message Queue - Redis or Azure Service Bus
[ ] Caching Layer - Redis for frequently accessed data
[ ] Monitoring - Application Insights, Prometheus, Grafana

🔗 Live Demo & Resources

Try It Yourself!

User Service:

🔗 API Documentation
🔗 Health Check

Template Service:

🔗 API Documentation
🔗 Health Check

Test User:

Email: test@example.com
Password: TestPass123!

Additional Resources

📖 Azure Container Apps Documentation
📖 NestJS Documentation
📖 Prisma Documentation
🐙 Source Code (Add your GitHub repo link)

💭 Final Thoughts

Deploying microservices to production is never as straightforward as tutorials make it seem. Between Docker compatibility issues, database connectivity problems, and platform-specific quirks, there's always something unexpected.

But that's also what makes it rewarding! Every issue solved is a lesson learned, and every deployment gets smoother.

Key Takeaways: