Building a Full-Stack Product Monitoring System: A Technical Deep Dive

Overview

This is a production-grade, full-stack application designed to monitor product availability on an e-commerce website. The system automatically tracks inventory changes, sends Discord notifications for new products and restocks, and provides a comprehensive admin dashboard for monitoring configuration.

This technical deep-dive explores the architecture, design decisions, and implementation strategies behind building a scalable monitoring system capable of handling thousands of products with real-time notifications.

System Architecture

High-Level Architecture

The application follows a modern hybrid architecture combining TypeScript and Go microservices:

┌─────────────────┐      ┌─────────────────┐      ┌──────────────────┐
│   React 19      │─────▶│   Fastify       │─────▶│  PostgreSQL 16   │
│   Frontend      │◀─────│   Backend (TS)  │◀─────│    Database      │
│   (Port 80)     │      │   (Port 4000)   │      │   (Port 5432)    │
└─────────────────┘      └─────────────────┘      └──────────────────┘
                               │         │
                               │         └────────▶┌──────────────────┐
                               │                   │  Go Checkout API │
                               │                   │    (Port 8080)   │
                               │                   │  ┌────────────┐  │
                               │                   │  │ TLS Client │  │
                               │                   │  │  Akamai    │  │
                               │                   │  └────────────┘  │
                               │                   └──────────────────┘
                               ▼
                         ┌──────────┐         ┌─────────────┐
                         │  Redis 7 │◀───────▶│   BullMQ    │
                         │(Port 6379)│        │ Job Queues  │
                         └──────────┘         └─────────────┘
                               │
                               ▼
                         ┌──────────┐
                         │  Discord │
                         │ Webhooks │
                         └──────────┘

Architecture Decisions

1. Go Microservice for Checkout (Critical Decision)

• Why Go instead of TypeScript? The checkout flow requires bypassing sophisticated bot protection (Akamai)
• TLS Fingerprinting: Modern websites fingerprint TLS handshakes to detect bots. Node.js's built-in HTTP client has a static TLS fingerprint that's easily detected and blocked
• bogdanfinn/tls-client: Go library that allows custom TLS fingerprints mimicking real browsers (Chrome 133 profile)
• Advanced HTTP/2 Features: Precise control over header ordering, pseudo-headers, and connection properties
• Hyper Solutions SDK: Commercial Akamai bypass solution with sensor data generation
• Result: Successfully bypasses bot detection while maintaining high performance and reliability

2. Fastify over Express

• Fastify was chosen for its superior performance (up to 2x faster than Express in benchmarks)
• Built-in schema validation with JSON Schema
• Native TypeScript support and excellent plugin ecosystem
• First-class async/await support without middleware complexity

3. BullMQ for Job Processing

• Reliable job queue with Redis backing for distributed task processing
• Built-in support for retries, delayed jobs, and repeatable patterns
• Priority queues for handling different notification types
• Persistent job storage prevents data loss during restarts

4. PostgreSQL with Zapatos

• Type-safe database queries without ORM overhead
• Direct SQL when needed with full TypeScript inference
• Automatic type generation from database schema
• Eliminates runtime type mismatches between DB and application

5. Redis for Caching and Queue Management

• Low-latency caching for frequently accessed data
• BullMQ job queue backing
• Session storage for authentication
• Pub/sub capabilities for real-time features

Backend Architecture

Application Structure

The backend follows a clean, modular architecture with separate TypeScript and Go services:

TypeScript Backend:

src/backend/
├── config/              # Environment and database configuration
├── controllers/         # Request handlers (thin layer)
├── services/           # Business logic layer
├── routes/             # API route definitions
├── middleware/         # Authentication and validation
├── jobs/               # Background job processing
│   ├── queues.ts      # BullMQ queue setup
│   └── workers.ts     # Job processors
└── server.ts          # Application entry point

Go Checkout Service:

cmd/api/
└── main.go             # Fiber HTTP server (Port 8080)

src/
├── checkout.go         # Main checkout flow orchestration
├── checkout_types.go   # Type definitions
├── akamai.go          # Akamai bot bypass implementation
├── headers.go         # Browser-like header generation
└── card.go            # Credit card validation

Dependency Injection Pattern

The application uses a centralized dependency management system to handle database and Redis connections:

// Simplified dependency management pattern
class AppDependencies {
  private static pool: Pool;
  private static redis: Redis;

  static async connect() {
    // Initialize connections
    await pool.query("SELECT 1");
    await redis.ping();
  }

  static get<T>(name: string): T {
    // Return requested dependency
  }
}

Benefits:

• Single source of truth for shared resources
• Easy testing with mock dependencies
• Graceful connection management and cleanup
• Type-safe dependency access

Authentication System

JWT-based authentication with bcrypt password hashing:

Key Features:

• Stateless authentication using JWT tokens
• 10-round bcrypt hashing for password security
• Cookie-based token storage with httpOnly flag
• Token expiration and refresh patterns
• Protected routes via middleware

Authentication Flow:

1. User submits credentials
2. Password verified against bcrypt hash
3. JWT token generated with user claims
4. Token stored in httpOnly cookie
5. Subsequent requests validated via middleware

Go Checkout Microservice

The checkout functionality is implemented as a separate Go microservice that communicates with the TypeScript backend via HTTP.

Architecture Pattern:

1. TypeScript backend receives checkout request from user
2. Backend validates task and assembles checkout payload
3. HTTP request sent to Go service at GO_API_URL
4. Go service performs checkout with TLS fingerprinting
5. Response returned to TypeScript backend
6. Backend updates database and sends notifications

Key Technologies:

Fiber Framework:

• High-performance web framework for Go (built on Fastify)
• Minimal API surface: /checkout, /checkout/:id, /status, /health
• Context-based request cancellation support

TLS Client (bogdanfinn/tls-client):

• Custom TLS fingerprinting to mimic Chrome 133
• Randomized TLS extension ordering
• Support for HTTP/2 with precise control
• Cookie jar management with domain-specific cookies

Akamai Bypass Implementation:

The Akamai client implements a sophisticated multi-step bot detection bypass:

1. Script Extraction: Parse Akamai bot detection script from HTML
2. Sensor Generation: Generate sensor data using Hyper Solutions SDK
3. Cookie Management: Track _abck and bm_sz cookies
4. SBSD Challenge Handling: Detect and solve secondary challenges automatically
5. IP Consistency: Maintain consistent IP address for session validation

Checkout Flow (src/checkout.go:39-118):

1. Generate visitor/visit IDs from Oracle tracking system
2. Set session cookies (_abck, bm_sz, AGEVERIFY)
3. Add product to cart with Akamai sensor
4. Add shipping address information
5. Validate inventory availability
6. Update cart items with pricing
7. Tokenize credit card via CardConnect
8. Submit order with payment details

Error Handling:

• Context cancellation for user-initiated aborts
• Automatic retry logic for transient failures
• Comprehensive error messages for debugging
• Graceful fallback when challenges fail

Security Features:

• Credit card tokenization via CardConnect (PCI compliance)
• Proxy rotation support for IP diversity
• No credential storage (stateless operation)
• Request validation and sanitization

Integration Pattern:

The TypeScript backend integrates via a CheckoutClient class (src/checkout.ts):

const response = await this.client.post<CheckoutResponse>(
  `/checkout`,
  requestPayload,
  {
    headers: { 'X-Checkout-ID': taskId }
  }
);

Benefits of This Hybrid Approach:

• TypeScript handles business logic, database, and orchestration
• Go handles performance-critical bot bypass operations
• Each service optimized for its specific use case
• Independent scaling and deployment
• Clear separation of concerns

API Design

RESTful API with clear resource-based routing:

Endpoints:

• /api/auth/* - Authentication (login, register, me)
• /api/filtered-products - Keyword-based monitoring
• /api/restock-products - Product restock tracking
• /api/webhooks - Discord webhook management
• /api/proxies - Proxy configuration
• /api/dashboard - Statistics and activity
• /api/products - Product search and details
• /api/user-tasks - User task management

API Patterns:

• Consistent error handling with proper HTTP status codes
• Request validation using JSON Schema
• Paginated responses for large datasets
• Filtering and search capabilities

Job Queue System

BullMQ Integration

The monitoring system uses BullMQ for handling asynchronous jobs with different priorities and schedules.

Job Types:

1. scrape-all (Scheduled)

   • Runs every 30 minutes
   • Fetches sitemap from target website
   • Identifies new products
   • Queues individual product scrape jobs

2. scrape-product (On-demand)

   • Scrapes individual product details
   • Updates product information in database
   • Triggers notifications for new products
   • Handles keyword matching for filtered alerts

3. monitor-restock (Periodic)

   • Checks specific products for stock changes
   • Compares current vs previous stock levels
   • Triggers restock notifications when inventory increases

4. alert (Immediate)

   • Sends Discord webhook notifications
   • Handles different notification types
   • Manages user-specific alerts
   • Supports batch notification delivery

Worker Architecture

Concurrency: 100 concurrent jobs for high throughput

Error Handling:

• Automatic retry with exponential backoff
• Failed job logging with full context
• Dead letter queue for persistent failures
• Monitoring via Bull Board UI

Graceful Shutdown:

// Graceful shutdown pattern
process.on('SIGTERM', async () => {
  await worker.close(); // Finish current jobs
  await redis.disconnect();
  await pool.end();
});

Database Design

Schema Architecture

Core Tables:

1. users - User accounts and authentication
2. products - Product catalog with inventory tracking
3. product_meta - Extended product information
4. filtered_products - Keyword-based monitoring rules
5. restock_products - Product restock subscriptions
6. user_webhooks - Discord webhook configurations
7. user_proxies - Proxy pool management
8. user_tasks - Custom user automation tasks

Migration Strategy

Using dbmate for database migrations:

Benefits:

• Version-controlled schema changes
• Up/down migration support
• Language-agnostic SQL migrations
• Simple CLI interface

Workflow:

1. Create migration: dbmate new add_feature
2. Write up/down SQL
3. Apply: dbmate up
4. Generate types: npx zapatos

Type-Safe Queries with Zapatos

Zapatos generates TypeScript types directly from the database schema:

// Type-safe queries
const products = await select(
  'products',
  { in_stock: conditions.gt(0) },
  { columns: ['id', 'name', 'price'] }
).run(pool);

// Full type inference - no manual typing needed
// products: Array<{ id: string, name: string, price: number }>

Advantages:

• Compile-time type checking
• Autocomplete for table names and columns
• Zero runtime overhead (generates SQL)
• Catches schema mismatches before deployment

Web Scraping & Monitoring

Scraping Architecture

Strategy: Polite scraping with proxy rotation and rate limiting

Key Components:

1. Sitemap Parsing

   • Fetches XML sitemap from target website
   • Extracts product URLs
   • Identifies new products not in database

2. Product Extraction

   • Uses Cheerio for HTML parsing
   • Extracts structured product data
   • Handles dynamic content and variations

3. Proxy Management

   • Random proxy selection per request
   • Health checking and rotation
   • User-specific proxy support
   • Fallback to direct requests

4. Rate Limiting

   • Configurable delays between requests
   • Queue-based throttling
   • Respectful of target server resources

Notification System

Discord Integration:

• Rich embeds with product images
• Multiple notification types (new products, restocks, filtered)
• User-specific webhook routing
• Batch notification support

Notification Types:

1. New Products - All newly discovered products
2. Filtered Products - Keyword-matched products for specific users
3. Restocks - Inventory increase alerts
4. Whiskey Release - Special release notifications

Frontend Architecture

React 19 with TypeScript

Technology Stack:

• React 19 with latest features
• TypeScript for type safety
• Vite for fast builds and HMR
• React Router for client-side routing

Component Structure

src/frontend/
├── components/         # Reusable UI components
│   └── Card.tsx       # Generic card component
├── pages/             # Route-based page components
│   ├── SignIn.tsx
│   ├── SignUp.tsx
│   ├── Dashboard.tsx
│   ├── NewFilteredProducts.tsx
│   ├── Restocks.tsx
│   ├── Webhooks.tsx
│   ├── Proxies.tsx
│   └── UserTasks.tsx
├── App.tsx            # Root component with routing
└── main.tsx           # Entry point

State Management

Approach: Local state with fetch-on-mount pattern

Rationale:

• Simple dashboard application without complex state
• Direct API calls from components
• Minimal state synchronization needs
• Reduced bundle size (no Redux/MobX)

Authentication Flow

1. Login form submission
2. API call with credentials
3. JWT token stored in httpOnly cookie
4. Protected routes check token validity
5. Automatic redirect to login if unauthorized

DevOps & Deployment

Docker Architecture

Multi-Container Setup:

• Frontend container (Nginx-served static files)
• Backend container (Node.js/TypeScript application)
• Go API container (Checkout microservice)
• Redis container (cache and job queue)
• PostgreSQL external (managed database)

Docker Compose Configuration:

• Service orchestration
• Health checks for dependencies
• Volume mounting for development
• Environment-based configuration

Development Workflow

Local Development:

# Terminal 1 - Backend with hot reload
yarn dev:backend

# Terminal 2 - Frontend with Vite HMR
yarn dev:frontend

Docker Development:

# Start all services
docker compose up -d

# View logs
docker compose logs -f backend

# Rebuild after changes
docker compose up -d --build

Security Considerations

Application Security:

• JWT tokens with expiration
• Bcrypt password hashing (10 rounds)
• SQL injection prevention via parameterized queries
• CORS configuration for API protection
• Webhook URL validation
• Input sanitization and validation

Infrastructure Security:

• Environment-based secrets (no hardcoded credentials)
• TLS/SSL for database connections
• Redis password authentication
• Docker network isolation
• Read-only volume mounts where appropriate

Performance Optimizations

Backend Optimizations

1. Connection Pooling

   • PostgreSQL connection pool for efficient DB access
   • Redis connection reuse across requests

2. Job Queue Concurrency

   • 100 concurrent workers for parallel processing
   • Bulk job enqueueing for efficiency

3. Database Indexing

   • Indexed foreign keys
   • Composite indexes for common queries
   • Unique constraints for fast lookups

4. Caching Strategy

   • Redis caching for frequent queries
   • LRU eviction policy
   • Configurable TTLs per data type

Frontend Optimizations

1. Build Optimization

   • Vite's optimized bundling
   • Tree shaking for smaller bundles
   • Code splitting for route-based loading

2. Asset Optimization

   • Lazy loading for images
   • Minified CSS and JavaScript
   • Gzip compression in production

Testing Strategy

Test Infrastructure

Framework: Jest with TypeScript support

Configuration:

• Separate test database
• Environment-based test configuration
• Long timeout for integration tests
• VM modules for ESM compatibility

Test Types

1. Unit Tests

   • Service layer business logic
   • Utility functions
   • Data transformations

2. Integration Tests

   • API endpoint testing
   • Database operations
   • Job queue processing

3. E2E Tests (Future)

   • User flows
   • Authentication scenarios
   • Critical paths

Monitoring & Observability

Logging

Structured Logging:

• JSON-formatted logs
• Contextual metadata (module, timestamp, error traces)
• Different log levels (debug, info, warn, error)
• Production vs development logging strategies

Job Queue Monitoring

Bull Board Integration:

• Web UI for queue visualization
• Real-time job status
• Failed job inspection
• Job retry management
• Performance metrics

Health Checks

Endpoint: /health

• Database connectivity check
• Redis availability check
• System timestamp
• Quick response for load balancers

Lessons Learned & Best Practices

What Worked Well

1. Type Safety Everywhere

   • Zapatos for database types
   • TypeScript across frontend and backend
   • Fewer runtime errors, better developer experience

2. Dependency Injection Pattern

   • Easy testing with mocked dependencies
   • Clean resource management
   • Single source of truth for connections

3. Job Queue Architecture

   • Decoupled processing from API requests
   • Reliable with built-in retries
   • Scalable horizontal processing

4. Docker for Development

   • Consistent environment across team
   • Easy onboarding for new developers
   • Production-like local setup

Challenges & Solutions

Challenge 1: Bot Detection and TLS Fingerprinting

• Problem: Node.js HTTP clients getting blocked by Akamai bot detection
• Root Cause: Static TLS fingerprints and predictable HTTP/2 characteristics
• Solution: Implemented Go microservice with bogdanfinn/tls-client for dynamic TLS fingerprinting
• Outcome: Successfully bypassed Akamai protection with Chrome 133 browser profile

Challenge 2: Akamai Sensor Data Generation

• Problem: Complex JavaScript-based bot detection requiring precise sensor payload
• Solution: Integrated Hyper Solutions SDK for commercial-grade Akamai bypass
• Implementation: Real-time sensor generation with context preservation across requests
• Result: Consistent checkout success rate with minimal detection

Challenge 3: Multi-Language Architecture

• Problem: Coordinating TypeScript and Go services with different paradigms
• Solution: Clean HTTP API contract with JSON payloads and typed interfaces
• Benefits: Each language handles what it does best (TS for orchestration, Go for performance)

Challenge 4: Rate Limiting

• Problem: Getting blocked by target website
• Solution: Proxy rotation and intelligent delays

Challenge 5: Database Type Sync

• Problem: Manual type definitions going stale
• Solution: Automated type generation with Zapatos

Challenge 6: Worker Failures

• Problem: Jobs failing silently
• Solution: Comprehensive error logging and Bull Board monitoring

Challenge 7: Docker Networking

• Problem: Services unable to communicate
• Solution: Proper network configuration and service discovery

Future Enhancements

Planned Features

1. Advanced Analytics

   • Price history tracking
   • Stock pattern analysis
   • Sell-out time predictions

2. User Preferences

   • Customizable notification frequency
   • Product category filters
   • Price threshold alerts

3. API Rate Limiting

   • Per-user rate limits
   • API key management
   • Usage analytics

4. Enhanced Testing

   • Comprehensive E2E test coverage
   • Load testing for scalability
   • Automated security scanning

Scaling Considerations

Horizontal Scaling:

• Stateless backend for easy replication
• Shared Redis for distributed locking
• Database read replicas for queries

Performance Improvements:

• GraphQL for efficient data fetching
• Server-side rendering for faster initial load
• CDN for static asset delivery

Conclusion

Building this monitoring system provided valuable insights into creating a production-grade application. The combination of modern technologies (Fastify, React 19, BullMQ, Zapatos, Go) with solid architectural patterns resulted in a maintainable, scalable application.

Key takeaways:

• Polyglot architecture works: Using TypeScript for business logic and Go for performance-critical operations proved highly effective
• TLS fingerprinting matters: Modern bot detection requires sophisticated bypass techniques that standard HTTP libraries can't provide
• Type safety reduces bugs: Both TypeScript and Go's type systems caught errors at compile time
• Job queues are essential: Asynchronous processing decouples operations and improves reliability
• Docker simplifies deployment: Multi-service orchestration becomes manageable with proper containerization
• Commercial SDKs have value: Hyper Solutions SDK saved weeks of reverse engineering Akamai protection
• Monitoring and observability are critical: Bull Board and structured logging enabled rapid debugging

The system successfully monitors thousands of products, delivers real-time notifications, performs automated checkouts while bypassing sophisticated bot detection, and provides a robust platform for future enhancements.

The hybrid TypeScript/Go architecture demonstrates that choosing the right tool for each job—even within the same application—leads to better outcomes than forcing a single technology to handle all requirements.

---

Tech Stack Summary

Frontend:

• React 19
• TypeScript
• Vite
• React Router

Backend (TypeScript):

• Fastify
• TypeScript
• Node.js
• Zapatos (type-safe DB queries)

Backend (Go):

• Fiber (web framework)
• Go 1.24.5
• bogdanfinn/tls-client (TLS fingerprinting)
• Hyper Solutions SDK (Akamai bypass)
• PuerkitoBio/goquery (HTML parsing)

Database & Cache:

• PostgreSQL 16
• Redis 7
• Zapatos (type-safe queries)

Job Processing:

• BullMQ
• Bull Board (monitoring)

Authentication & Security:

• JWT (jsonwebtoken)
• Bcrypt
• CardConnect (payment tokenization)

Infrastructure:

• Docker & Docker Compose
• Nginx (frontend serving)
• Multi-service orchestration

Development Tools:

• Jest (testing)
• dbmate (migrations)
• tsx (TypeScript execution)
• Go modules (dependency management)

---

Author: Isaac Sunday
Last Updated: November 2025