Mayuresh Smita Suresh

Posted on Feb 3

High concurrency live Auction platform System Architecture

#rust #ovenmediaengine #softwaredevelopment #programming

The Problem: The "Cloud Bill" Trap

When building a live-streaming auction app like Whatnot, the default advice is always: "Just use AWS IVS or Mux."

It works great, until you do the math.

For a single 1-hour event with 10,000 viewers, AWS IVS (Interactive Video Service) charges based on "minutes watched."

10,000 viewers × 60 minutes = 600,000 minutes
Cost per 1,000 minutes = ~$1.50
Total Cost = $900 (₹75,000 INR) per hour

If you run one event a day, your monthly bill is ₹22 Lakhs ($27,000). For a bootstrapped startup, this is a death sentence.

The Solution: The "Hybrid" Bare Metal Stack

We optimized this by moving away from "Pay-Per-Minute" (PaaS) to "Pay-Per-Bandwidth" (Infrastructure as a Service).

By switching to a self-hosted OvenMediaEngine (OME) instance on a dedicated server with unmetered bandwidth, we dropped the cost from $27,000/mo to $300/mo.

The High-Level Architecture

We use a Hybrid Approach:

Video (Heavy Lifting): Hosted on a massive Bare Metal server (MilesWeb/Hetzner) using OvenMediaEngine
Logic (The Brain): A lightweight Rust backend handling Auth, Payments, and the Auction Timer
Real-Time (The Glue): WebSockets (Rust) for instant bidding, completely separate from the video stream

graph TD
    User[Mobile App User]
    Streamer[Live Seller]

    subgraph "Bare Metal Data Center"
      OME[OvenMediaEngine Video Server]
    end

    subgraph "Cloud API - DigitalOcean"
      Rust[Rust API & WebSocket Server]
      DB[(Postgres & Redis)]
    end

    Streamer -- "RTMP (OBS/Mobile)" --> OME
    OME -- "Webhook Validation" --> Rust
    Rust -- "200 OK" --> OME

    OME -- "LL-HLS (Video)" --> User
    User -- "WebSocket Bid (Data)" --> Rust
    Rust -- "Broadcast Winner" --> User

Figure 1: Decoupled Video and Logic Architecture

Step 1: The Video Engine (OvenMediaEngine)

We chose OvenMediaEngine (OME) because it is open-source (C++) and supports Sub-Second Latency via LL-HLS (Low Latency HLS) and WebRTC.

Docker Configuration

We deploy OME using Docker on a MilesWeb Dedicated Server (32 Cores, 30TB Bandwidth).

# docker-compose.yml
version: "3.8"
services:
  ome:
    image: airensoft/ovenmediaengine:latest
    container_name: ome
    ports:
      - "1935:1935"    # RTMP Ingest
      - "8080:8080"    # LL-HLS Playback
      - "3333:3333"    # WebRTC Signaling
      - "3334:3334/udp" # WebRTC ICE
    environment:
      - OME_HOST_IP=${SERVER_PUBLIC_IP}
    volumes:
      - ./conf/Server.xml:/opt/ovenmediaengine/bin/origin_conf/Server.xml
    restart: always

The "Secret Sauce" Config (Server.xml)

To get the latency down to 3 seconds (acceptable for auctions if the timer is synced separately), we tune the LL-HLS settings.

<Application>
    <n>auction</n>
    <Providers>
        <RTMP />
    </Providers>
    <Publishers>
        <LLHLS>
            <ChunkDuration>0.5</ChunkDuration>
            <SegmentDuration>6</SegmentDuration>
            <SegmentCount>3</SegmentCount>
            <CrossDomains>
                <Url>*</Url>
            </CrossDomains>
        </LLHLS>
        <WebRTC>
            <Timeout>30000</Timeout>
        </WebRTC>
    </Publishers>
</Application>

Step 2: The Rust "Brain" (Stream Authentication)

You don't want just anyone streaming to your server. We use OME's Webhook feature to validate streamers against our database.

When a user starts streaming to rtmp://server/app/stream_key, OME hits our Rust API.

// src/routes/webhook.rs
use axum::{extract::Json, response::IntoResponse, http::StatusCode};
use serde::{Deserialize, Serialize};

#[derive(Deserialize)]
pub struct OmeHook {
    name: String,      // This is the stream_key
    app_name: String,
}

#[derive(Serialize)]
pub struct OmeResponse {
    allowed: bool,
    reason: Option<String>,
}

pub async fn on_publish(Json(payload): Json<OmeHook>) -> impl IntoResponse {
    // 1. Verify Stream Key in Redis/Postgres
    // let is_valid = verify_stream_key(&payload.name).await;
    let is_valid = true; // Mock for demo

    if is_valid {
        println!("✅ Stream authorized: {}", payload.name);
        return (
            StatusCode::OK,
            Json(OmeResponse {
                allowed: true,
                reason: None
            })
        );
    }

    println!("❌ Stream rejected: {}", payload.name);
    (
        StatusCode::FORBIDDEN,
        Json(OmeResponse {
            allowed: false,
            reason: Some("Invalid Stream Key".to_string())
        })
    )
}

Step 3: The Frontend (SolidStart Player)

On the client side (SolidStart), we use Hls.js optimized for low latency. We don't wait for the video to catch up; the Bidding Interface connects directly to the Rust WebSocket for real-time state.

// components/VideoPlayer.tsx
import { onMount, onCleanup } from "solid-js";
import Hls from "hls.js";

interface AuctionPlayerProps {
  streamUrl: string;
}

export default function AuctionPlayer(props: AuctionPlayerProps) {
  let videoRef: HTMLVideoElement | undefined;
  let hls: Hls | null = null;

  onMount(() => {
    if (Hls.isSupported() && videoRef) {
      hls = new Hls({
        lowLatencyMode: true,      // Critical for <3s delay
        backBufferLength: 90,
        liveSyncDurationCount: 3,  // Stay close to live edge
      });

      hls.loadSource(props.streamUrl);
      hls.attachMedia(videoRef);

      // Handle auto-play policies
      hls.on(Hls.Events.MANIFEST_PARSED, () => {
        videoRef?.play().catch(e => console.log("Autoplay blocked", e));
      });
    }
  });

  onCleanup(() => {
    if (hls) hls.destroy();
  });

  return (
    <div class="relative w-full h-full bg-black">
      <video
        ref={videoRef}
        controls={false}
        muted={true} // Start muted to allow autoplay
        class="w-full h-full object-cover"
        poster="/images/loading-auction.jpg"
      />
      {/* Live Badge Overlay */}
      <div class="absolute top-4 left-4 bg-red-600 text-white px-2 py-1 text-xs font-bold rounded">
        🔴 LIVE
      </div>
    </div>
  );
}

The Result: Financial Breakdown

By moving the heavy video processing to a dedicated server and keeping the logic in efficient Rust microservices, we achieved massive savings.

Cost Component	Cloud Native (AWS IVS)	Our Hybrid Stack
Video Streaming	₹23,00,000 / mo	₹25,000 / mo (MilesWeb Dedicated)
API Servers	₹15,000 / mo	₹4,000 / mo (DigitalOcean)
Database	₹5,000 / mo	₹2,000 / mo (Supabase)
Latency	~3 Seconds	~3 Seconds
Total Monthly	~₹24 Lakhs	~₹31,000

Cost Savings: 99% Reduction 💰

Why This Works for Startups

Predictable Billing: We pay for the pipe (bandwidth), not the usage (minutes). A 30TB dedicated server costs the same whether you stream 100 hours or 1000 hours.

Rust Efficiency: Handling 10k WebSockets on a $10 server is trivial for Rust (Axum/Tokio), whereas Node.js would require a larger cluster.

Data Sovereignty: Hosting in India (MilesWeb) ensures compliance and faster speeds for local users without the AWS price tag.

Key Takeaways

Separate Video from Logic: Video streaming is bandwidth-heavy but computationally light. Run it on bare metal with unmetered bandwidth
Use Rust for Real-Time: WebSocket servers in Rust can handle 10x more connections per dollar than Node.js
LL-HLS is Good Enough: For auctions, 3-second latency works if your bidding state is synchronized separately via WebSockets
OvenMediaEngine is Production-Ready: We've been running OME for 6 months with zero downtime

Ready to Build Your Own?

Check out these resources:

Questions? Feel free to reach out to me on LinkedIn or check out what we're building at Ambicube Limited.

Written by Mayuresh Smita Suresh, Founder @ Ambicube Limited UK - I have vision for building the future of live commerce. Interested? Join me!

DEV Community