Most DevOps write-ups talk about deploying an application to a cluster. This one is about deploying the same AI platform to hundreds of independent, physically separate sites each with its own hardware, its own network, and its own failure modes and still being able to see all of them from one dashboard.
That's the problem I work on as a DevOps Engineer at Beta Codes.pk, building the infrastructure behind BetaVision, our AI-powered smart surveillance platform for urban and organizational security.
What BetaVision Does
BetaVision is an AI-driven surveillance platform built around continuous, automated monitoring rather than passive CCTV footage. On the product side, it combines:
- 24/7 AI-driven monitoring across critical infrastructure, public spaces, and high-risk areas
- Video analytics for spotting suspicious activity and unusual patterns automatically
- License plate recognition (LPR) for traffic control, parking, and law-enforcement use cases
- Facial recognition for identifying individuals of interest and managing access
- Real-time alerts that notify security personnel the moment something needs attention
- A centralized command-and-control dashboard giving a single, unified view across every connected feed
- A mobile app for remote monitoring
It's aimed at city governments, law enforcement agencies, corporate security teams, event organizers, retailers, and smart-city / property developments — anywhere a lot of camera feeds need to become actionable intelligence instead of just recorded video. The platform is deployed on GPU-enabled client and government infrastructure, running object detection, facial recognition, and ANPR/LPR on live feeds, with results streamed back to a central production dashboard.
The AI/ML side is only half the story. The harder, less glamorous half is: how do you take a platform like this and reliably run it on 400 sites, each behind different networks, different hardware, different connectivity — and still ship updates, catch failures, and keep every stream visible in real time?
That's the part I own.
The Core Challenge: One Codebase, Hundreds of Independent Deployments
Every site runs its own instance of BetaVision — backend, frontend, and supporting services — on Docker containers, on-premise, on the client's own GPU hardware. That means:
- 400 separate deployment targets, not one cluster to scale horizontally
- No guarantee of stable public IPs or open inbound ports on client networks
- Updates that need to reach every site without 400 manual SSH sessions
- A central command-and-control dashboard that needs live visibility into sites it can't directly reach
Here's how each of those gets solved.
Making 400 Private Networks Publicly Reachable
Client sites are, by design, locked-down networks. We don't control their firewalls, and we can't always get inbound access. But the whole point of BetaVision is that streams and results need to be visible centrally, 24/7.
The solution is FRP (Fast Reverse Proxy), running as a client on each site and a server on our side. Every site's containers only need to make an outbound connection — no inbound firewall rules required on the client's end. FRP tunnels the stream back through, and we wrap every tunnel in TLS so nothing is exposed in plaintext across the public internet.
This one decision is what turns "400 isolated on-prem boxes" into "400 endpoints visible from a single pane of glass."
Docker + Networking at Site Level
Each site runs BetaVision's backend and frontend as Docker containers, with internal networks configured so services can talk to each other — the AI inference services, the API layer, the database layer, and the streaming layer all need to communicate reliably on-box before anything goes external through FRP.
Multiply the usual "make sure your containers can talk to each other" problem by 400, across hardware you don't physically control, and debugging becomes as much about reproducibility as it is about the fix itself. A problem on site #217 needs to be diagnosable and fixable using the same playbook as site #12.
Automating 400 Pipelines Without Losing Control
Manually deploying to 400 sites isn't a scaling problem, it's a guaranteed-to-fail problem. So the deployment pipeline is built on GitHub Actions, with dedicated pipelines per site for backend and frontend releases.
A few things matter more here than in a typical single-cluster CI/CD setup:
- Release history has to be real, not aspirational. With this many deployment targets, "what version is site X running right now" needs to be answerable instantly, not reconstructed from memory.
- One-click releases replace multi-step manual processes. What used to be a manual, error-prone rollout became a single pipeline trigger — this matters even more at 400-site scale than it did rolling this pattern out internally at 10-microservice scale.
- Failures need to be isolated. A bad deploy to one site should never cascade or block releases to the other 399.
Monitoring: Seeing Problems Before Clients Report Them
None of this matters if you find out about outages from a client phone call. Monitoring across this many independent deployments has to answer two different questions:
- Is each individual site healthy?
- Is the aggregate platform healthy?
We run a Prometheus, Grafana, and Loki stack for visibility and alerting, so incidents surface through dashboards and automated alerts instead of user reports. At single-cluster scale this is standard practice; at 400-site scale, it's the difference between catching a dead stream in minutes versus finding out days later.
Data: MinIO S3 and Databases, Per Site
Each deployment needs its own storage and database layer — provisioned, backed up, and secured independently. We use MinIO S3 buckets for object storage and manage the database infrastructure per deployment, with backup and recovery built in from day one rather than bolted on after a data loss incident.
Security runs through all of this, not around it — from access control down to how data moves off each site, aligned with our ISO 9001 and ISO 27001 certified information security standards.
From POC to Production, Repeatedly
Before a site goes live, it usually starts as a proof of concept — deployed on a client's own GPU server, demoed live, sometimes directly to CIOs, under real production conditions. That means the deployment tooling has to work identically whether it's POC #1 or production site #380. There's no separate "demo mode" — if the automation only works for the polished cases, it doesn't scale to 400 real ones.
What This Actually Teaches You About DevOps
Working on BetaVision reframed how I think about scale. It's not just "more replicas of the same thing." It's:
- Reliability without central control — most of your infrastructure lives on hardware and networks you don't own.
- Observability as a first-class deliverable, not an afterthought — you cannot manually check 400 sites.
- Automation as risk reduction, not just convenience — every manual step is a failure point multiplied by 400.
If you're building infrastructure for a platform that needs to run in many independent, disconnected environments rather than one big cluster, I'd genuinely love to compare notes — feel free to reach out.
I'm a DevOps Engineer at Beta Codes.pk, working on Kubernetes, GitOps, and infrastructure for AI platforms deployed across hundreds of sites. Find me on LinkedIn or check out my portfolio.
Top comments (0)