Building a multi-tenant PaaS application: Part 1 (Architecture & Initial Design)

TLDR: I’m building Stratus, a multi-tenant PaaS, to understand how platforms like Heroku handle orchestration and scaling. Part 1 covers the architecture, node scheduling, agent design, and deployment flow. Follow along for lessons learned from building distributed infrastructure from scratch.

There's a difference between using a tool and understanding how it works.

I've been deploying applications to Heroku, Railway, and similar platforms for years. They abstract away the infrastructure complexity - which is great for shipping products, but terrible for learning how distributed systems actually work.

I wanted to understand:

• How does a platform decide which server should run your code?
• How do multiple servers coordinate without stepping on each other?
• What happens when things fail? How does the system recover?

You can read about these problems in books and blog posts. But for me, the only way to truly understand something is to build it.

That's why I'm building Stratus - a multi-tenant PaaS infrastructure that handles container orchestration across distributed nodes. It's not fully production-ready (yet), and that is not the real goal. The goal is to understand the foundational patterns that platforms like Heroku rely on.

Turns out, building even a simplified version teaches you more than a dozen textbooks.

This is Part 1 - covering the foundation of Stratus. I'll be writing more parts as the project progress.

P.S. This approach is exactly the kind of infrastructure design I help startups implement to scale reliably without downtime.

What is Stratus?

Its a multi-tenant Platform-as-a-Service infrastructure I built to understand how platforms like Heroku work under the hood.

It handles code uploads and deployments, intelligently selects compute nodes based on resource availability, agent-based orchestration, fault-tolerant task distribution with health checks, etc.

At its current state, the project is optimized for batch jobs and worker tasks. HTTP ingress and external traffic routing is planned next.

High-Level Architecture

I tried to go for a K8s-like architecture for this project whilst keeping it as simple as possible.
There are four main components:

• API Server (user-facing, accepts deployments)
• Management Plane (The puppet master i.e. orchestration brain, schedules tasks and keeps track of compute nodes along with health checks)
• Deployment/Compute Nodes (Virtualized or bare metal servers that run containers, each are a part of the Stratus cluster, managed by the Management Plane)
• Node Agent (An agent software that runs on each Deployment Node in order to respond to health check requests, deploy and manage containers)

In a nutshell, the architecture follows a control plane/data plane pattern. The Management Plane makes decisions about where to run workloads. Deployment Nodes execute those decisions. Agents on each node handle communication and container lifecycle management.

The Deployment Flow (Step-by-Step)

The entire deployment flow starts with the user uploading the code as an entire folder, which gets stored as a ZIP file on MinIO/S3.

Step 1: User uploads code

• API receives code upload
• Creates deployment record
• Sends gRPC deployment task to Management Plane

Step 2: Management Plane schedules the deployment

• Queries available compute nodes
• Evaluates resource availability (CPU, memory, existing containers)
• Selects the least-burdened node
• Sends task to that node's Agent

Here's a snippet of how the Management Plane selects the optimal node

def get_least_burdened_node():
    """
    Returns the node_id of the least burdened node based on CPU and memory.
    """
    if not _nodes:
        return None

    sorted_nodes = sorted(
        _nodes.items(), key=lambda item: (item[1]["cpu"], item[1]["mem"])
    )
    return sorted_nodes[0][0]

Step 3: Agent executes the deployment

• Receives deployment task
• Pulls user code from S3 and mounts it onto Docker container
• Runs container with stratus_init.sh as entrypoint
• Reports success/failure back to Management Plane

Step 4: Deployment is live

• Container is running
• Management Plane tracks it
• User sees deployment status

Agent-Based Architecture

In order to register a worker machine as a part of the Stratus cluster and to run containers on it, I created Agent applications to run on each Deployment Node.

Initially, these are the problems I faced:

• Management Plane needs to communicate with many compute nodes
• Direct SSH or API calls don't scale well
• Need resilient, asynchronous communication

And after implementing Agents:

• Each compute node runs a persistent Agent
• Agent registers with Management Plane on startup
• Receives tasks, executes them, reports status
• Like Kubernetes kubelet or Nomad agent
• Responds to health check requests

I chose Golang for the Agent because it has low overhead and it's very good for this purpose due to its concurrency model.

A simplified example of how the Agent runs an application container

func RunDeploymentContainer(ctx context.Context, deploymentId string, deploymentFilesPath string) error {
    client, err := getAPIClient()
    if err != nil {
        return err
    }

    // Generate a logical container ID for internal tracking
    containerId := util.GenerateCryptoID()
    config := &container.Config{...}

    hostConfig := &container.HostConfig{...}

    // Create container with deterministic name
    resp, err := client.ContainerCreate(ctx, config, hostConfig, nil, nil, fmt.Sprintf("deploy-%s-%s", deploymentId, containerId))
    if err != nil {
        return err
    }

    // Start container
    if err := client.ContainerStart(ctx, resp.ID, container.StartOptions{}); err != nil {
        return err
    }
    return nil
}

Fault Tolerance & Health Checks

At the moment, I have not implemented a robust method to handle failures in the system.

Currently, the Management Plane sends gRPC requests to every Deployment Node and the Agent must respond with the system resources available (CPU and RAM).

• The Management Plane keeps an internal state of each node and later on, decides to offload deployment tasks based on it.
• In case a Node dies or becomes irresponsive, the failed health check will cause the Management Plane to not send any tasks to that Node.
• Rather, it will keep sending health check request until the Node is responsive again. Once that happens, it will start receiving tasks as usual.

A simple yet effective approach for now.

Here's what I learned

When I first thought of building Stratus, I was somewhat clueless on where to start, because there's so many moving pieces and every one of them is required for the entire thing to function properly.

I'll have to admit, distributed state management isn't easy. Tracking which nodes have what containers requires careful design.

There's so many edge cases which need to be taken into consideration when running something of this scale. After all, anything can go wrong (and it will).
This has helped me to understand why Kubernetes is so complex - it handles every edge case I'm discovering.

Things like container rescheduling, dynamic routing (since the internal network is completely isolated) will require lots of planning and careful considerations.

And gVisor had made it a complete pain for me to access the containers through Docker's internal network, without tweaking the network isolation levels (or completely disabling it).

If there's something I'd do differently for this, I'd add observability from day 1. Developing this would have been so much easier if I had logs and metrics from the get go. That was probably one of the mistakes I made with this.

What's Next

Part 2 will cover the routing layer - how to expose containerized apps to the internet when they're running on internal networks across multiple nodes. P.S. it involves Consul, OpenResty, and sidecar proxying.

I'll also be working on implementing Horizontal scaling for this so it can run multiple containers application deployment. Stay tuned.

---

If you're interested in the code: Stratus GitHub

Have ideas, suggestions, or questions? Reply here or reach me at [e-mail] - I’d love to hear from you!

Thank you for reading and have an amazing day!