A cloud native platform showcasing how AKS, Crossplane, Terraform, vClusters and ArgoCD enable scalable, secure, and efficient multi-tenant management with GitOps principles.
Introduction
This article presents a proof of concept (POC) for Tenark, a multi-tenant Software as a Service (SaaS) system designed as an online bookstore. The POC implements a SaaS control plane using the GitOps Bridge pattern, leveraging technologies like Azure Kubernetes Service (AKS), Crossplane, Terraform, Virtual Clusters (vCluster), and ArgoCD. This architecture provides a scalable, secure, and cost-effective solution for managing multi-tenant environments, balancing isolation and resource efficiency.
The goal is to demonstrate how modern cloud-native tools can simplify tenant provisioning, management, and application deployment while maintaining flexibility for future scalability, such as supporting premium offerings.
Use Case
In many enterprises, multi-tenant SaaS systems are critical for delivering scalable, service-centric solutions. Tenark addresses the need for seamless tenant creation, isolation, and management in a SaaS environment. As an online bookstore, Tenark allows customers to browse, purchase, and manage books, with features like inventory tracking, order processing, and personalized recommendations. The system supports enterprise needs for multi-tenant management with features like:
- Security : Ensuring tenant data and operations are isolated and protected.
- Tenant Provisioning : Streamlining the creation and configuration of new tenants.
- Tenant Data Storage : Managing tenant-specific databases and caches.
- Metrics, Logging, and Monitoring : Providing insights into system performance and usage.
A planned premium tier for high-value customers will offer exclusive books, fully isolated Kubernetes clusters for better performance and security, and advanced analytics. Though not implemented in this POC, this feature highlights the architecture’s flexibility.
Architecture Overview
The following diagram illustrates Tenark’s high-level architecture:
Key Components
- GitHub and GitHub Actions : Hosts source code and automates deployment workflows for AKS, applications, and infrastructure.
- Terraform Cloud : Provisions the main AKS cluster and bootstraps system components.
- AKS : Hosts two node pools:
- System Pool : Runs system components like ArgoCD and Crossplane.
- User Pool : Hosts the SaaS Control Plane and Application Plane via vClusters.
- Tenant Data : Managed in Azure Resource Groups, containing tenant-specific infrastructure like Redis caches and SQL Server databases.
- Azure Platform Services : Includes Entra ID for identity management, Azure Monitor for observability, KeyVault for secrets, and DNS for routing.
These components are detailed in the following sections.
What is SaaS?
Software as a Service (SaaS) is a delivery model where software is hosted centrally and provided to customers over the internet on a subscription basis. Unlike traditional software, SaaS emphasizes a shared experience, reducing the need for customer-specific customizations. This approach lowers maintenance costs and enables rapid scaling.
For Tenark, SaaS means delivering a seamless bookstore experience to multiple tenants (e.g., individual users, libraries, or bookstores) while maintaining a single codebase and infrastructure core.
Multi-Tenancy in Tenark
Multi-tenancy allows a single application instance to serve multiple customers (tenants) while ensuring data and configuration isolation. Tenark employs a hybrid multi-tenancy model, balancing shared and isolated components based on business and performance needs.
Multi-Tenancy Approaches
Common multi-tenancy strategies include:
- Single Application, Single Database : All tenants share one application and database.
- Single Application, Multiple Databases : One application, but each tenant has a dedicated database.
- Multiple Applications, Single Database : Each tenant has a dedicated application instance sharing a database.
Tenark combines these approaches for different components:
- SaaS Control Plane:
✅ Deployment : Shared across all tenants in a single vCluster.
✅ Database : Shared database in a common Azure Resource Group.
- SaaS Web Portal :
✅ Deployment : Shared across all tenants in a single vCluster.
✅ Database : Shared database in a common Azure Resource Group.
- SaaS Identity Server :
✅ Deployment : Shared across all tenants in a single vCluster.
✅ Database : Shared database in a common Azure Resource Group.
- Tenant Microservices (Ordering, Inventory, Recommendation):
✅ Deployment : Each microservice per tenant runs in a dedicated vCluster.
✅ Database : Separate databases per tenant and service in distinct Azure Resource Groups, with a shared Redis cache per tenant.
Benefits of Tenark’s Approach
- Enhanced Security: Dedicated vClusters and isolated databases for tenant-specific microservices ensure robust data separation and protection against unauthorized access.
- Data and Performance Isolation : Tenant-specific microservices and databases prevent “noisy neighbor” issues, especially for resource-intensive services like recommendations.
- Cost Efficiency : Shared components reduce maintenance overhead.
- Scalability : The architecture supports premium tiers with fully isolated clusters for high-value tenants.
- Flexibility : A common codebase simplifies updates and feature rollouts.
For example, the recommendation microservice, which may experience usage spikes, benefits from isolated deployments to ensure performance stability across tenants. By running in dedicated vClusters, each tenant’s recommendation engine operates independently, preventing one tenant’s high demand (e.g., during a promotional campaign) from impacting others. This isolation also allows for tenant-specific customization, such as tailored machine learning models, without affecting the shared infrastructure, and ensures dedicated resource allocation for consistent response times.
GitOps Bridge Pattern
The GitOps Bridge pattern, a community-driven best practice, streamlines Kubernetes cluster provisioning and management using GitOps principles. It integrates tools like Terraform, ArgoCD, and Crossplane to automate infrastructure and application deployments.
This was originally created for AWS, but as a Cloud Native pattern can be extended to any HyperScaler.
In Tenark, the GitOps Bridge pattern provisions the AKS control plane and orchestrates tenant-specific infrastructure and applications.
Control Plane
Tenark leverages the GitOps Bridge pattern to establish a primary AKS control plane, which orchestrates the deployment of all infrastructure and applications required for the system. This main control plane, hosted on an Azure Kubernetes Service (AKS) cluster, serves as the central management layer, coordinating resources and ensuring seamless operation of the SaaS platform. It is based on the AKS Platform Engineering project.
Microsoft and AWS provide specific guidelines for building SaaS control planes. In Tenark, it’s critical to distinguish that the AKS control plane acts as the global orchestrator, managing all deployments. Within these deployments, we define two key SaaS components: the SaaS Control Plane, which handles tenant management, and the SaaS Application Plane, which delivers the user-facing experience. These components, detailed in the following sections, operate within the broader AKS control plane, leveraging tools like ArgoCD and Crossplane to automate and streamline their deployment.
SaaS Control Plane
The SaaS Control Plane is foundational to any multi-tenant SaaS model. It provides a centralized set of services that enable operators to manage and operate all tenants through a single, unified interface, regardless of the underlying deployment or isolation model. This control plane handles critical tasks like tenant provisioning or user authentication ensuring a consistent and scalable experience for managing the SaaS environment.
This control plane can be visualized as shown in the following diagram:
Implementation
Tenark’s SaaS Control Plane is implemented via the Tenark.Web.ControlPlane administration application, built using the ABP.io framework.
ABP.io is an open-source framework for building modular, multi-tenant applications with features like:
- Built-in multi-tenancy support with UI for tenant management.
- Role-based access control (RBAC) and user management.
- Extensible product and service management.
The Tenark.AuthServer , based on IdentityServer , handles the identity, admin user management, authentication and authorization, ensuring secure tenant and user access.
All of these components are deployed in a shared vCluster named tenant-common. Below we can see the isolation for all the pods on just that vCluster.
Future Enhancements
While the POC focuses on core functionality, future iterations could include:
- Onboarding : Automating tenant creation with predefined YAML templates and connection strings.
- Billing Integration : Supporting subscription models for premium tiers.
- Metrics and Monitoring : Integrating Azure Monitor for tenant-specific analytics.
Onboarding, while ideal, requires complex dynamic automation (e.g., generating tenant-specific YAMLs, configurations and infrastructure). For this POC, operators manually configure tenant properties, which is sufficient for demonstration purposes.
SaaS Application Plane
The SaaS Application Plane is central to delivering the user-facing functionality in a multi-tenant SaaS model. It provides the services and interfaces that enable tenants to interact with the application, ensuring a seamless and customized experience while leveraging the underlying multi-tenant architecture.
This application plane can be visualized as shown in the following diagram:
Implementation
The Tenark.Web.Portal implements the SaaS application which consumes a set of tenant-specific application services, implemented as microservices running as Kubernetes pods. Each microservice operates in a dedicated vCluster, ensuring complete isolation between tenants, as detailed in later sections. These microservices include:
- Inventory : Manages book stock and availability, ensuring accurate tracking of inventory for each tenant.
- Ordering : Processes purchases and order tracking, handling tenant-specific transactions.
- Recommendation : Generates personalized book suggestions using tenant-specific data, leveraging isolated compute resources for performance.
Each microservice is supported by its own SQL Server database and a shared, tenant-specific Redis cache for optimized performance.
And as before we can all the pods deployed in an isolated vCluster with only the needed services for this application plane.
Virtual Clusters
Virtual Clusters (vClusters) are lightweight, isolated Kubernetes clusters running within a single physical AKS cluster. Each vCluster has its own API server, providing stronger isolation than namespaces without the overhead of separate clusters.
Benefits
- Robust Isolation :
✅Granular permissions per tenant.
✅Isolated control planes and networking.
✅Customizable security policies.
- Cost Efficiency :
✅Lightweight infrastructure reduces resource consumption.
✅Simplified management compared to dedicated clusters.
- Scalability :
✅Reduced API server load.
✅Conflict-free Custom Resource Definition (CRD) management.
- Flexibility :
✅Supports diverse Kubernetes environments.
✅Runs on any Kubernetes-compatible platform.
Tenark uses vClusters for:
- Tenant-Common : Hosts shared components like the SaaS Control Plane and Web Portal.
- Tenant-Specific : Isolates microservices (Ordering, Inventory, Recommendation) per tenant.
ArgoCD
ArgoCD is the GitOps engine for deploying applications and infrastructure. It uses ApplicationSets and Kustomize manifests to manage tenant deployments across environments (Dev, QA, Prod). For the POC, only the Dev environment is implemented, but the system is designed for multi-environment support.
Example: ApplicationSet for Tenant Deployment
The following ApplicationSet deploys the tenant-common vCluster and tenant-specific vClusters (tenant1, tenant2, tenant3) using a matrix generator:
This setup allows operators to add or remove tenants by updating the elements list, ensuring flexibility in tenant management.
Kustomize Overlay
The Kustomize overlay defines environment-specific configurations, such as the Dev environment:
Tenant Folder Structure
Each tenant (e.g., tenant2) and the tenant-common follows a Kustomize-based folder structure for environment-specific deployments:
Terraform
While most infrastructure in Tenark is provisioned through Crossplane, the initial setup of the AKS Control Plane, which hosts Crossplane and other system components, is created using Terraform via the GitOps Bridge pattern. This pattern enables the provisioning of the AKS cluster with all necessary add-ons, such as ArgoCD and Crossplane, to support the SaaS platform.
Tenark leverages Terraform Cloud , a managed service platform designed to streamline Terraform usage in collaborative, team-oriented environments. Terraform Cloud enhances team workflows with features like state management and automated runs, though the system could also operate with the standard version of Terraform if needed.
Policies
Terraform Cloud includes Sentinel , a policy-as-code tool that enforces rules to control what users of HashiCorp products can do. Sentinel proactively prevents unauthorized changes by validating infrastructure configurations before deployment. In Tenark, Sentinel policies enforce requirements such as mandatory tags on all Azure resources, ensuring compliance and consistency across the infrastructure.
Crossplane
To create all the infrastructure resources needed for Tenark (except for the AKS Control Plane), Crossplane is used, which is a Kubernetes-native control plane framework for platform engineering. Crossplane enables the creation of custom control planes to manage cloud-native software, allowing the design of APIs and abstractions that users interact with to provision and manage infrastructure.
This notion of a control plane differs from the main AKS control plane and the SaaS Control Plane but similarly serves as a component that manages and orchestrates the delivery of the desired infrastructure in Tenark’s scenario. This aligns with the Kubernetes concept of a control plane, which oversees resource management.
A key advantage of Crossplane is its use of a reconciliation loop, leveraging Kubernetes to continuously ensure infrastructure matches the desired state. Unlike Terraform’s push-based model, which requires explicit terraform plan and apply commands and state management, Crossplane automates this process, eliminating the need for such manual operations.
Crossplane uses YAML as its declaration language and is integrated with ArgoCD to deploy these configurations seamlessly.
Managed Resources
A managed resource in Crossplane represents an external service in a provider, in this case, the Upbound Azure Provider. Upbound, the primary maintainer of Crossplane, offers providers for various platforms, including AWS and Google Cloud. For Tenark, managed resources define Azure services like Redis caches and SQL databases.
Here is an example of a Redis cache configuration:
Compositions
Crossplane’s Compositions are a powerful feature, allowing multiple Kubernetes resources to be represented as a single Kubernetes object. Composite resources are created when users interact with a custom API defined in a CompositeResourceDefinition.
Tenark uses compositions to create the common tenant infrastructure stack, which includes a SQL Server database and a Redis cache. An example of such a composition is shown below:
Lessons Learned
- GitOps Bridge Pattern : Simplifies AKS provisioning and integrates seamlessly with ArgoCD and Crossplane.
- vClusters : Provide robust isolation for multi-tenancy, balancing cost and performance compared to namespaces or dedicated clusters.
- ArgoCD and vClusters : Enable flexible tenant management with minimal configuration changes.
- Crossplane Reconciliation : Offers powerful automation but requires time to stabilize; occasional sync delays were observed.
- Crossplane Compositions : Complex to design but really helpful for abstracting infrastructure stacks. Tools like ClaudeCode assisted in their creation.
- Crossplane Learning Curve : Steeper than Terraform due to its Kubernetes-native approach, but its reconciliation model reduces operational overhead.
Git Repositories
https://github.com/zodraz/tenark-identity-server
https://github.com/zodraz/tenark-services
Top comments (0)