In the previous article, we saw how to simulate the network and nodes of a cloud with a relatively small amount of Linux networking.
But as we already said, a cloud is first and foremost software.
We are obviously not going to manually create and configure the network every time a new customer signs up.
To handle that, we need to build what is called a Control Plane.
The Control Plane
A cloud is usually divided into two categories: those with loaded guns, and those who dig...
No, wait. Wrong movie.
A cloud is usually divided into two main parts: the Control Plane and the Data Plane.
The Control Plane
The Control Plane is the entry point for requests made to the cloud, both user requests and requests coming from the system itself.
It is responsible for organizing and orchestrating actions across the cloud.
The Data Plane
The other part is where the actual resources and data live.
In our simplified model, the Data Plane is where we find our nodes, VMs, disks, networks, and workloads.
Right now, our cloud is still just a skeleton.
We have no actual workloads, and we do not even have real machines yet. Our nodes are currently simulated using isolated Linux environments, mainly network namespaces.
But that is already enough to start building a Control Plane capable of creating tenants, managing nodes, allocating resources, and deleting them when they are no longer needed.
The PandaCloud Project
We are going to create a C# project for our Control Plane.
It could obviously be written in almost any language, so if you want to implement yours in Java or Go, go ahead.
If you want to do it in JavaScript, leave this blog and never come back.
The Model
We will start by defining a few model classes representing:
- a tenant
- a node
- the overall state of the cloud
The classes are fairly self-explanatory, so there is no need to spend too much time on them.
public class Tenant
{
public string Name { get; set; } = string.Empty;
public string BridgeName { get; set; } = string.Empty;
public string Subnet { get; set; } = string.Empty;
public string Gateway { get; set; } = string.Empty;
public int NextHostAddress { get; set; } = 10;
}
public enum NodeState
{
Available,
Allocated
}
public class Node
{
public string Name { get; set; } = string.Empty;
public int CpuCores { get; set; }
public NodeState State { get; set; } = NodeState.Available;
public string? TenantName { get; set; } = null;
public string? IpAddress { get; set; } = null;
}
public class CloudState
{
public List<Tenant> Tenants { get; init; } = [];
public List<Node> Nodes { get; init; } = [];
}
The next step is to actually manipulate the infrastructure.
We want to create nodes, create tenant networks, allocate nodes to tenants, release them, and eventually destroy them.
For now, everything runs on a single Linux system.
But later, the same Control Plane could manage resources through another infrastructure provider.
It could talk to Azure, libvirt, another cloud provider, or something entirely different.
At that point, instead of running Bash commands, the provider could call APIs, deploy ARM templates, use Terraform, or communicate with another infrastructure system.
The Infrastructure Provider
To keep things flexible, we will define an interface describing the infrastructure operations our Control Plane needs.
public interface IInfrastructureProvider
{
Task CreateTenantNetworkAsync(Tenant tenant);
Task DeleteTenantNetworkAsync(Tenant tenant);
Task CreateNodeAsync(Node node);
Task AttachNodeToTenantAsync(Node node, Tenant tenant, string ipAddress);
Task ReleaseNodeAsync(Node node);
Task DeleteNodeAsync(Node node);
}
The interface is fairly straightforward, but there is one important detail worth mentioning.
All of these methods return Task.
You could simply say this is good practice because we are executing external operations and do not want to block the Control Plane while waiting for them.
And that is true.
But cloud operations can eventually become much more complex than a simple network or shell call.
Creating, configuring, repairing or moving resources can sometimes take minutes, hours, or even longer.
So from the beginning, we should think about these operations asynchronously.
"Wait. I've created plenty of Azure resources before and it never took hours."
Well, congratulations. You have been lucky.
But the Control Plane does not only process direct customer requests.
It also receives requests from the cloud itself.
Those operations can involve several services, retries, dependencies and, in some cases, even human intervention.
For now, we are simply using asynchronous C# methods because our operations are still short-lived.
Later, these operations should be thought of as actual persistent workflows.
A Task is not a replacement for a durable workflow engine, but it already prevents us from designing the Control Plane around blocking operations.
For now, our infrastructure operations will execute shell commands.
So let's create a small helper class.
public sealed class ShellRunner
{
public async Task RunAsync(string command)
{
Console.WriteLine($"> {command}");
var startInfo = new ProcessStartInfo
{
FileName = "wsl.exe",
UseShellExecute = false,
RedirectStandardOutput = true,
RedirectStandardError = true
};
startInfo.ArgumentList.Add("-u");
startInfo.ArgumentList.Add("root");
startInfo.ArgumentList.Add("--");
startInfo.ArgumentList.Add("bash");
startInfo.ArgumentList.Add("-lc");
startInfo.ArgumentList.Add(command);
using var process = Process.Start(startInfo)
?? throw new InvalidOperationException("Unable to start shell process.");
var stdoutTask = process.StandardOutput.ReadToEndAsync();
var stderrTask = process.StandardError.ReadToEndAsync();
await process.WaitForExitAsync();
Console.Write(await stdoutTask);
if (process.ExitCode != 0)
throw new InvalidOperationException(await stderrTask);
}
}
This class is intentionally simplified for the article.
You can find the complete version in the PandaCloud repository:
https://github.com/MarieLePanda/PandaCloud
Now we can start building our infrastructure provider.
The Infrastructure
If you read the previous article carefully, we created Alice's network using something like this:
sudo ip link add pc-alice type bridge
sudo ip link set pc-alice up
We are going to reuse the same idea inside our Control Plane, but with one important difference.
What happens if Alice's network already exists?
Maybe the operation is retried.
Maybe the Control Plane restarted.
Maybe something failed halfway through the previous operation.
Whatever the reason, running infrastructure operations more than once must not randomly break the system.
A command executed a second time could fail or, depending on the operation, create duplicate resources.
So our infrastructure operations should be idempotent.
Running the same operation several times should lead to the same final state.
public async Task CreateTenantNetworkAsync(Tenant tenant)
{
await _shellRunner.RunAsync(
$"sudo ip link show {tenant.BridgeName} >/dev/null 2>&1 || " +
$"sudo ip link add {tenant.BridgeName} type bridge");
await _shellRunner.RunAsync(
$"sudo ip link set {tenant.BridgeName} up");
}
The same applies to nodes.
Previously, we created them with:
sudo ip netns add node10
Now we simply make sure the namespace does not already exist before creating it.
public async Task CreateNodeAsync(Node node)
{
await _shellRunner.RunAsync(
$"if ! sudo ip netns list | grep -qw {node.Name}; then " +
$"sudo ip netns add {node.Name}; " +
$"fi");
}
The final operation we need for now is attaching a node to a tenant.
This part is slightly more complicated.
We need to connect the node to the tenant bridge using a veth pair, our virtual network cable.
And once again, we want the operation to be idempotent.
private static string ShortInterfaceName(string name)
{
// Linux interface names are limited to 15 characters.
// Enhance the method in the future to generate unique names.
return name.Length <= 15 ? name : name[..15];
}
private static string GetHostInterfaceName(string nodeName)
{
var hostVeth = ShortInterfaceName($"v-{nodeName}-h");
return hostVeth;
}
private static string GetNodeInterfaceName(string nodeName)
{
var nodeVeth = ShortInterfaceName($"v-{nodeName}-n");
return nodeVeth;
}
public async Task AttachNodeToTenantAsync(
Node node,
Tenant tenant,
string ipAddress)
{
var hostVeth = GetHostInterfaceName(node.Name);
var nodeVeth = GetNodeInterfaceName(node.Name);
await _shellRunner.RunAsync(
$"if ! sudo ip link show {hostVeth} >/dev/null 2>&1; then " +
$"sudo ip link add {hostVeth} type veth peer name {nodeVeth} && " +
$"sudo ip link set {nodeVeth} netns {node.Name}; " +
$"fi");
await _shellRunner.RunAsync(
$"sudo ip link set {hostVeth} master {tenant.BridgeName}");
await _shellRunner.RunAsync(
$"sudo ip link set {hostVeth} up");
await _shellRunner.RunAsync(
$"sudo ip netns exec {node.Name} ip link set lo up");
await _shellRunner.RunAsync(
$"sudo ip netns exec {node.Name} ip link set {nodeVeth} up");
await _shellRunner.RunAsync(
$"sudo ip netns exec {node.Name} ip addr flush dev {nodeVeth}");
await _shellRunner.RunAsync(
$"sudo ip netns exec {node.Name} ip addr add {ipAddress}/24 dev {nodeVeth}");
}
Managing the Lifecycle
We can now create resources and attach nodes to tenant networks.
But creating things is only half the problem.
A node can be created, allocated to a tenant, released later, reassigned somewhere else, and eventually destroyed.
So we also need to manage the rest of its lifecycle.
Deleting a node is straightforward, but we still want the operation to succeed if the namespace has already disappeared.
public async Task DeleteNodeAsync(Node node)
{
await _shellRunner.RunAsync(
$"if sudo ip netns list | grep -qw {node.Name}; then " +
$"sudo ip netns delete {node.Name}; " +
$"fi");
}
The same principle applies when deleting a tenant network.
public async Task DeleteTenantNetworkAsync(Tenant tenant)
{
await _shellRunner.RunAsync(
$"if ip link show dev {tenant.BridgeName} >/dev/null 2>&1; then " +
$"sudo ip link delete dev {tenant.BridgeName} type bridge; " +
$"fi");
}
And finally, releasing a node means removing its host-side veth interface.
public async Task ReleaseNodeAsync(Node node)
{
var hostInterface = GetHostInterfaceName(node.Name);
await _shellRunner.RunAsync(
$"if sudo ip link show dev {hostInterface} >/dev/null 2>&1; then " +
$"sudo ip link delete dev {hostInterface}; " +
$"fi");
}
So, are we done with the Control Plane?
No.
Actually, we haven't even started building it yet.
So far, we have created infrastructure operations that are reasonably safe to execute.
But there are still a lot of missing rules.
What happens to allocated resources when a tenant is deleted?
Can I allocate a node that is already assigned to someone else?
How do I decide which IP address should be assigned?
How much capacity is available?
That logic belongs in the service layer.
That is where our actual Control Plane starts.
The Real Control Plane
Our infrastructure functions will now be called by a service layer responsible for validating the requested operation before modifying the infrastructure.
We will start with tenant and node creation.
public async Task CreateTenantAsync(string name)
{
if (_cloudState.Tenants.Any(t =>
t.Name.Equals(name, StringComparison.OrdinalIgnoreCase)))
{
throw new Exception($"Tenant with name {name} already exists.");
}
string networkId = FindNextNetworkId(_cloudState).ToString();
Tenant newTenant = new Tenant
{
Name = name,
BridgeName = $"br-{name.ToLower()}",
Subnet = $"10.{networkId}.0.0/24",
Gateway = $"10.{networkId}.0.1",
};
await _infrastructureProvider.CreateTenantNetworkAsync(newTenant);
_cloudState.Tenants.Add(newTenant);
Console.WriteLine(
$"Tenant {name} created with subnet {newTenant.Subnet} " +
$"and gateway {newTenant.Gateway}.");
}
For now, .1 is simply reserved as the future gateway address in our model.
Routing will come later.
Node creation follows exactly the same idea.
public async Task CreateNodeAsync(string nodeName, int cpuCores)
{
if (_cloudState.Nodes.Any(n =>
n.Name.Equals(nodeName, StringComparison.OrdinalIgnoreCase)))
{
throw new Exception($"Node with name {nodeName} already exists.");
}
Node newNode = new Node
{
Name = nodeName,
CpuCores = cpuCores,
};
await _infrastructureProvider.CreateNodeAsync(newNode);
_cloudState.Nodes.Add(newNode);
Console.WriteLine($"Node {nodeName} created.");
}
At this stage, CpuCores is only capacity metadata inside the Control Plane.
Our network namespace is not actually restricted to 12 or 6 CPUs.
Later, a provider based on real virtual machines, containers or cgroups could enforce that capacity.
Now comes the slightly more interesting part: allocating a node to a tenant.
In addition to validating the request, we also need to allocate an available IP address.
public async Task AllocateNodeAsync(string nodeName, string tenantName)
{
Node? node = _cloudState.Nodes.FirstOrDefault(n =>
n.Name.Equals(nodeName, StringComparison.OrdinalIgnoreCase));
if (node == null)
throw new Exception($"Node {nodeName} does not exist.");
Tenant? tenant = _cloudState.Tenants.FirstOrDefault(t =>
t.Name.Equals(tenantName, StringComparison.OrdinalIgnoreCase));
if (tenant == null)
throw new Exception($"Tenant {tenantName} does not exist.");
if (node.State != NodeState.Available)
throw new Exception($"Node {nodeName} is not available for allocation.");
string networkId = tenant.Subnet.Split('.')[1];
string ipAddress = $"10.{networkId}.0.{tenant.NextHostAddress}";
tenant.NextHostAddress++;
await _infrastructureProvider.AttachNodeToTenantAsync(
node,
tenant,
ipAddress);
node.TenantName = tenantName;
node.IpAddress = ipAddress;
node.State = NodeState.Allocated;
Console.WriteLine(
$"Node {nodeName} allocated to tenant {tenantName} " +
$"with IP address {ipAddress}.");
}
I will skip the other service methods for releasing nodes, deleting them, and so on.
They follow the same pattern as what we have already seen.
As always, the complete code is available in the PandaCloud repository:
https://github.com/MarieLePanda/PandaCloud
Let's Test Our Control Plane
It is finally time to run it.
We still need one small piece: an entry point to call our Control Plane methods.
We could expose an HTTP API.
Eventually, we probably will.
But for now, let's keep things simple and use command-line arguments.
CloudState cloudState = new CloudState();
IInfrastructureProvider provider = new LinuxProvider();
ControlPlane controlPlane = new ControlPlane(cloudState, provider);
try
{
switch (args)
{
case ["tenant", "create", var tenantName]:
await controlPlane.CreateTenantAsync(tenantName);
break;
case ["tenant", "list"]:
await controlPlane.ListTenantsAsync();
break;
case ["node", "create", var nodeName, var cpuCores]:
await controlPlane.CreateNodeAsync(nodeName, int.Parse(cpuCores));
break;
case ["node", "allocate", var nodeName, var tenantName]:
await controlPlane.AllocateNodeAsync(nodeName, tenantName);
break;
case ["node", "list"]:
await controlPlane.ListNodesAsync();
break;
case ["node", "release", var nodeName]:
await controlPlane.ReleaseNodeAsync(nodeName);
break;
case ["node", "delete", var nodeName]:
await controlPlane.DeleteNodeAsync(nodeName);
break;
case ["capacity", "request", var tenantName, var cpuCores]:
await controlPlane.RequestCapacity(tenantName, int.Parse(cpuCores));
break;
case ["reconcile"]:
await controlPlane.ReconcileAsync();
break;
default:
Console.Error.WriteLine("Unknown command.\n");
PrintHelp();
Environment.ExitCode = 1;
break;
}
}
catch (Exception ex)
{
Console.ForegroundColor = ConsoleColor.Red;
Console.Error.WriteLine($"Error: {ex.Message}");
Console.ResetColor();
Environment.ExitCode = 1;
}
And now we can run our first command:
dotnet run -- tenant list
No tenant.
Wait.
No tenant?
Where are Alice and Bob?
Until now, our resources were created directly through Bash.
Linux knows they exist.
Our Control Plane does not.
The CloudState represents what our Control Plane knows about the cloud, and right now that state only exists in memory.
That is not particularly useful for a console application that starts, executes one command, and immediately exits.
So we need persistence.
Remembering the State
For now, we will keep this extremely simple and save the state to a JSON file.
public class StateStore
{
private readonly string _stateDirectory;
private readonly string _stateFile;
private readonly JsonSerializerOptions _jsonOptions = new()
{
WriteIndented = true,
Converters = { new JsonStringEnumConverter() }
};
public StateStore()
{
_stateDirectory = Path.Combine(
Environment.CurrentDirectory,
".pandacloud");
_stateFile = Path.Combine(
_stateDirectory,
"state.json");
}
public async Task<CloudState> LoadAsync()
{
if (!File.Exists(_stateFile))
return new CloudState();
var json = await File.ReadAllTextAsync(_stateFile);
return JsonSerializer.Deserialize<CloudState>(json, _jsonOptions)
?? new CloudState();
}
public async Task SaveAsync(CloudState state)
{
Directory.CreateDirectory(_stateDirectory);
var json = JsonSerializer.Serialize(state, _jsonOptions);
await File.WriteAllTextAsync(_stateFile, json);
}
}
At startup, we load the current state.
After modifying the infrastructure, we save it again.
Would a real cloud store its entire Control Plane state in one JSON file?
Obviously not.
The state changes constantly, several operations can happen concurrently, and we eventually need stronger consistency guarantees.
But for PandaCloud at this stage, JSON is perfectly good enough.
It lets us understand the important part first.
The Control Plane needs memory.
Let's Actually Test It This Time
Now we finally have everything we need.
Let's create Alice.
PS C:\Users\lgirardin\source\repos\PandaCloud> dotnet run -- tenant create alice
> sudo ip link show br-alice >/dev/null 2>&1 || sudo ip link add br-alice type bridge
> sudo ip link set br-alice up
Tenant alice created with subnet 10.10.0.0/24 and gateway 10.10.0.1.
PS C:\Users\lgirardin\source\repos\PandaCloud> dotnet run -- tenant list
TENANT NETWORK BRIDGE
--------------------------------------------------------
alice 10.10.0.0/24 br-alice
It looks good.
But let's not blindly trust the Control Plane.
Let's ask Linux directly.
lgirardin@lgirardin-pc:~$ ip link show br-alice
22: br-alice: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc noqueue state DOWN mode DEFAULT group default qlen 1000
link/ether d2:5b:d8:9c:56:59 brd ff:ff:ff:ff:ff:ff
The bridge exists.
It is administratively UP, but operationally still DOWN because no active interface is connected to it yet.
Let's create a few nodes.
dotnet run -- node create Node25 12
dotnet run -- node create Node26 12
dotnet run -- node create Node27 6
dotnet run -- node create Node28 6
dotnet run -- node list
NODE STATE TENANT IP CPU
--------------------------------------------------------------------
Node25 Available - - 12
Node26 Available - - 12
Node27 Available - - 6
Node28 Available - - 6
Now let's allocate them to Alice.
PS C:\Users\lgirardin\source\repos\PandaCloud> dotnet run -- node allocate node25 alice
> if ! sudo ip link show v-Node25-h >/dev/null 2>&1; then sudo ip link add v-Node25-h type veth peer name v-Node25-n && sudo ip link set v-Node25-n netns Node25; fi
> sudo ip link set v-Node25-h master br-alice
> sudo ip link set v-Node25-h up
> sudo ip netns exec Node25 ip link set lo up
> sudo ip netns exec Node25 ip link set v-Node25-n up
> sudo ip netns exec Node25 ip addr flush dev v-Node25-n
> sudo ip netns exec Node25 ip addr add 10.10.0.10/24 dev v-Node25-n
Node node25 allocated to tenant alice with IP address 10.10.0.10.
I'll let you do the other three.
Eventually, we should end up with:
PS C:\Users\lgirardin\source\repos\PandaCloud> dotnet run -- node list
NODE STATE TENANT IP CPU
--------------------------------------------------------------------
Node25 Allocated alice 10.10.0.10 12
Node26 Allocated alice 10.10.0.11 12
Node27 Allocated alice 10.10.0.12 6
Node28 Allocated alice 10.10.0.13 6
Again, this is what the Control Plane tells us.
Let's verify reality.
First, the network namespaces:
sudo ip netns list
Node28 (id: 12)
Node27 (id: 11)
Node26 (id: 10)
Node25 (id: 9)
Now let's check the tenant bridge.
ip link show br-alice
22: br-alice: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
link/ether d2:5b:d8:9c:56:59 brd ff:ff:ff:ff:ff:ff
This time the bridge is operationally UP because active interfaces are connected to it.
Host Side
We can inspect the interfaces connected to the bridge:
ip link show master br-alice
or:
bridge link show
We should see our host-side veth interfaces:
24: v-Node25-h@if23: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br-alice state forwarding priority 32 cost 2
26: v-Node26-h@if25: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br-alice state forwarding priority 32 cost 2
28: v-Node27-h@if27: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br-alice state forwarding priority 32 cost 2
30: v-Node28-h@if29: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 master br-alice state forwarding priority 32 cost 2
Node Side
Now let's actually look inside Node25.
sudo ip netns exec Node25 ip addr show
Or only inspect its PandaCloud interface:
sudo ip netns exec Node25 ip addr show v-Node25-n
We can also inspect its routing table:
sudo ip netns exec Node25 ip route
And finally, let's verify that two nodes belonging to Alice can communicate with each other.
sudo ip netns exec Node25 ping -c 3 10.10.0.11
If everything is working properly, Node25 should be able to reach Node26 directly through br-alice.
And that brings us to the end of our Control Plane for now.
We could keep testing edge cases, but the important architecture is starting to appear.
We now have:
- infrastructure operations
- tenant networks
- nodes
- resource allocation
- persistent state
- idempotent operations
- a service layer validating requests
But our Control Plane is still far from finished.
In the next part, we will start looking at capacity allocation.
Top comments (0)