Spinifex 1.19.0: Nineteen Releases Later

After five months and nineteen releases, we have shipped Spinifex 1.19.0. In this time we have taken the platform from supporting three AWS services into eleven, and turned a claim about production readiness into real production-tested code.

When we released 1.0.0 at the end of March, we only supported EC2, EBS and S3. You could launch an instance, attach a volume, and put an object. There were no load balancers, no Kubernetes, no containers and no databases. Since then we have expanded significantly, going from 129 supported actions to 371.

Eleven services

What arrived since 1.0.0, on top of the EC2, EBS and S3 we started with, plus VPC and IAM:

• ALB / NLB (1.1.0) — L7 on HAProxy and L4 on nginx, HTTPS listeners, listener rules, health checks
• EKS (1.7.0) — K3s control plane, managed node groups, IAM-authenticated kubectl, IRSA, add-ons
• ACM (1.7.0) — Certificate import, then issuance and renewal from a tenant Private CA
• ECR (1.9.0) — OCI registry for docker, crane and skopeo, lifecycle policies, immutable tags
• ECS (1.10.0) — Task definitions, services, awsvpc ENI-per-task, capacity providers, task IAM roles
• RDS (1.15.0) — Managed PostgreSQL 18.1 and MariaDB, snapshots, parameter groups, backups
• Ochre (Bedrock) (work in progress) — Converse and InvokeModel, knowledge bases, guardrails, self-hosted weights on GPU

Load balancers were the first thing we added. ALB runs on HAProxy, NLB on nginx stream, and both handle health checks and HTTPS listeners with ACM certificates. Listener rules route on host, path, header, method or source IP, so you can put one load balancer in front of several services and split traffic the way you would on AWS.

EKS was the hardest of these to build, and the first major milestone. Aim eksctl or stock terraform-aws-eks at a Spinifex endpoint and you get a cluster, with a K3s control plane doing the work underneath. aws eks get-token authenticates your kubectl against IAM access entries, so there is no aws-auth ConfigMap to keep in sync. Pods assume IAM roles through IRSA, backed by a per-cluster OIDC provider.

It started out single-node. In 1.8.0 the control plane ran three etcd servers on separate hosts, and by 1.15.0 those spread across availability zones, so losing an AZ no longer takes the cluster with it. Persistent volumes work through the aws-ebs-csi-driver add-on backed by Viperblock: a gp3 PVC binds, mounts, and survives a pod reschedule. The AWS load balancer controller gives you real L7 ALB ingress. If you need GPUs, 1.12.0 exposed them to pods through the device plugin, and threw in etcd snapshot and restore, so you can rebuild a whole cluster from a snapshot.

ECS and ECR came in together, and we built them for sites with no internet. Workers pull images from the internal registry without needing external DNS, which is the difference between working and not working at an air-gapped site. Under awsvpc each task gets its own ENI and VPC IP, assumes IAM roles through the task credential endpoint, and registers with ELBv2 target groups so your containers sit behind a load balancer. Reboot a host and the ECS agent picks its containers back up. Delete a cluster and it tears itself down instead of stranding task definitions and container instances.

RDS landed with PostgreSQL in 1.15.0 and MariaDB in 1.17.0. Each database sits on a private endpoint inside your VPC, scoped by security group, with no public address anywhere. Subnet groups decide where that endpoint lands and parameter groups tune the engine. Backups run in your window and get swept once they pass retention, and you can take a snapshot by hand and restore it into a new instance.

Core EC2 functionality kept growing amongst all the new features: capacity reservations in 1.9.0, spot instances and AMIs from snapshots in 1.11.0 and launch templates in 1.12.0. GPU passthrough covers NVIDIA and AMD, with MIG slicing if you want to carve a card into pieces. Tagging now works the way you expect across instances, volumes, snapshots, AMIs, key pairs, VPCs and gateways, at creation or after the fact, with tag: filters on describes.

Identity, and why a stock cloud image boots

In 1.0.0, IAM had users, policies and SigV4; the absolute minimum. Today it implements 75 of the 159 operations in the IAM API: the core of users, groups, roles, policies, instance profiles and OIDC providers. Most of what is missing is stuff that doesn't fit as part of a self hosted deployment, so MFA devices, login profiles, password policies, SAML federation and credential reports. The real gap you are most likely to hit from Terraform is policy versioning, and it is on the TODO list.

Roles and instance profiles came first, in 1.6.0, with STS on top of them: AssumeRole, GetCallerIdentity, GetSessionToken, and the AssumeRoleWithWebIdentity that IRSA on EKS depends on. Groups followed in 1.11.0, members inheriting their permissions, and you can attach managed policies or write inline ones on users, roles and groups alike. Tagging arrived across users, roles, policies, instance profiles and OIDC providers in the same release, and 1.12.0 put inline policy editors in the console. From 1.8.0 the console itself dropped static long-lived keys and instead signs in with short-lived STS session credentials.

Registering an operation is the easy part. The hard part is enforcement: getting every service to reach the same verdict on the same policy, whether the caller is a user, a role session, an instance holding IMDS credentials, or a pod holding an IRSA token.

At 1.8.0 a session from AssumeRole was blanket-denied on the control plane, so instance-role credentials coming out of IMDS could authenticate and then do nothing. That release started authorizing those sessions against the policies attached to the role, which is what made IMDS credentials useful. Inline role policies followed in 1.10.0, enforced by S3 rather than stored and ignored, and 1.11.0 extended that to inline policies on users and to permissions a user inherits from a group. ECR began enforcing IAM on registry operations in 1.13.0. 1.15.0 enforced iam:PassRole on ECS task and service role ARNs, and scoped rds:* by principal class and resource.

1.18.0 and 1.19.0 brought policy evaluation up to the AWS model. Policies evaluate against the resource ARNs a request names, across every service, so a policy scoped to one bucket or one instance authorises that one. Condition blocks are enforced, and S3 supplies condition context keys on the data path so bucket and object policies can use them the way they do on AWS. Policy variables like ${aws:username} and ${aws:userid} resolve before matching, and one that cannot resolve fails closed. sts:AssumeRole is gated on the caller's own identity policy, PassRole authorizes against the stored role ARN, and a role or policy ARN handed in by a caller is checked against the stored canonical form and re-anchored onto that caller's account, so nobody reaches into another tenant's namespace by writing a convincing string.

IMDS is the piece that makes an unmodified cloud image work. Each instance gets its own metadata endpoint on the link-local address, served on a per-tap datapath, and cloud-init's stock Ec2 datasource reads it the way it does on AWS. Boot a stock Ubuntu image and it collects its own hostname, network configuration, SSH keys and user-data at runtime. One AMI serves every instance and you customise nothing.

The rest of it matters once you are moving real workloads across. A client pinning any dated IMDS version resolves the same metadata tree, so SDKs and cloud-init work against it unchanged. You can drop an instance back to IMDSv1 with HttpTokens=optional when something old needs it. And since 1.17.0 IMDS serves the deployment CA at /spinifex/ca.pem, so rotating certificates reaches your guests without anyone rebuilding an image.

Pointing Terraform at it

The whole promise falls over if your existing tooling can tell the difference, so a large share of these nineteen releases went into behaviour you notice by nothing going wrong.

Idempotency came first. A retried create carrying a ClientToken stops duplicating resources across EC2, EKS and ELBv2, and 1.19.0 extended that to seven more EC2 creates plus RunTask and CreateLaunchTemplateVersion. Deletes are idempotent on already-deleted resources, so a tofu destroy that half-failed converges when you run it again rather than erroring on the resources it already removed.

Error codes carry more weight than you would think, since the SDK's retry logic reads them. InsufficientInstanceCapacity used to return a 4xx, which meant SDK retries never fired, so it moved to 503 alongside the other capacity errors. AWS error codes propagate through nested wraps rather than collapsing into something generic. DeleteSecurityGroup names the resource blocking it in the DependencyViolation.

Underneath all of that sits a conformance harness. Every successful modelled response the integration suite produces gets validated against the pinned aws-sdk-go models: shapes, required members, types. Error envelopes are checked against the service models where those declare errors. EC2's model declares none, so EC2 is checked against a catalog curated from AWS's published error reference, including the documented split between 4xx client and 5xx server codes.

Performance

The load balancer microVM was the first big win. In 1.4.0 we moved it to QEMU direct boot and dropped its start time from 3.2 seconds to 264 milliseconds, which also removed the system AMI import from the install path.

Block storage came next and took longer. 1.12.0 bounded parallel chunk uploads with coalesced checkpoints and removed write amplification from the live checkpoint path. 1.15.0 replaced the linear extent scan with an ordered index, batched read resolution, repaired the persisted-data cache, and narrowed the write-path lock so concurrent volume throughput stopped serialising. 1.16.0 made multipart uploads stream, so a large object no longer costs memory in proportion to its size. 1.17.0 started serving erasure-coded reads from parity and hedging reads across shards, which keeps one stalled node from setting the latency of every read.

1.19.0 finishes two more. QEMU now maps virtio-blk virtqueues across an IOThread pool sized from the vCPU count, where boot volumes used to run every queue through a single thread. And a node can put each volume's write-ahead log on its own device. That second one matters more than it sounds: on our end-to-end stack, competing filesystem traffic alone moves fsync p99 from 4.7 ms to 22 ms with no change to Viperblock at all, and an etcd guest reports that tail as its commit latency. Set wal_base_dir and the WAL leaves the shared queue.

The reconcile loops got the same treatment in 1.19.0. DNS, the VPC drift loop, the ECS scheduler and the EKS cluster reconciler all ran on tickers, which put a timer between your API call and the datapath agreeing with it. They now wake on the change. For a dropped VPC event, repair latency falls from a uniform 0 to 300 seconds to the debounce plus a five-second floor. We kept the floor on purpose, because a pass loads eight buckets and scans the OVN northbound database, and a launch storm would otherwise turn a five-minute scan into a continuous one.

Two smaller ones from the same release: EBS metadata now partitions by owning account, which cuts describe latency on clusters carrying many tenants, and IMDS resolves credentials on the node, removing a network hop from every instance credential fetch.

Reliability

The key focus across these nineteen releases has been ensuring Spinifex handles failure like an air gapped system should. Three examples spread across five months:

1.3.0 taught the daemon to survive a NATS outage. It keeps serving from local instance state when the control plane is unreachable, starts in two phases with a background reconnect, and offers operators a read-only /local/* API in standalone mode.

1.10.0 made shutdown safe. Stopping the service or the host drains running guests first, so a VM filesystem does not lose the storage underneath it mid-write. 1.17.0 narrowed that further, so stopping one node drains that node's guests instead of powering down guests across the cluster.

1.19.0 pauses a guest that hits a backend I/O error rather than handing it EIO, which turns a transient storage fault into a stall you can recover from. A node upgrade refuses to continue when a volume seal fails, instead of reporting success and leaving an unbootable root volume. Instances on a node that stopped answering report impaired, and DescribeInstances judges completeness by which node replied rather than by counting frames, so a partial fan-out stops looking like a full answer.

Between those, the platform learned to bound its own resource use, reclaim its own space, and report exhaustion rather than hide it.

Security

FIPS 140-3 cryptography has been enforced at startup across every binary since 1.4.0, on the Go Cryptographic Module v1.0.0 under CMVP certificate #5247. 1.5.0 set a TLS 1.3 floor with hybrid post-quantum key exchange (X25519 with ML-KEM). At-rest volume encryption went on by default for fresh installs in 1.8.0.

1.17.0 added an nftables host firewall that scopes cluster ports to known peers, so OVN, NATS and the storage listeners stopped being reachable from anything that could route to the node. The peer set reconciles from the daemon at startup, so replacing a node keeps the rules right.

The network defaults now fail closed too. Security group default-deny rules match every ethertype, unsupported protocols are rejected instead of emitting an allow-all rule, and unscoped accepts are family-qualified to IPv4. Outbound SMTP to ports 25, 465 and 587 is blocked for public destinations by default, matching AWS, with a per-VPC exemption an operator holds. A node's resolver stopped answering recursive queries for arbitrary clients.

Where that leaves us

1.0.0 was the release we were willing to put a 1 in front of. 1.19.0 is where Spinifex is ready to tackle almost all workloads you can throw at it.

Get it

# Bare metal from cold: write the ISO to a USB stick and boot it
curl -fsSLO https://iso.mulgadc.com/spinifex.iso

# Or install onto an existing Linux host
curl -fsSL https://install.mulgadc.com | bash

Check the release notes before you upgrade an existing cluster. The documentation site covers single-node, multi-node and air-gapped installs, and the live sandbox gives you a real endpoint in under a minute if you would rather not install anything yet.

If you are running Spinifex somewhere interesting, tell us about it.

Comments

[Tom]

Very well written and informative! The EKS side is particularly impressive.