DEV Community: Jonathan Pitter

The rollback endpoint took a deployment ID and did nothing with it

Jonathan Pitter — Mon, 27 Jul 2026 18:55:55 +0000

This is a submission for DEV's Summer Bug Smash: Clear the Lineup powered by Sentry.

Project Overview

Staxa is a multi-tenant deployment platform I am building solo under Stackforge Labs. The backend is a single Go binary (staxad) using the chi router, with about 60 API endpoints, running on K3s on a Hetzner CAX21 ARM64 server that costs around $11/month. Each tenant gets an isolated Kubernetes namespace with their own app container, a PostgreSQL 16 or MySQL 8 database, a subdomain with automatic SSL, and resource quotas. Container builds run through Buildah, and the frontend is Next.js (App Router) with shadcn/ui and Clerk for auth.

Bug Fix or Performance Improvement

The symptom: POST /api/v1/tenants/{id}/deployments/{depId}/rollback accepted a deployment ID in the URL path and then completely ignored it. Whatever version you asked for, you got the most recent successful deployment instead.

The route was wired up correctly in internal/api/router.go:149:

r.Post("/tenants/{id}/deployments/{depId}/rollback", srv.handleRollbackDeployment)

But handleRollbackDeployment never called chi.URLParam(r, "depId"). It read {id} for the tenant and stopped there.

How I found it: I was auditing my published API docs against the actual handlers, endpoint by endpoint. When I got to the rollback entry I went to write down what {depId} did, went to the handler to confirm, and found nothing reading it. The docs described an ID that the code never looked at.

The worst part is that it returned 202 Accepted and then performed a real, successful rollback. Just not the one you asked for. There was no error to notice, no failed request in any log. The frontend had been passing the deployment ID into the URL since it was written (src/lib/api.ts), so the UI always believed the parameter was honored.

Root cause: the handler created a rollback deployment row with no reference to any target, and the worker independently decided what to restore. In internal/worker/pipeline.go, runRollbackPipeline did this:

prevDep, err := cfg.Store.GetPreviousDeployment(ctx, tenant.ID)
if err != nil {
    return failPipeline(ctx, cfg, dep, "deploy",
        fmt.Errorf("no previous successful deployment to roll back to: %w", err))
}

GetPreviousDeployment was a single query: most recent deployment for this tenant where status='success' AND is_current=FALSE, ordered by version descending, limit 1. It is a perfectly good "roll back one version" implementation. It just is not "roll back to the version the user named," and nothing in between the HTTP layer and the worker was carrying that information. (That query has since changed again, for reasons in the last section.)

Why it mattered: rollback is the button you press when production is broken. If v9 is bad you might roll back to v8, find that also bad, and go for v5. On Staxa you would click v5 and get v8 again, three times in a row, while your tenant stayed down. Meanwhile the deployment history would show three successful rollbacks.

The row it wrote was misleading too. It hardcoded ImageTag: "rollback", a placeholder that is not a real image reference, so the deployment history could not tell you what had actually been deployed even after the fact.

Then fixing it exposed a second, older bug underneath. More on that below.

Code

Three PRs on staxa-api-v2:

#21 fix(deployments): roll back to the deployment named in the path (merged) - the fix itself, 332 insertions across 4 files
#22 fix(deployments): scope is_current to the service, not the tenant - the schema bug the fix exposed
#24 feat(deployments): snapshot deployment config so rollback replays it - the design problem the fix exposed

The handler, before and after

internal/api/deployments.go. Before:

func (s *Server) handleRollbackDeployment(w http.ResponseWriter, r *http.Request) {
    provider := middleware.GetProvider(r.Context())
    tenantID := chi.URLParam(r, "id")
    // depId is never read

    if err := s.assertTenantOwnership(r, provider.ID, tenantID); err != nil {
        pkgapi.ErrNotFound(w, r, "tenant not found")
        return
    }

    version, err := s.store.GetNextDeploymentVersion(r.Context(), tenantID, "")
    if err != nil {
        pkgapi.ErrInternal(w, r)
        return
    }

    d := &store.Deployment{
        ID:         id.Generate("dep"),
        TenantID:   tenantID,
        ProviderID: provider.ID,
        DeployType: "rollback",
        Trigger:    "api",
        Status:     "pending",
        ImageTag:   "rollback",
        Version:    version,
        Metadata:   map[string]any{},
        CreatedAt:  time.Now(),
    }
    // ... create + enqueue
}

After:

func (s *Server) handleRollbackDeployment(w http.ResponseWriter, r *http.Request) {
    provider := middleware.GetProvider(r.Context())
    tenantID := chi.URLParam(r, "id")
    depID := chi.URLParam(r, "depId")

    if err := s.assertTenantOwnership(r, provider.ID, tenantID); err != nil {
        pkgapi.ErrNotFound(w, r, "tenant not found")
        return
    }

    target, err := s.store.GetDeployment(r.Context(), depID)
    if err != nil || target.TenantID != tenantID {
        pkgapi.ErrNotFound(w, r, "deployment not found")
        return
    }
    if err := validateRollbackTarget(target); err != nil {
        pkgapi.ErrUnprocessable(w, r, err.Error())
        return
    }

    version, err := s.store.GetNextDeploymentVersion(r.Context(), tenantID, "")
    if err != nil {
        pkgapi.ErrInternal(w, r)
        return
    }

    // The new deployment carries the target's image and source so the row is an
    // honest record of what ends up running; the pipeline reads the target back
    // out of metadata to decide what to deploy.
    d := &store.Deployment{
        ID:         id.Generate("dep"),
        TenantID:   tenantID,
        ProviderID: provider.ID,
        ImageTag:   target.ImageTag,
        SourceRef:  target.SourceRef,
        SourceType: target.SourceType,
        DeployType: "rollback",
        Trigger:    "api",
        Status:     "pending",
        Version:    version,
        Metadata: map[string]any{
            "rollback_to":         target.ID,
            "rollback_to_version": target.Version,
        },
        CreatedAt: time.Now(),
    }
    // ... create + enqueue
}

The validation rules are their own function so both the handler and the worker can apply them:

// validateRollbackTarget reports whether a deployment can be rolled back to.
// Only a deployment that finished successfully has an image known to run, and
// only a tenant-level one can be restored by the rollback pipeline (that
// pipeline redeploys the tenant's primary workload, so restoring one service's
// image through it would deploy that image tenant-wide).
func validateRollbackTarget(d *store.Deployment) error {
    if d.Status != "success" {
        return errors.New("can only roll back to a deployment that completed successfully")
    }
    if d.ServiceID != "" {
        return errors.New("cannot roll back to a service-scoped deployment, redeploy the service instead")
    }
    // "pending" and "rollback" are placeholders written at create time; a
    // deployment carrying one never had a real image assigned to it.
    switch d.ImageTag {
    case "", "pending", "rollback":
        return errors.New("deployment has no image to roll back to")
    }
    return nil
}

The worker

internal/worker/pipeline.go. The GetPreviousDeployment call became:

// resolveRollbackTarget returns the deployment whose image this rollback should
// restore. The API records the caller's chosen target under "rollback_to" in the
// rollback deployment's metadata; jobs enqueued before that field existed carry
// no target, so those fall back to the most recent successful deployment.
func resolveRollbackTarget(ctx context.Context, cfg WorkerConfig, dep *store.Deployment, tenantID string) (*store.Deployment, error) {
    targetID, _ := dep.Metadata["rollback_to"].(string)
    if targetID == "" {
        prev, err := cfg.Store.GetPreviousDeployment(ctx, tenantID)
        if err != nil {
            return nil, fmt.Errorf("no previous successful deployment to roll back to: %w", err)
        }
        return prev, nil
    }

    target, err := cfg.Store.GetDeployment(ctx, targetID)
    if err != nil {
        return nil, fmt.Errorf("rollback target %s not found: %w", targetID, err)
    }
    // Re-check what the handler checked. The target could have been deleted or
    // rewritten between enqueue and execution, and deploying the wrong image is
    // worse than failing the rollback.
    if target.TenantID != tenantID {
        return nil, fmt.Errorf("rollback target %s belongs to another tenant", targetID)
    }
    // Mirrors validateRollbackTarget: this pipeline deploys the result as the
    // tenant's primary workload, so a service-scoped row would be pushed
    // tenant-wide and then marked current in the service's is_current scope.
    if target.ServiceID != "" {
        return nil, fmt.Errorf("rollback target %s is service-scoped and cannot be restored tenant-wide", targetID)
    }
    switch target.ImageTag {
    case "", "pending", "rollback":
        return nil, fmt.Errorf("rollback target %s has no image to restore", targetID)
    }
    return target, nil
}

That ServiceID check is a late addition. It came out of code review, and the story behind it is below.

The schema bug underneath (PR #22)

internal/store/migrations/004_deployments.sql:28, untouched since I wrote it:

CREATE UNIQUE INDEX idx_deployments_current ON deployments(tenant_id, is_current) WHERE is_current = TRUE;

At most one current deployment per tenant. Migration 027 replaces it:

DROP INDEX IF EXISTS idx_deployments_current;

CREATE UNIQUE INDEX idx_deployments_current
    ON deployments (tenant_id, COALESCE(service_id, ''))
    WHERE is_current = TRUE;

And SetCurrentDeployment in internal/store/deployments.go went from clearing the flag across the whole tenant:

// before
tx.Exec(ctx, `UPDATE deployments SET is_current=FALSE WHERE tenant_id=$1 AND is_current=TRUE`, tenantID)
tx.Exec(ctx, `UPDATE deployments SET is_current=TRUE WHERE id=$1`, deploymentID)

to deriving its scope from the target row, in one statement:

// after
_, err := s.pool.Exec(ctx, `
    UPDATE deployments d SET is_current = (d.id = $2)
    FROM deployments t
    WHERE t.id = $2
      AND d.tenant_id = $1
      AND COALESCE(d.service_id, '') = COALESCE(t.service_id, '')
      AND (d.is_current OR d.id = $2)`,
    tenantID, deploymentID,
)

The signature is unchanged, so all five existing call sites got the correct scoping without edits.

My Improvements

Diagnosing it

The diagnosis was trivial once I looked. What is worth writing down is why I had not looked for four months. Rollback worked. I had tested it by deploying twice, clicking rollback, and watching the previous version come back up. With exactly two deployments, "the target you named" and "the most recent successful one" are the same row. My manual test could not distinguish the correct behavior from the bug, and I never went back with three versions.

Tracing it took two greps: grep -rn "depId" internal/api/ showed handleGetDeployment reading it and handleRollbackDeployment not, and reading runRollbackPipeline showed where the decision was actually being made.

How to get the target to the worker

Jobs go through a Redis list as queue.Job structs. The obvious move was adding a TargetDeploymentID field to queue.Job.

I put it in the deployment's Metadata map instead, for three reasons. Jobs are transient and the deployments table is not, so the metadata answers "what did that rollback restore?" months later, when the job is long gone. It survives a requeue, since the pipeline re-reads the deployment row by ID and the target comes back with it. And it is already serialized into the API response, so the caller sees rollback_to and rollback_to_version without me adding response fields.

The cost is that it is a map[string]any, so the read is an unchecked type assertion (dep.Metadata["rollback_to"].(string)) instead of a typed field. I decided the durability was worth losing the compile-time check, and resolveRollbackTarget treats a missing or non-string value as "no target given" and falls back, which is also the migration path for jobs already sitting in Redis when the binary is replaced.

Validating twice on purpose

validateRollbackTarget runs in the handler, and resolveRollbackTarget re-checks tenant ownership and the image tag in the worker. That is deliberate duplication. Between the enqueue and the execution the target row can be deleted or rewritten, and jobs can retry minutes later. Deploying the wrong image is worse than failing the rollback, so the worker does not trust the ID it was handed.

The bug the fix exposed

Rejecting service-scoped targets was correct for the pipeline as written, since runRollbackPipeline calls Orchestrator.DeployApp, which redeploys the tenant's primary workload. Restoring one service's image through it would push that image tenant-wide.

But it meant multi-service tenants now had no rollback at all, because every deployment after tenant creation is service-scoped. Before my fix the same call "worked" only by accident: GetPreviousDeployment filtered on tenant_id alone, so it could pick up a service-scoped row and deploy that service's image as the entire tenant. I had turned a silent corruption into a loud 422, which is an improvement, but it is still a regression from the user's side.

Chasing why a service row could surface there at all led to the index above. is_current was tenant-scoped at the schema level, left over from before Staxa supported multiple services per tenant. Migration 011 had already made the sibling version index service-aware, so this one was just missed.

The masking was almost funny: runServiceScopedRedeploy never called SetCurrentDeployment at all, so no service deployment was ever marked current, so the unique index was never violated and nobody noticed it was wrong. Two bugs cancelling out. Adding the missing call without fixing the index would have started throwing SQLSTATE 23505 on the second service deploy.

I confirmed the tests actually catch it by deleting migration 027 and re-running them:

--- FAIL: TestDeploymentsCurrentPerService/two_services_in_one_tenant_can_both_be_current
    Error: Received unexpected error:
    ERROR: duplicate key value violates unique constraint "idx_deployments_current" (SQLSTATE 23505)

That check is the one I would skip if I were rushing, and it is the one that proves the test is testing anything.

What code review caught that I missed

I had fixed the index and written the service-scoped GetPreviousServiceDeployment, and I considered the scoping problem closed. A reviewer pointed out that I had made the new function service-safe and left its older sibling alone, and that the two now interacted badly.

GetPreviousDeployment still filtered on tenant_id, status='success' and is_current=FALSE, with nothing excluding service rows. Marking service deployments current, which my fix had just started doing, removes only each service's currently running row from candidacy. Every superseded service row remains a candidate. Worse, version is a per-(tenant, service) counter from GetNextDeploymentVersion, so ORDER BY version DESC was ranking independent sequences against each other: a service's v7 outranks a tenant-level v3.

So the exact failure I thought I had closed was still reachable. runRollbackPipeline would deploy that service's image as the whole tenant, then call SetCurrentDeployment, which after migration 027 marks it current in the service's scope, leaving the tenant scope with no current deployment at all.

The fix is one clause:

WHERE tenant_id=$1 AND service_id IS NULL AND status='success' AND is_current=FALSE

Both problems collapse into it. The filter excludes service rows outright, and restricting to one scope makes the version ordering compare a single counter again, so the mixed-sequence problem disappears rather than needing its own fix.

The exposure was narrower than it first looked, and I want to be accurate about that: validateRollbackTarget in the handler already rejected service-scoped targets, and after this fix the API always writes rollback_to, so the only path that reached the bad query was the metadata-less fallback for jobs enqueued before that field existed. I fixed it rather than filing it, because the fallback is permanent code and the failure mode is silent. Reviewing my own diff also showed resolveRollbackTarget re-checking tenant and image tag under a comment reading "re-check what the handler checked" while skipping scope, which was the one check that mattered here. That is the ServiceID guard in the code above.

The lesson I took is narrower than "get your code reviewed." I had written the new function carefully and never re-read the old one next to it, because in my head the work was "add service scoping" rather than "make this table's queries agree about scope." Adding a dimension to a data model is not done when the new code handles it. It is done when the code that predates it has been checked against it.

The design problem underneath that

Once rollback restored the image you asked for, a different question surfaced: which configuration does it restore? It was deploying the old image with today's environment variables. If a bad env change caused the incident, rolling back did not undo it.

I went with Octopus Deploy's release model, since that matches how I think about deploys: a deployment is an immutable snapshot of its inputs, and rollback means deploying that snapshot again. That is PR #24, with refresh_config: true as the explicit escape hatch, mirroring Octopus's "update variables" action.

The interesting tradeoff was retention. Snapshots duplicate secret material, so keeping them forever means rotating a leaked secret never purges every copy. My first version kept the last three snapshots and fell back to current config beyond that.

The better answer came from noticing that the bound only ever existed because of secrets. Ports, resource sizes, health check config and non-secret env are small and not sensitive. My env_vars table already had an is_secret column, so I split the snapshot: config and the names of the secret variables are kept for the life of the deployment, and only the secret values are bounded. A rollback past the window restores everything else and resolves the secret names against their current values.

That also turned out to be more correct than replaying, not just cheaper. If you rotate an API key between v1 and v9, restoring v1's key breaks the app. For rotated credentials, resolving to current is the behavior you actually want.

One case stays genuinely lossy and now fails loudly: a variable that was secret then and does not exist now can be filled from neither side, so the rollback fails naming the keys rather than starting the app without values it needs.

Verifying

Unit tests with a mock store for the handler (target not found, target belonging to another tenant, unsuccessful target, placeholder image tag, happy path asserting metadata and image tag) and the pipeline (metadata target wins over GetPreviousDeployment, and a bad target deploys nothing at all, which I assert explicitly since "fails" and "fails after deploying the wrong thing" are very different outcomes).

Three existing tests broke when I made the change, all with 404s, because they only set the id URL param and never depId. That was a good sign: they had been passing against a handler that ignored the parameter.

The schema work needed real Postgres, since a mock store cannot catch a unique index regression. That went into the testcontainers-go suite behind -tags integration: two services in one tenant both holding a current deployment, promoting one service leaving the other untouched, tenant-level promotion not disturbing service rows, and GetPreviousServiceDeployment ignoring other services and the running deployment.

The scoping fix from review got TestGetPreviousDeploymentIgnoresServiceScopes, and I ran the same check on it by reverting only the SQL clause. It fails on the returned row's scope:

Error: Should be empty, but was svc_9697128d7951c74c24b2c46b
Messages: must not return a service-scoped row

The setup deliberately deploys the service enough times to run its version counter past the tenant's, so the mixed-sequence ordering is what is under test rather than just the presence of service rows. A test that only created one service deployment would pass for the wrong reason.

What I would do differently

Two things.

First, my manual test was shaped so it could not fail. Two deployments made the correct behavior and the bug identical. Now when I test anything that selects one item from a history, I use at least three and pick the middle one, because off-by-one and ignore-the-parameter bugs both hide at n=2.

Second, I found this by reading my own API docs against my own code. That audit found several other drifts the same afternoon (a response envelope documented with the wrong shape, an endpoint documented as replacing a set of environment variables when the SQL is INSERT ... ON CONFLICT DO UPDATE and never deletes anything). Writing docs from the handler rather than from memory is a decent bug-finding technique on its own, and I would rather do it on a schedule than stumble into it.

Third, when I add a dimension to a data model I now grep for every query against that table instead of trusting that I have thought of them. The review catch above cost one clause to fix and would have cost nothing to find, if I had listed the callers rather than reasoning about the two functions I happened to be looking at.

The broader lesson is about silent success. None of these bugs produced an error. The original returned 202 every time, the schema bug was masked by a second bug that kept it from ever firing, and the query scoping issue would have left a tenant with no current deployment without anything logging that. The ones that hurt are not the ones that throw. They are the ones that confidently do the wrong thing and report that it went fine.

I Run a Full PaaS on One $11/Month ARM Server. Here's What the Resource Usage Actually Looks Like.

Jonathan Pitter — Tue, 30 Jun 2026 16:35:51 +0000

A managed Kubernetes cluster on AWS starts at ~$73/month, just the control plane, before you run anything on it. Add a load balancer, a managed Postgres instance, and a few worker nodes and you're at $300-400/month before your first tenant deploys.

I run a multi-tenant deployment platform on a single Hetzner ARM server that costs €10.99/month. That includes the API server, dashboard, build pipeline, PostgreSQL, Redis, container registry, ingress controller, automatic SSL, and room for 10-25 tenant environments.

Here's the full breakdown.

The Server

Hetzner CAX21, one of their Ampere ARM64 cloud instances:

4 vCPU (ARM64)
8 GB RAM
80 GB NVMe SSD
20 TB traffic included
€10.99/month (~$11)

For context, the closest equivalent on AWS would be a t4g.large (2 vCPU, 8GB ARM) at ~$49/month on-demand, or a t4g.xlarge (4 vCPU, 16GB) at ~$98/month. That's a 4.5-9x price difference for comparable compute.

Why ARM? Every piece of software I'm running (Go, K3s, Traefik, PostgreSQL, Redis, Node.js, and Buildah) has native ARM64 support. Container images need to target linux/arm64, but the tooling handles this transparently. There's no performance penalty for my workload. The only thing I gave up was familiarity. Most tutorials and Stack Overflow answers assume x86. That cost me a few hours of debugging early on and nothing since.

What's Running on 8 GB

Here's the actual memory layout. I'm running K3s (lightweight Kubernetes), so everything is organized into namespaces.

System Components (~1.5 GB)

These are the Kubernetes infrastructure services that run outside the staxa-system namespace:

Component	What it does	RAM
k3s server	Kubernetes API server, scheduler, controller manager, etcd replacement (SQLite)	~500 MB
CoreDNS	Cluster DNS, resolves service names inside the cluster	~50 MB
cert-manager	Watches for new Ingress/IngressRoute resources and provisions Let's Encrypt certificates automatically	~50 MB
local-path-provisioner	Storage, creates PersistentVolumes backed by the server's NVMe SSD	~30 MB
Miscellaneous	kubelet overhead, pause containers, system daemons	~200 MB

Total system footprint: ~1-1.5 GB out of 8 GB.

K3s deserves special mention here. A full Kubernetes distribution (kubeadm + etcd) would use 2-3 GB for the control plane alone. K3s gets this under 600 MB by replacing etcd with SQLite, stripping unused cloud provider code, and shipping as a single ~70 MB binary. On a constrained server, that 1.5 GB savings is the difference between running the platform and not.

Platform Services (~750 MB)

These are the services that make up the actual deployment platform, all in the staxa-system namespace. These aren't estimates. This is kubectl top pods -n staxa-system from the live production server:

A note on these numbers: The platform is currently in public testing with low traffic. These represent baseline memory, what each component uses just by running, not under heavy load. Concurrent builds, active API traffic, and more tenants will push these higher. I'll update this article as the platform scales.

Component	What it does	RAM
Go API (`staxad`)	Single binary that handles all 60 API endpoints, orchestrates Kubernetes, manages jobs	13 Mi
Redis 7	Job queue (deployment pipeline) and pub/sub (SSE events to dashboard)	10 Mi
Notifier	Node.js service that bridges Redis pub/sub events to the dashboard via Socket.IO	20 Mi
Container registry	Stores built tenant images. Buildah pushes here, Kubernetes pulls from here	33 Mi
PostgreSQL 16	Platform database storing tenants, deployments, providers, and configs	44 Mi
Traefik	Ingress controller that routes all HTTP/HTTPS traffic to the right tenant	65 Mi
Next.js dashboard	Frontend for visual tenant management, connects to the Go API	85 Mi
Grafana	Dashboards for log volume, API requests, error rates, and active tenants	98 Mi
Loki	Log aggregation. Receives logs from Promtail, stores in Hetzner Object Storage	154 Mi
Strapi CMS	Headless CMS serving API documentation and support center content	200 Mi

Promtail (the log shipper) runs in a separate observability namespace and adds another ~30-50 Mi.

Total platform footprint: ~750 Mi. The entire deployment platform, including the API, database, queue, dashboard, CMS, build registry, ingress, observability stack, and real-time event streaming, fits in under 800 MB.

The Go API is the star here. A single compiled binary handles authentication, tenant provisioning, build orchestration, deployment management, SSE streaming, and admin tools. At baseline, it uses 13 megabytes of RAM, less than the container registry and less than PostgreSQL. It's a fraction of any single JavaScript service in the stack. Even under load, Go APIs of this size typically stay under 50-80 MB. Compare that to a Node.js or Java equivalent that would start at 100-200 MB idle. The compiled language advantage isn't theoretical at this scale. It's the reason the platform fits on this server at all.

What's Left for Tenants: ~5 GB

At current baseline usage, after the system and platform services (~1.5 GB + ~750 MB), roughly 5 GB is available for actual tenant workloads. Under heavier platform load this shrinks. Realistically, plan for 4-5 GB of tenant headroom. Each tenant gets a Kubernetes namespace with:

An app container (the user's deployed application)
A database container (PostgreSQL 16 or MySQL 8, their choice)
Resource quotas enforced at the namespace level

The resource sizes I offer:

Size	CPU Request	RAM Request	CPU Burst	RAM Burst	Typical use
Small	25m	64 MB	500m	512 MB	Static sites, light frontends
Medium	50m	128 MB	1 CPU	1 GB	APIs with database queries
Large	100m	256 MB	2 CPU	2 GB	Heavy backends, workers

"Requests" are guaranteed minimums. "Burst" is what they can use if other tenants aren't competing for resources. On a lightly loaded server, a "small" tenant can burst up to 500m CPU and 512 MB. That's plenty for a real web application.

A small tenant (app + Postgres) uses roughly 200-300 MB of guaranteed memory. A medium tenant uses 400-500 MB. That gives me:

~15-20 small tenants for static sites, SPAs, and light APIs
~10-12 medium tenants for full-stack apps with databases
~5-8 large tenants for heavier workloads

In practice, a mix of sizes. Call it 10-25 tenants depending on the workload profile. For a platform in early access, that's more than enough.

What Costs $0

Several things that would cost money on a managed cloud setup are effectively free here:

DNS: Cloudflare free tier. Wildcard DNS (*.tenants.staxa.dev) pointing to the server IP. Custom domains work via CNAME records that tenants add on their end.

SSL certificates: Let's Encrypt via cert-manager. Every tenant gets automatic SSL on both their subdomain and custom domain. cert-manager watches for new IngressRoutes and provisions certificates without any manual intervention. For wildcard certificates, it uses Cloudflare's DNS challenge.

Load balancing: Traefik handles all HTTP/HTTPS routing. On a single server there's no external load balancer to pay for. Traefik discovers new tenants automatically through Kubernetes CRDs. When the Go API creates an IngressRoute in a tenant's namespace, Traefik picks it up and starts routing traffic within seconds.

Container registry: Self-hosted inside the cluster. Buildah pushes built images here, Kubernetes pulls from here. No Docker Hub, no ECR, no per-image charges.

Monitoring and logging: Grafana + Loki + Promtail. Promtail ships pod logs to Loki (single-binary mode), which stores them in Hetzner Object Storage via S3. Grafana at logs.staxa.dev gives me dashboards for log volume by service, API request counts, error rates, and active tenant namespaces, with live log tailing and filtering by app or namespace. 30-day retention. The whole stack runs inside the cluster, and log storage costs are negligible on Hetzner's object storage pricing.

The One Thing That Would Blow Up the Budget

Managed databases.

A single AWS RDS db.t4g.micro (2 vCPU, 1GB) runs ~$12/month. If I gave every tenant their own RDS instance, 10 tenants would cost $120/month in databases alone, more than 10x the entire server.

Instead, each tenant's database runs in a container inside their Kubernetes namespace. PostgreSQL 16 or MySQL 8, backed by a PersistentVolume on the server's NVMe SSD. Is this as resilient as RDS with automated backups and failover? No. But for a platform in early access, it's the right tradeoff. I can always migrate individual tenants to managed databases later as a premium tier. The important thing is that the current architecture doesn't force a $12/tenant/month floor cost from day one.

When $11 Isn't Enough

The single-server setup has real limitations:

Single point of failure. If the server goes down, everything goes down, platform and tenants. There's no redundancy. For the current stage (early access, demo environments), this is acceptable. For paying customers expecting uptime SLAs, it's not.

Build pipeline contention. Container builds (Buildah) are CPU-intensive. When multiple tenants deploy simultaneously, builds queue up. On 4 vCPUs, I can realistically run 1-2 concurrent builds before performance degrades.

Disk pressure. Container images accumulate. Each tenant's built image sits in the registry, plus layers are cached for rebuild speed. At 80 GB total disk with the OS, Kubernetes, and databases taking their share, I'd start sweating around 20-30 tenants depending on image sizes. Registry garbage collection helps, but disk is the most likely bottleneck.

The Scaling Path

K3s makes horizontal scaling surprisingly simple. Adding a second server is one command:

# On the new server:
curl -sfL https://get.k3s.io | K3S_URL=https://<existing-server>:6443 \
  K3S_TOKEN=<node-token> sh -

That joins it as a worker node. Kubernetes starts scheduling tenant pods across both servers automatically.

Setup	Monthly Cost	Tenant Capacity
1x CAX21 (current)	~€10.99	10-25 tenants
CAX21 + CAX31 worker	~€33	30-60 tenants
CAX21 + AX42 dedicated	~€200	80-150 tenants
3-node CAX21 cluster	~€33	40-60 tenants with HA

The jump from €10.99 to €33 roughly triples capacity. Going up to ~€200 gets you 10-15x more headroom with a dedicated bare metal worker. Compare that to scaling on AWS where adding a worker node costs $50-100/month.

The Full Monthly Bill

Line Item	Monthly Cost
Hetzner CAX21 server	€10.99 (~$11)
Cloudflare DNS	$0 (free tier)
SSL certificates	$0 (Let's Encrypt)
Container registry	$0 (self-hosted)
Load balancer	$0 (Traefik, self-hosted)
Managed database	$0 (PostgreSQL in containers)
Email (Resend)	$0 (free tier, up to 3k/month)
Domain (staxa.dev)	~$1.50/month amortized
Total	~$13/month

For comparison, a minimal equivalent stack on AWS: EKS control plane ($73) + t4g.medium worker ($25) + RDS micro ($12) + ALB ($16) + Route53 ($0.50) = ~$127/month. And that's before bandwidth, storage IOPS, or a second availability zone.

What I'd Do Differently

I'd start with an even smaller server. The CAX11 (2 vCPU, 4GB, ~€6.49/month) can technically run K3s + the platform services, but it's tight. I went with the CAX21 for breathing room during development. If I were optimizing purely for cost from day one, I'd start on the CAX11 and upgrade when the first 5 tenants were running.

I'd separate control plane and tenant workloads earlier. Right now everything shares one server. In hindsight, starting with a cheap control plane node (CAX11, €6.49) and a separate tenant worker (CAX21, €10.99) would give better isolation for ~€18 total. If a runaway tenant build pegs the CPU, it wouldn't affect the API and dashboard.

I wouldn't change the "no managed services" decision. Running databases in containers sounds heretical to some people, and I get it. RDS gives you automated backups, failover, and point-in-time recovery for free. But at the early stage, that $12/tenant/month cost floor would have fundamentally changed the economics. I can add managed database support as a premium tier later. The architecture supports it. The tenant database configuration is just a field in the API.

What's the cheapest infrastructure you've run something real on? I'm curious whether anyone else is pushing single-server setups further than "it's just a side project."

I'm Jonathan. I build infrastructure tools with Go and Kubernetes from Jamaica. Currently building Staxa, a multi-tenant deployment platform. Follow me for more Go and DevOps content.

I Chose Chi Over Gin for 60 API Endpoints. Here's Why I Don't Regret It.

Jonathan Pitter — Thu, 07 May 2026 14:07:35 +0000

Every Go web framework comparison post says the same thing: "Gin is the most popular, Fiber is the fastest, Echo is a good balance." Chi barely gets a paragraph.

I'm building a multi-tenant deployment platform in Go. The API has 60 endpoints across 12 route groups: account management, tenant CRUD, deployments, services, environment variables, custom domains, network rules, GitHub integration, registry credentials, real-time SSE events, admin tools, and health checks.

I chose Chi. Months later, I still think it was the right call. Here's why.

The Decision Point

When I started, the choice came down to Gin and Chi. I'd used Gin before and liked it fine. But three things pushed me toward Chi:

Chi handlers are just http.HandlerFunc. That's it. No custom context type, no framework-specific signature. Every handler I write looks like this:

func (s *Server) handleCreateTenant(w http.ResponseWriter, r *http.Request) {
    provider := middleware.GetProvider(r.Context())
    // ... business logic
}

Standard http.ResponseWriter. Standard *http.Request. If I ever needed to rip out Chi and drop in something else or just use the standard library's http.ServeMux, my handlers don't change. With Gin, every handler takes *gin.Context, which means every handler is married to Gin.

Chi's middleware is just func(http.Handler) http.Handler. The standard net/http middleware signature. I can use any middleware from the Go ecosystem without an adapter. Rate limiters, CORS handlers, logging middleware, if it works with net/http, it works with Chi. No gin.HandlerFunc wrappers needed.

Chi nests routes with r.Route(), not method chains. This matters when you have 60 endpoints.

How the Router Actually Looks

Here's a simplified version of my real router. The full thing is in internal/api/router.go:

func NewRouter(deps Deps) http.Handler {
    r := chi.NewRouter()

    // Global middleware — runs on every request
    r.Use(middleware.RequestID)
    r.Use(middleware.Logger(deps.Logger))
    r.Use(middleware.Recovery(deps.Logger))
    r.Use(cors.Handler(cors.Options{
        AllowedOrigins: []string{"https://*", "http://*"},
        AllowedMethods: []string{"GET", "POST", "PUT", "PATCH", "DELETE", "OPTIONS"},
    }))

    // Health — no auth, no rate limiting
    r.Get("/healthz", handleHealthz)
    r.Get("/readyz", handleReadyz(hc))

    // Public — rate limited by IP, no auth
    r.With(rl.IPMiddleware(10)).Post("/api/v1/waitlist", srv.handleJoinWaitlist)

    // Internal — service-to-service, shared secret
    r.Route("/api/v1/internal", func(r chi.Router) {
        r.Use(srv.internalAuth(deps.InternalAPIKey))
        r.Get("/tenants/{id}", srv.handleInternalGetTenant)
    })

    // Provider API — authenticated, rate limited
    r.Route("/api/v1", func(r chi.Router) {
        r.Use(auth.Middleware)
        r.Use(rl.Middleware)

        r.Get("/account", srv.handleGetAccount)
        r.Patch("/account", srv.handleUpdateAccount)

        r.Get("/api-keys", srv.handleListAPIKeys)
        r.Post("/api-keys", srv.handleCreateAPIKey)
        r.Delete("/api-keys/{id}", srv.handleDeleteAPIKey)

        r.Get("/tenants", srv.handleListTenants)
        r.Post("/tenants", srv.handleCreateTenant)
        r.Get("/tenants/{id}", srv.handleGetTenant)
        // ... 40+ more endpoints

        r.Route("/admin", func(r chi.Router) {
            r.Use(srv.adminGuard)
            r.Get("/metrics", srv.handleAdminMetrics)
            r.Get("/config", srv.handleListAdminConfig)
            r.Put("/config/{key}", srv.handleSetAdminConfig)
        })
    })

    return r
}

Three things I want to highlight about this structure.

1. Middleware scoping is visual

Look at the nesting. You can see the auth boundaries:

/healthz: no middleware beyond the global stack
/api/v1/waitlist: IP rate limiting via r.With(), no auth
/api/v1/internal/*: internal API key auth
/api/v1/*: Clerk JWT or API key auth + per-key rate limiting
/api/v1/admin/*: all of the above, plus admin role check With Gin, you'd create route groups and attach middleware to each group. The structure is similar, but Chi's r.Route() nesting makes the middleware inheritance obvious just by reading the indentation. When a new developer (or new team member) opens this file.

2. `r.With()` for one-off middleware

The waitlist endpoint is public but rate-limited. Instead of creating a whole route group for one endpoint, Chi lets you do:

r.With(rl.IPMiddleware(10)).Post("/api/v1/waitlist", srv.handleJoinWaitlist)

r.With() applies middleware to just that route. No new group, no nesting ceremony. This is surprisingly useful when you have a few endpoints that need special treatment.

3. The `Deps` pattern

This isn't Chi-specific, but Chi makes it natural. Because handlers are methods on a Server struct, and the router is built by a function that takes a Deps struct, wiring dependencies is explicit:

type Deps struct {
    Store         store.Store
    Redis         *redis.Client
    Logger        *slog.Logger
    JWKSURL       string
    EventBus      *events.EventBus
    Queue         *queue.Queue
    Orchestrator  orchestrator.Orchestrator
    // ... 
}

No globals. No init() magic. Every dependency is injected through Deps, and the Server struct holds what handlers need. This makes testing straightforward; you mock the store.Store interface and pass it in.

The Dual Auth Middleware

Here's where Chi's net/http compatibility paid off the most. My API serves two types of clients:

The frontend dashboard: a Next.js frontend that sends Clerk JWTs
Programmatic consumers: B2B SaaS providers using API keys Both hit the same endpoints. The middleware auto-detects which auth mode is in play:

func (a *Auth) Middleware(next http.Handler) http.Handler {
    return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        token := extractBearerToken(r)

        var provider *store.Provider
        var err error

        if strings.HasPrefix(token, "sk_") {
            provider, err = a.authenticateAPIKey(r.Context(), token)
        } else {
            provider, err = a.authenticateJWT(r.Context(), token)
        }

        if provider == nil || err != nil {
            api.ErrUnauthorized(w, r, "invalid or expired credentials")
            return
        }

        ctx := context.WithValue(r.Context(), providerContextKey{}, provider)
        next.ServeHTTP(w, r.WithContext(ctx))
    })
}

This is a standard func(http.Handler) http.Handler. I didn't need a Chi-specific middleware type, a Gin adapter, or anything framework-aware. The JWKS caching, bcrypt comparison with prefix lookup, and auth result caching all live in the Auth struct; the framework never enters the picture.

Any handler downstream calls middleware.GetProvider(r.Context()) and gets the authenticated provider, regardless of whether they authenticated with a JWT or an API key.

What I Miss From Gin

I'll be honest about the tradeoffs.

Binding and validation. Gin has c.ShouldBindJSON(&body) which validates struct tags automatically. With Chi, I decode JSON manually:

var body struct {
    Name     string `json:"name"`
    DbEngine string `json:"db_engine"`
}
if err := json.NewDecoder(r.Body).Decode(&body); err != nil {
    pkgapi.ErrBadRequest(w, r, "invalid request body")
    return
}
if body.Name == "" {
    pkgapi.ErrBadRequest(w, r, "name is required")
    return
}

More boilerplate? Yes. But I know exactly what's happening. There's no magic struct tag that silently returns a 400 with a format I don't control. My error responses are consistent because I control them.

Less handholding for beginners. Gin has more examples, more Stack Overflow answers, more tutorials. Chi's documentation is good but minimal. If you're building your first Go API, Gin is a smoother onramp.

No built-in response helpers. Gin has c.JSON(200, data). With Chi, I wrote a small pkg/api package:

func JSON(w http.ResponseWriter, r *http.Request, status int, data interface{}) {
    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(status)
    json.NewEncoder(w).Encode(envelope{Data: data, Meta: meta(r)})
}

Ten lines of code I had to write once. Not a big deal, but it's not free either.

When I'd Choose Gin Instead

If I were building a quick CRUD API with 10 endpoints, no complex middleware layering, and I wanted to ship fast, then it's Gin. Its batteries-included approach is a real productivity boost for smaller projects.

If I were onboarding a team of junior Go developers, then of course, Gin. More resources, more patterns to copy, less "figure it out yourself."

When Chi Wins

If your API has multiple authentication modes, if your middleware stack has real layering (public vs authenticated vs admin vs internal), if you want your handlers to be portable http.HandlerFunc functions, if you care about net/http ecosystem compatibility, then it's Chi.

For me, the deciding factor was simple: Chi doesn't own my code. My handlers, my middleware, my response format, they're all standard Go. Chi is just the router. If Chi disappeared tomorrow, I'd swap in http.ServeMux from Go 1.22+ and change maybe 20 lines.

That's the kind of dependency I'm comfortable with.

What's your go-to Go router? Have you made a framework switch mid-project that you didn't regret?

I'm Jonathan, I build infrastructure tools with Go and Kubernetes. Currently building Staxa, a multi-tenant deployment platform. Follow me for more Go and DevOps content.

I Detect 26 Frameworks Without AI. Here's How Deterministic File-Based Detection Works.

Jonathan Pitter — Mon, 27 Apr 2026 15:00:00 +0000

When I built Staxa, a platform that deploys isolated customer environments from source code, I needed to answer a deceptively hard question.

"The user pushed a GitHub repo with no Dockerfile. What framework is this, and how do I containerize it?"

The "easy" answer is to "call an LLM." Send the file listing, maybe the first 50 lines of the main config file, and ask it to identify the framework and generate a Dockerfile.

I tried this approach early on and rejected it. Here's why, and what I built instead.

Why Not an LLM?

Three reasons, all deal-breakers for a deployment pipeline:

Latency. Staxa promises a running tenant environment in ~60 seconds. That's 7 pipeline stages: provision, detect, generate, build, deploy, route, health check. An LLM API call adds 2-5 seconds of latency to the detection stage alone. On a bad day, you're waiting for rate limits or retries. For something that should take milliseconds, that's unacceptable.

Cost at scale. Every deployment triggers a detection. At scale, thousands of API calls per month for a task that's fundamentally pattern matching. The cost scales linearly with deployments for no good reason.

External dependency in a critical path. If the LLM provider has an outage, your deployment pipeline stops. For a platform that automates infrastructure provisioning, adding a third-party API call to the critical path is a liability. The platform is designed to minimize external dependencies, the fewer things that can go down, the more reliable the pipeline.

Deterministic file-based detection solves all three: sub-millisecond execution, zero marginal cost, zero external dependencies.

The Detection System: Three Phases

The FrameworkDetector receives a cloned repo directory and returns a DetectResult:

type DetectResult struct {
    Framework  string            // "nextjs", "django", "go", etc.
    Confidence string            // "high" or "medium"
    DetectedBy string            // "found next.config.js" or "found \"next\" in package.json"
    Metadata   map[string]string // Framework-specific extras
}

Detection runs in three phases, each more expensive than the last. If a phase finds a match, the remaining phases are skipped.

Phase 1: Root File Markers (High Confidence)

The cheapest check. Read the root directory listing (one syscall) and look for framework-specific marker files.

// Phase 1: root file markers (high confidence).
for _, fw := range priorityOrder {
    tmpl, ok := templates[fw]
    if !ok || tmpl.Alias != "" {
        continue
    }
    for _, marker := range tmpl.DetectFiles {
        if rootFiles[marker] {
            result = &DetectResult{
                Framework:  fw,
                Confidence: "high",
                DetectedBy: fmt.Sprintf("found %s", marker),
                Metadata:   make(map[string]string),
            }
            break
        }
    }
    if result != nil {
        break
    }
}

Each framework defines its own marker files in the database. For example:

Framework	Marker Files
Next.js	`next.config.js`, `next.config.mjs`, `next.config.ts`
Nuxt	`nuxt.config.js`, `nuxt.config.ts`
Rails	`bin/rails`
Go	`go.mod`
Django	`manage.py`
Spring Boot	`pom.xml`, `build.gradle`, `build.gradle.kts`

If the repo root contains next.config.js, we know it's Next.js with high confidence. No file contents need to be read.

Phase 2: Dependency Markers (Medium Confidence)

If no root file marker matches, we open dependency files and search for framework-specific packages.

// Phase 2: dependency markers (medium confidence).
for _, fw := range priorityOrder {
    tmpl, ok := templates[fw]
    if !ok || tmpl.Alias != "" {
        continue
    }
    for depFile, packages := range tmpl.DetectDeps {
        if !rootFiles[depFile] {
            continue
        }
        content, err := readFileContent(filepath.Join(repoDir, depFile))
        if err != nil {
            continue
        }
        for _, pkg := range packages {
            if containsDependency(depFile, content, pkg) {
                result = &DetectResult{
                    Framework:  fw,
                    Confidence: "medium",
                    DetectedBy: fmt.Sprintf("found %q in %s", pkg, depFile),
                    Metadata:   make(map[string]string),
                }
                break
            }
        }
    }
}

The dependency matching is format-aware. containsDependency knows that package.json is JSON (check for a key), Cargo.toml is TOML (check for a dependency line), and Gemfile is plain text (case-insensitive search). It understands the structure well enough to avoid false positives.

Examples:

Framework	Dependency File	Package to Find
Fastify	`package.json`	`fastify`
Express	`package.json`	`express`
Actix	`Cargo.toml`	`actix-web`
Phoenix	`mix.exs`	`phoenix`
Rails	`Gemfile`	`rails`

This phase is "medium confidence" because a dependency file tells you what libraries are installed, but not necessarily which one is the primary framework. A repo with both express and next in package.json is Next.js, not Express, which is where priority ordering comes in.

Phase 3: Deep Scan (.NET Projects)

Some frameworks don't follow the "config file in root" convention. .NET projects bury their .csproj or .fsproj files in subdirectories, for example, src/MyApp/MyApp.csproj. Phase 3 does a shallow directory walk to find them:

// Phase 3: shallow scan for .NET projects in subdirectories.
if result == nil {
    if projectFile, err := findDotnetProject(repoDir); err == nil && projectFile != "" {
        result = &DetectResult{
            Framework:  "dotnet",
            Confidence: "high",
            DetectedBy: fmt.Sprintf("found %s", projectFile),
            Metadata:   map[string]string{"project_file": projectFile},
        }
    }
}

This is the most expensive phase (directory traversal), so it only runs when phases 1 and 2 found nothing.

The Priority System: Resolving Ambiguity

What happens when a repo has both go.mod and package.json? Or both rails and sinatra in the same Gemfile?

The detector iterates frameworks in a fixed priority order, most specific first:

var priorityOrder = []string{
    "nextjs", "nuxt", "svelte", "remix", "nestjs",
    "phoenix",
    "rails", "sinatra",
    "django", "fastapi", "flask",
    "laravel",
    "actix", "axum",
    "spring-boot",
    "dotnet",
    "go",
    "fastify", "express",
}

The ordering is deliberate:

Frontend meta-frameworks first. Next.js, Nuxt, SvelteKit, and Remix are checked before their underlying runtimes. A Next.js project has both next.config.js and package.json; if Express were checked first, it would incorrectly match on package.json.

Full-stack frameworks before micro-frameworks. Rails before Sinatra. Django before Flask.
Phoenix before generic Elixir. A repo with both rails and sinatra in the Gemfile gets
detected as Rails because Rails is checked first in Phase 2 priority order.

Go and generic Node last. go.mod and package.json are present in many polyglot repos. They're the catch-all; if nothing more specific matches, these are the right fallbacks.

The priority list is evaluated in the same order for both Phase 1 (file markers) and Phase 2 (dependency markers). The first framework to match in priority order wins.

Metadata Enrichment: Framework-Specific Intelligence

After detection, the system enriches the result with framework-specific metadata that the Dockerfile generator needs:

func (d *FrameworkDetector) enrichMetadata(result *DetectResult, repoDir string) {
    switch result.Framework {
    case "nextjs":
        if !inspectNextStandalone(repoDir) {
            result.Metadata["template_override"] = "nextjs_standard"
            result.Metadata["suggest_standalone"] = "true"
        }
    case "actix", "axum":
        if name := parseCargoBinaryName(repoDir); name != "" {
            result.Metadata["binary_name"] = name
        }
    case "dotnet":
        if pf, err := findDotnetProject(repoDir); err == nil && pf != "" {
            result.Metadata["project_file"] = pf
        }
    case "phoenix":
        if name := parseElixirAppName(repoDir); name != "" {
            result.Metadata["binary_name"] = name
        }
    }
}

Next.js: Standalone Detection

Next.js has two deployment modes. The output: 'standalone' config produces a minimal, self-contained build. Without it, you get a full node_modules directory in the image.

The detector inspects the Next config file, strips comment lines to avoid false positives from commented-out config, normalizes whitespace, and checks for the standalone output setting:

func inspectNextStandalone(repoDir string) bool {
    for _, name := range []string{"next.config.js", "next.config.mjs", "next.config.ts"} {
        content, err := readFileContent(filepath.Join(repoDir, name))
        if err != nil {
            continue
        }
        // Strip // line comments to avoid false positives
        var uncommented strings.Builder
        for _, line := range strings.Split(content, "\n") {
            trimmed := strings.TrimSpace(line)
            if strings.HasPrefix(trimmed, "//") {
                continue
            }
            uncommented.WriteString(line)
        }
        normalized := strings.ReplaceAll(uncommented.String(), " ", "")
        if strings.Contains(normalized, "output:'standalone'") ||
            strings.Contains(normalized, `output:"standalone"`) {
            return true
        }
    }
    return false
}

If standalone isn't configured, the detector sets template_override: "nextjs_standard" so the Dockerfile generator uses a different template and emits a suggestion to the user:

Tip: Add output: 'standalone' to your next.config.js for smaller, optimized images

Rust: Binary Name from Cargo.toml

Rust builds produce a binary named after the [package] name in Cargo.toml. The Dockerfile template needs this name to copy the correct binary from the builder stage. The detector parses it out:

func parseCargoBinaryName(repoDir string) string {
    content, err := readFileContent(filepath.Join(repoDir, "Cargo.toml"))
    if err != nil {
        return ""
    }
    inPackage := false
    for _, line := range strings.Split(content, "\n") {
        trimmed := strings.TrimSpace(line)
        if strings.HasPrefix(trimmed, "[") {
            inPackage = trimmed == "[package]"
            continue
        }
        if inPackage && strings.HasPrefix(trimmed, "name") {
            parts := strings.SplitN(trimmed, "=", 2)
            if len(parts) == 2 {
                return strings.Trim(strings.TrimSpace(parts[1]), "\"'")
            }
        }
    }
    return ""
}

Without this, the Dockerfile would use a generic binary name and the build would fail at the COPY --from=builder stage when the binary doesn't exist at the expected path.

Everything Lives in the Database

Here's the design decision that ties this all together: every detection rule, every Dockerfile template, and every default is stored in the platform_config database table as JSONB.

INSERT INTO platform_config (key, value, description) VALUES
('dockerfile_templates.nextjs', '{
  "description": "Next.js (Node.js)",
  "detect_files": ["next.config.js", "next.config.mjs", "next.config.ts"],
  "detect_deps": {"package.json": ["next"]},
  "default_port": 3000,
  "default_build_cmd": "npm run build",
  "default_start_cmd": "npm start",
  "template": "FROM node:{{.RuntimeVersion}}-alpine AS deps\n..."
}', 'Dockerfile template for Next.js applications');

Each framework entry contains:

detect_files: root file markers for Phase 1
detect_deps: dependency file → package name mappings for Phase 2
template: Go text/template string for Dockerfile generation
default_port, default_build_cmd, default_start_cmd: defaults that can be overridden per-deployment An operator can add support for a new framework or modify detection rules for an existing one with a single database insert. No code changes. No redeployment. The detector loads all templates from the database on every detection run.

Framework aliases handle sub-frameworks that share the same Dockerfile template. Gin, Fiber, Echo, and Chi all alias to the go template:

INSERT INTO platform_config (key, value, description) VALUES
('dockerfile_templates.gin',   '{"alias": "go"}', 'Alias → dockerfile_templates.go'),
('dockerfile_templates.fiber', '{"alias": "go"}', 'Alias → dockerfile_templates.go'),
('dockerfile_templates.echo',  '{"alias": "go"}', 'Alias → dockerfile_templates.go'),
('dockerfile_templates.chi',   '{"alias": "go"}', 'Alias → dockerfile_templates.go');

From Detection to Dockerfile

Once the framework is detected, the DockerfileGenerator takes over:

gen := builder.NewDockerfileGenerator(cfg.Store)
tmpl, err := gen.LoadTemplate(ctx, templateKey)
// ...
rendered, err := gen.Generate(tmpl, builder.DockerfileVars{
    RuntimeVersion: runtimeVersion,
    Port:           svc.Port,
    Framework:      result.Framework,
    AppName:        svc.Name,
    BinaryName:     result.Metadata["binary_name"],
    ProjectFile:    result.Metadata["project_file"],
})

The template variables flow through naturally:

{{.RuntimeVersion}}: pinned per-service, or falls back to a sensible default (Node 20, Python 3.12, Go 1.23, etc.)
{{.Port}}: from service config, or the template's default_port
{{.BinaryName}}: parsed from Cargo.toml or mix.exs by the metadata enrichment phase
{{.ProjectFile}}: the relative path to the .csproj for .NET projects The generated Dockerfile is then passed to the Buildah build pipeline (covered in the previous post) as if the user had written it themselves.

The Dashboard Integration: Real-Time Detection

Detection doesn't just happen at build time. When a user connects a GitHub repo in the dashboard wizard, the Go API runs the same detection logic against the GitHub API, before any code is cloned:

func (s *Server) handleAnalyzeRepo(w http.ResponseWriter, r *http.Request) {
    // 1. Get root tree via GitHub API
    tree, _ := s.githubApp.GetTree(ctx, installationID, owner, repo, branch)

    // 2. Determine which dependency files to fetch
    filesToFetch := ghpkg.DetectFilesToFetch(tree)

    // 3. Fetch them in parallel (bounded to 4 concurrent requests)
    fileContents := s.fetchFilesParallel(ctx, installationID,
        owner, repo, branch, filesToFetch)

    // 4. Detect framework from the fetched contents
    fwResult := ghpkg.DetectFrameworkFromDeps(tree, fileContents)
}

The parallel file fetching uses a bounded semaphore pattern, up to 4 concurrent GitHub API calls:

sem := make(chan struct{}, 4) // bounded concurrency

for _, name := range files {
    wg.Add(1)
    go func(filename string) {
        defer wg.Done()
        sem <- struct{}{}
        defer func() { <-sem }()

        content, err := s.githubApp.GetFileContent(ctx,
            installationID, owner, repo, filename, branch)
        if err != nil {
            return // Non-fatal — skip files we can't fetch
        }

        mu.Lock()
        results[filename] = []byte(content)
        mu.Unlock()
    }(name)
}

The result appears instantly in the wizard UI: the framework card gets an "Auto-detected" badge, the runtime version auto-populates, and the user can override any of these before deploying.

The Fallback Strategy

What if detection fails? The platform has a configurable fallback strategy, stored in ... you guessed it, the database:

INSERT INTO platform_config (key, value, description) VALUES
('dockerfile_gen.fallback_strategy', '"error"',
 'What to do when framework is unknown and no Dockerfile: "error" or "generic_node"');

In "error" mode (the default), the deploy fails with a clear message telling the user to add a Dockerfile. In "generic_node" mode, it falls back to a generic Node.js Express template, a reasonable bet for repos where package.json exists but no specific framework was identified.

The fallback result is clearly marked:

result = &builder.DetectResult{
    Framework:  "express",
    Confidence: "low",
    DetectedBy: "fallback:generic_node",
    Metadata:   make(map[string]string),
}

The "low" confidence and "fallback:generic_node" detection source are both surfaced to the user via SSE events, so there's no silent guessing.

The Numbers

Metric	Value
Frameworks detected	26
Dockerfile templates	19 (+ 7 aliases)
Detection phases	3 (file markers → deps → deep scan)
Confidence levels	3 (high, medium, low/fallback)
External API calls	0
Detection time	Sub-millisecond (after clone)
Template variables	8 (RuntimeVersion, Port, BuildCommand, StartCommand, Framework, AppName, BinaryName, ProjectFile)
Configuration	100% database-driven, zero hardcoded rules

Try It

Push a repo with no Dockerfile and watch the detection work in real-time through the dashboard wizard or SSE event stream.

👉 Join the waitlist at staxa.dev

This is the third post in the Building Staxa series. Previously: the multi-tenancy problem and the Buildah build pipeline. Next up: how Kubernetes NetworkPolicies, ResourceQuotas, and namespace isolation create real tenant boundaries, and why tenant_id columns aren't enough.

Kaniko Is Dead. Here's How I Build Tenant Images in Kubernetes Now.

Jonathan Pitter — Tue, 21 Apr 2026 15:00:00 +0000

Google archived Kaniko in June 2025. Docker-in-Docker requires privileged containers. I needed a third option for my project so I built a Buildah-based image pipeline that runs as Kubernetes Jobs, clones source, auto-generates Dockerfiles, and pushes to a private registry. Here's exactly how it works.

If you're building container images inside a Kubernetes cluster in 2026, your options have narrowed faster than you might think.

Docker-in-Docker requires mounting the Docker socket or running privileged containers, effectively giving the build container root access to the host. Every security guide tells you not to do this. OWASP's Docker Security Cheat Sheet says it plainly says it plainly: "Do not run containers with the --privileged flag."

Kaniko was the answer for years, Google's purpose-built tool for building images inside unprivileged containers. No Docker daemon needed. Then in June 2025, Google archived the repository. It's now read-only. GitLab has already removed their Kaniko documentation and recommends Buildah or Podman instead.

I was building Staxa, a multi-tenant PaaS that provisions isolated customer environments via API and I needed an image build pipeline that would run reliably inside k3s on a single ARM64 server. I started with Kaniko, saw the writing on the wall, and switched to Buildah.

This post is a walkthrough of how that pipeline actually works.

The Problem: Building Tenant Images From Source

When a Staxa user deploys an app, here's what happens at the build stage:

User pushes source code (GitHub repo) or provides a Dockerfile
The platform needs to turn that source into a container image
The image gets pushed to an internal registry
The orchestrator deploys it into the tenant's isolated namespace All of this happens inside a Kubernetes cluster. There's no external CI service, no GitHub Actions, no cloud build service. The build runs as a Kubernetes Job in the staxa-system namespace.

The constraints:

Must work on a single-node k3s cluster (ARM64)
Must handle private GitHub repos (token injection)
Must support auto-generated Dockerfiles (framework detection feeds into this)
Must clean up after itself (no build job accumulation)
Must surface build logs back to the user on failure

Why Buildah

After Kaniko's archival, three alternatives emerged as the main contenders:

BuildKit: Docker's modern build backend. Fast, parallel DAG-based execution and great caching. But running BuildKit as a standalone service inside Kubernetes adds another long-lived component to manage. For a platform that values minimal moving parts, that's overhead I didn't want.

Buildah: Red Hat's OCI image builder, now part of the Podman Container Tools suite accepted into the CNCF Sandbox in January 2025. Buildah builds standard Dockerfiles, pushes to any registry, and runs as a one-shot process, no daemon required. It fits naturally into the Kubernetes Job model: spin up a pod, build, push, exit.

werf / Kimia: Higher-level tools that wrap Buildah or BuildKit. More features, more opinions, more dependencies. Not what I wanted for a platform where every external dependency is a liability.

Buildah won because it maps cleanly onto a single pattern: one Kubernetes Job per build, no persistent infrastructure.

The Architecture

staxa-system namespace

staxad (Go API) receives deploy request

Creates a Kubernetes Job running Buildah

Job clones repo → builds image → pushes to registry

Go API polls Job status until success or failure

On failure: captures last 50 lines of build logs

On completion: cleans up the Job

The Go API orchestrates everything through client-go. No shell scripts wrapping kubectl. No Helm charts. The builder is a Go struct that implements a Builder interface:

type Builder interface {
    Build(ctx context.Context, job BuildJob) (imageTag string, err error)
}

This interface is the seam that let me swap Kaniko for Buildah without touching the rest of the pipeline. The deploy pipeline calls builder.Build() and gets back an image tag. It doesn't know or care which tool actually built the image.

The Build Job

Here's what the Buildah Kubernetes Job looks like when the Go API creates it:

bJob := &batchv1.Job{
    ObjectMeta: metav1.ObjectMeta{
        Name:      jobName, // "build-dep_abc123"
        Namespace: "staxa-system",
        Labels:    map[string]string{
            "staxa.dev/deployment-id": job.DeploymentID,
        },
    },
    Spec: batchv1.JobSpec{
        BackoffLimit: &zero, // No retries — fail fast
        Template: corev1.PodTemplateSpec{
            Spec: corev1.PodSpec{
                RestartPolicy: corev1.RestartPolicyNever,
                Containers: []corev1.Container{{
                    Name:    "buildah",
                    Image:   cfg.BuilderImage,
                    Command: []string{"sh", "-c", buildScript()},
                    Env:     env,
                    SecurityContext: &corev1.SecurityContext{
                        RunAsUser: &rootUser,
                        Capabilities: &corev1.Capabilities{
                            Add: []corev1.Capability{
                                "CHOWN",
                                "DAC_OVERRIDE",
                                "FOWNER",
                                "SETUID",
                                "SETGID",
                                "SETFCAP",
                                "SYS_CHROOT",
                            },
                        },
                    },
                }},
            },
        },
    },
}

So I made a few deliberate choices here:

BackoffLimit: 0: If a build fails, it fails. No automatic retries that could mask a real error. The user sees the failure immediately via SSE events and gets build logs.

RestartPolicy: Never: Same philosophy. The Job runs once. If something goes wrong, the user redeploys with a fix rather than the platform silently retrying with the same broken code.

Minimal capabilities: Instead of running the build container with full privileges, only seven specific Linux capabilities are granted: CHOWN, DAC_OVERRIDE, and FOWNER for file ownership during COPY --chown; SETUID and SETGID for USER directives; SETFCAP for file capabilities in base images; and SYS_CHROOT for Buildah's chroot isolation mode. This is the minimum set that lets Buildah build arbitrary Dockerfiles.

Labels: Every build Job is labeled with the deployment ID. This is how we find the associated pods later when we need to grab logs from a failed build.

The Build Script

The actual build runs as a shell script inside the Buildah container:

set -e

# Inject GitHub token for private repos if available
CLONE_URL="$REPO_URL"
if [ -n "${GITHUB_TOKEN:-}" ]; then
  CLONE_URL="$(echo "$REPO_URL" | \
    sed "s|https://|https://x-access-token:${GITHUB_TOKEN}@|")"
fi

# Shallow clone — only the branch we need, one commit deep
git clone --depth 1 --branch "$BRANCH" "$CLONE_URL" /workspace
cd /workspace

# Build with VFS storage driver + chroot isolation (no privileged mode needed)
buildah --storage-driver vfs build \
  --isolation chroot \
  --layers \
  $BUILD_ARGS \
  --tag "$IMAGE_TAG" \
  --file "$DOCKERFILE_PATH" \
  "${BUILD_CONTEXT:-.}"

# Push to internal registry (TLS disabled for in-cluster registry)
buildah --storage-driver vfs push \
  --tls-verify=false \
  "$IMAGE_TAG" \
  "docker://$IMAGE_TAG"

--storage-driver vfs: This is the key to running Buildah without overlayfs kernel support. VFS copies each layer fully rather than using filesystem overlays. It's slower than overlayfs, but it works everywhere including inside a container on k3s on an ARM64 Hetzner box. No special kernel modules required.

--isolation chroot: Instead of spinning up a full OCI runtime (runc) for each build step, Buildah uses a simple chroot. This keeps the capability requirements minimal, chroot isolation only needs SYS_CHROOT rather than the broad set of privileges a full OCI runtime would demand.

--layers: Enables layer caching. Subsequent builds reuse unchanged layers, which significantly speeds up redeploys where only application code changed.

--depth 1: Shallow clone. We don't need git history, so we save time and bandwidth by only pulling the latest commit on the target branch.

$BUILD_ARGS: Tenant environment variables are injected as --build-arg flags by the Go API. This is critical for frameworks like Next.js that inline NEXT_PUBLIC_* variables at build time they need to be available during the image build, not just at runtime.

Image Tagging Strategy

Every image is tagged with a deterministic, versioned path:

{registry}/tenants/{tenant_id}/{service_name}:{version}

For example: registry.staxa-system:5000/tenants/ten_abc/app:3

The Go code that generates this:

imageTag := fmt.Sprintf("%s/tenants/%s/%s:%d",
    job.RegistryURL, job.TenantID, name, job.Version)

This gives us:

Tenant isolation in the registry: each tenant's images live under their own path
Multi-service support: a tenant with a frontend and backend gets separate image paths
Version history: rollback is just redeploying a previous version number
No tag collisions: version is an incrementing integer from the database

GitHub Token Resolution

Private repos need authentication. Staxa handles this with a two-tier fallback:

func (b *BuildahBuilder) resolveGitHubToken(ctx context.Context, token string) []corev1.EnvVar {
    // Tier 1: Per-deployment token (from GitHub App installation)
    if token != "" {
        return []corev1.EnvVar{{Name: "GITHUB_TOKEN", Value: token}}
    }

    // Tier 2: Cluster-level secret (for environments without GitHub App)
    _, err := b.client.CoreV1().Secrets(buildahNamespace).
        Get(ctx, "github-credentials", metav1.GetOptions{})
    if err != nil {
        return nil // No token available — public repo only
    }

    return []corev1.EnvVar{{
        Name: "GITHUB_TOKEN",
        ValueFrom: &corev1.EnvVarSource{
            SecretKeyRef: &corev1.SecretKeySelector{
                LocalObjectReference: corev1.LocalObjectReference{
                    Name: "github-credentials",
                },
                Key: "token",
            },
        },
    }}
}

If a GitHub App is configured, each deployment gets a short-lived installation access token (Tier 1). If not, it falls back to a cluster-level secret. If neither exists, the build proceeds without auth which works fine for public repos, fails fast for private ones with a clear error.

Registry Credentials

Similarly, pushing to registries that require auth uses a Docker-compatible credentials mount:

func (b *BuildahBuilder) registryCredentials(ctx context.Context) ([]corev1.Volume, []corev1.VolumeMount) {
    _, err := b.client.CoreV1().Secrets(buildahNamespace).
        Get(ctx, "registry-credentials", metav1.GetOptions{})
    if err != nil {
        return nil, nil // No credentials, internal registry, no auth needed
    }

    volumes := []corev1.Volume{{
        Name: "registry-credentials",
        VolumeSource: corev1.VolumeSource{
            Secret: &corev1.SecretVolumeSource{
                SecretName: "registry-credentials",
            },
        },
    }}
    mounts := []corev1.VolumeMount{{
        Name:      "registry-credentials",
        MountPath: "/root/.docker/config.json",
        SubPath:   "config.json",
        ReadOnly:  true,
    }}
    return volumes, mounts
}

Buildah reads Docker-compatible auth from /root/.docker/config.json, so the same credentials format works for both tools which is another reason the Kaniko → Buildah migration was straightforward.

Polling, Failure Handling, and Cleanup

The Go API doesn't use Kubernetes watches for build status. It polls:

err = wait.PollUntilContextTimeout(ctx, 5*time.Second, 10*time.Minute, false,
    func(ctx context.Context) (bool, error) {
        j, err := b.client.BatchV1().Jobs(buildahNamespace).
            Get(ctx, jobName, metav1.GetOptions{})
        if err != nil {
            return false, err
        }
        if j.Status.Succeeded >= 1 {
            return true, nil
        }
        if j.Status.Failed >= 1 {
            buildErr = fmt.Errorf("buildah: build failed for %s", imageTag)
            if logs := b.tailBuildLogs(ctx, jobName); logs != "" {
                buildErr = fmt.Errorf("%w\nbuild logs:\n%s", buildErr, logs)
            }
            return false, buildErr
        }
        return false, nil
    },
)

Why polling over watches? The build is a synchronous step in an async pipeline. The worker goroutine is already dedicated to this deployment. Polling every 5 seconds for up to 10 minutes is simple, predictable, and doesn't require setting up watch infrastructure. The Kubernetes API handles it fine.

When a build fails, the builder grabs the last 50 lines of logs from the pod:

func (b *BuildahBuilder) tailBuildLogs(ctx context.Context, jobName string) string {
    pods, err := b.client.CoreV1().Pods(buildahNamespace).List(ctx,
        metav1.ListOptions{LabelSelector: "job-name=" + jobName})
    if err != nil || len(pods.Items) == 0 {
        return ""
    }
    tailLines := int64(50)
    req := b.client.CoreV1().Pods(buildahNamespace).
        GetLogs(pods.Items[0].Name, &corev1.PodLogOptions{TailLines: &tailLines})
    rc, err := req.Stream(ctx)
    if err != nil {
        return ""
    }
    defer rc.Close()
    var buf bytes.Buffer
    io.Copy(&buf, rc)
    return buf.String()
}

These logs get surfaced to the user through the SSE event stream. When a build fails, they see exactly why, not a generic "build failed" message.

After every build (success or failure), the Job is deleted:

func (b *BuildahBuilder) cleanupBuildJob(ctx context.Context, jobName string) {
    prop := metav1.DeletePropagationBackground
    b.client.BatchV1().Jobs(buildahNamespace).Delete(ctx, jobName,
        metav1.DeleteOptions{PropagationPolicy: &prop})
}

No TTL controllers, no cron cleanup. The build created the Job, the build cleans it up. Simple.

The Builder Interface Pattern

The reason swapping from Kaniko to Buildah was a one-file change: the Builder interface.

type Builder interface {
    Build(ctx context.Context, job BuildJob) (imageTag string, err error)
}

Both KanikoBuilder and BuildahBuilder implement this interface. The deploy pipeline doesn't know which one is active. Switching was a matter of changing which struct gets instantiated at startup no changes for the pipeline, the worker, or the API handlers.

There's also a NoopBuilder for testing that returns a deterministic image tag without building anything. The entire deployment pipeline can be tested end-to-end without ever running a real build.

What I'd Do Differently

VFS performance. The vfs storage driver works everywhere but copies every layer fully. For a platform optimized for initial deploy speed ("60 seconds to a running tenant"), build time is the largest chunk. I'm exploring whether fuse-overlayfs could work in this environment to speed up builds without requiring privileged overlayfs access.

Build caching across tenants. Currently each build starts fresh. Tenants deploying the same framework (say, Next.js 15) all download and install the same base dependencies independently. A shared cache layer could cut repeat build times significantly.

BuildKit as an alternative. BuildKit's parallel DAG execution would be faster for multi-stage Dockerfiles. If build performance becomes a bottleneck at scale, running a BuildKit daemon as a persistent service in the cluster rather than per-build Jobs would be worth the operational tradeoff.

The Numbers

Metric	Value
Build tool	Buildah (VFS storage driver)
Execution model	One Kubernetes Job per build
Build timeout	10 minutes
Poll interval	5 seconds
Log capture on failure	Last 50 lines
Cleanup	Immediate (background propagation)
Migration from Kaniko	1 file changed, 0 pipeline changes

Try It

Staxa handles the entire build pipeline, framework detection, Dockerfile generation, image building, and deployment all behind a single API call or dashboard wizard.

👉 Join the waitlist at staxa.dev

This is the second post in the Building Staxa series. The first post covered the multi-tenancy problem and why I built a platform that automates full tenant isolation. Next up: how the framework detection system auto-detects 19 frameworks without any AI using deterministic file-marker matching and dependency-file inspection.

There Are 5 Ways to Do Multi-Tenancy. I Automated the Hardest One.

Jonathan Pitter — Wed, 15 Apr 2026 12:00:00 +0000

If you've ever built a SaaS product that serves multiple customers, you've stared at this decision:

How do I keep Customer A's data away from Customer B?

The answer seems simple until you actually try to implement it.

Then you discover that every approach on the multi-tenancy spectrum is a different flavor of pain.

The Multi-Tenancy Spectrum (And Why Every Option Hurts)

Level 1: Shared Database, Shared Schema

The most common starting point. Everyone's data lives in the same tables, separated by a tenant_id column.

-- Every query needs this.
SELECT * FROM orders WHERE id = $1 AND tenant_id = $2;

-- Forget the tenant_id filter once and you have a data breach.
SELECT * FROM orders WHERE id = $1;

It's cheap and simple, until it isn't. A single missed WHERE clause leaks data across tenants. Junior devs joining the team won't always remember. Your ORM might not enforce it. And PostgreSQL itself isn't immune CVE-2024-10976, disclosed in late 2024, showed that row-level security policies could be bypassed when queries were reused across different roles, potentially letting one tenant's query return another tenant's rows.

Then there's the noisy neighbor problem. One tenant runs a massive data import and suddenly every other tenant's dashboard grinds to a halt. You can't set per-tenant resource limits because everyone shares the same compute and storage.

Level 2: Shared Database, Separate Schemas

More isolation as each tenant gets their own schema inside one database. tenant_a.orders, tenant_b.orders.
The problem shifts from simply remembering the where clause to making sure you can coordinate schema migrations across 'N' number of schemas with zero downtime. At 10 tenants, that's manageable. At 500, you've built a migration orchestration system. And tenants still share the underlying database resources, so noisy neighbors are still a thing.

Multiple sources I reviewed while building Staxa recommend avoiding this approach entirely. It combines the operational complexity of separate databases with the resource-sharing downsides of a shared one. You can check this out

Level 3: Database Per Tenant

Now we're getting to real isolation. Each tenant has their own database instance, Backup and restores are on a per-tenant level. A failure in one tenant's database doesn't cascade to others.

But you're now running migrations across 'N' separate databases. Schema version drift becomes a real risk. Your CI/CD pipeline needs to understand which tenants are on which version, infrastructure costs scale linearly and you still haven't isolated the application layer. Your API server is shared, your compute is shared and your networking is shared.

Level 4: Namespace or Container Isolation

Pushing a step further, each tenant runs in their own Kubernetes namespace or container group, with their own database. Now you've isolated compute, storage, and networking.

This is where things get architecturally correct but operationally brutal. You need to automate namespace creation, deploy applications into each one, configure ingress routing per tenant, provision and inject database credentials, set up SSL certificates, manage resource quotas, configure network policies, and wire up health checks, for every single tenant.

Most teams that attempt this build a custom control plane. That control plane becomes its own product to maintain.

Level 5: Fully Siloed, Dedicated Infrastructure Per Customer

The golden standard, each customer gets their own everything

Application
Database
Subdomain
SSL
Network isolation
No shared resources.
No chance of cross-tenant data leakage.
No noisy neighbor issues.

Enterprise customers in regulated industries demand this, Government contracts require it and even B2B SaaS customers increasingly ask for it as a premium tier.

The tradeoff? It's the hardest approach to operationalize by far. Without automation, provisioning a new customer environment typically takes two to six weeks, accounting for infrastructure setup, security configuration, networking, and testing. Supporting one or two dedicated deployments manually is feasible. However supporting ten or more without a purpose-built system requires significant dedicated engineering investment.

This is the problem I decided to solve.

Staxa: Full Tenant Isolation in 60 Seconds

Staxa is an API-first platform that automates Level 5 multi-tenancy. One API call, and ~60 seconds later your customer has:

Their own application container running their framework
A dedicated PostgreSQL or MySQL database
A unique subdomain with automatic SSL via Let's Encrypt
Full Kubernetes namespace isolation: network policies, resource quotas, RBAC
Environment variables encrypted at rest with AES-256-GCM

curl -X POST https://api.staxa.dev/api/v1/tenants \
  -H "Authorization: Bearer sk_live_xxx" \
  -d '{
    "name": "acme-corp",
    "source": {
      "type": "github",
      "repo_url": "https://github.com/acme/app"
    },
    "database": { "engine": "postgres" }
  }'

That's it. Hook this into a Stripe webhook and your SaaS provisions isolated customer environments automatically on payment. No Terraform configs, No DevOps tickets, No two-week onboarding process.
There's also a visual dashboard with a deployment wizard for teams that prefer point-and-click over API calls. Both interfaces hit the same backend, the dashboard just builds the same JSON payload through a multi-step UI flow.

How It Works Under the Hood

The entire platform is a single Go binary (staxad) running on a k3s Kubernetes cluster. When a deploy request comes in, it moves through a 7-stage pipeline:

Provision → Create the Kubernetes namespace, RBAC rules, ResourceQuotas, and NetworkPolicies for the tenant.
Detect → The framework detection system identifies which frameworks your project uses from file markers and dependency files. 26 frameworks supported, deterministic file-based detection with a two-tier confidence system. No external API calls, zero latency overhead, zero external dependencies.
Generate → This one came from a friend but if you have no Dockerfile in your project? Then auto-generate a Dockerfile using Go text/template with framework-specific configs pulled from the database. Ports, health check paths, build commands, start commands, all operator-tunable without code changes.
Build → Buildah runs as a Kubernetes job, builds the image, pushes to the internal registry. No Docker-in-Docker, no privileged containers.
Deploy → Kubernetes Deployment created with resource limits, health checks, and environment variables. Database provisioned in the same namespace if requested.
Route → Traefik IngressRoutes for the tenant's subdomain. cert-manager handles SSL automatically.
Health Check → Poll the health endpoint until the app responds, then mark as ready.

The caller gets real-time progress via Server-Sent Events throughout all seven stages. The dashboard shows a live progress bar; an API consumer streams the same events programmatically.

Hetzner CAX21 (~€8/month) — k3s single-node cluster

staxa-system namespace

staxad — Go API, single binary, 49 endpoints

PostgreSQL — platform database

Redis — job queue + SSE pub/sub

Next.js — dashboard + deployment wizard

tenant-acme namespace

app container (Next.js)

dedicated postgres

tenant-widgetco namespace

frontend service

backend service

dedicated postgres

Traefik — ingress controller, auto-SSL via cert-manager, *.staxa.dev routing

A tenant can have multiple services (frontend + backend) sharing a namespace, internal networking, and database credentials. Service URL injection (STAXA_SERVICE_{NAME}_URL) handles cross-service discovery automatically.

Why Go, Why Kubernetes, Why a $10 Server?

Go because infrastructure tooling lives in Go. Docker, Kubernetes, Terraform, Traefik, Caddy, all Go and I can't help trying new things :).

The client-go library for Kubernetes is first-class. Goroutines handle parallel tenant deployments naturally. And a single compiled binary means the entire control plane is one scp away from running on any server.

Kubernetes (k3s) because the abstractions map perfectly to the multi-tenancy problem.

Namespaces: tenant boundaries.
ResourceQuotas: noisy neighbor prevention.
NetworkPolicies: traffic isolation.
Deployments: rolling updates and rollbacks for free.

I'm not bolting isolation onto a system that wasn't designed for it, I'm using a system where isolation is a first-class primitive.

Hetzner server because the old version of this platform ran on AWS and cost $200+/month with zero tenants deployed, literally just from developing, I almost gave up entirely after that pain. It's the same architecture, databases in containers instead of managed services, commodity ARM compute. The entire platform and 10-15 demo tenants run comfortably on one box.

Design Decisions That Matter

No magic numbers: Every tunable value — resource limits, port ranges, framework templates, detection rules; lives in a platform_config database table. An operator can change any default setting without redeploying the whole platform. This was a hard rule from day one.

No external AI for framework detection: I evaluated using an LLM API to detect project frameworks and rejected it since that would be easier. However the latency, cost, and external dependency were all liabilities. Deterministic file-based detection with parallel file fetching worked out faster, cheaper (the cost would scale with projects deployed), and has no external dependencies.

49 API endpoints across 12 groups: Tenants, deployments, services, domains, environment variables, network rules, container registry credentials, templates, and more.

Dual auth: Clerk JWTs for dashboard users, API keys with prefix lookup for programmatic access.

Multi-service tenants: Staxa models tenants as isolation boundaries that can contain multiple services because real applications have frontends and backends that need to talk to each other.

Who This Is For

SaaS founders whose enterprise customers are asking for dedicated environments. Instead of months of infrastructure work, you integrate one API call into your onboarding flow, that's it.
Agencies who need isolated environments per client, each with their own domain and SSL without one client's traffic spike affecting another.
Platform engineers who need on-demand isolated environments for dev teams or staging, without giving everyone kubectl access.

The Numbers

Metric	Value
Infrastructure cost	~€8/month
Time to provision a fully isolated tenant	~60 seconds
Provider-facing API endpoints	49
Frameworks detected	26
Auto-generated Dockerfile templates	19
External SaaS dependencies	2 (Clerk for auth, Cloudflare for DNS)
V1 AWS cost (zero tenants)	$200+/month

Try It
Staxa is in early access and I'm working directly with early adopters.

👉 Play around and join the waitlist if interested at staxa.dev

DEV Community: Jonathan Pitter

The rollback endpoint took a deployment ID and did nothing with it

Project Overview

Bug Fix or Performance Improvement

Code

The handler, before and after

The worker

The schema bug underneath (PR #22)

My Improvements

Diagnosing it

How to get the target to the worker

Validating twice on purpose

The bug the fix exposed

What code review caught that I missed

The design problem underneath that

Verifying

What I would do differently

I Run a Full PaaS on One $11/Month ARM Server. Here's What the Resource Usage Actually Looks Like.

The Server

What's Running on 8 GB

System Components (~1.5 GB)

Platform Services (~750 MB)

What's Left for Tenants: ~5 GB

What Costs $0

The One Thing That Would Blow Up the Budget

When $11 Isn't Enough

The Scaling Path

The Full Monthly Bill

What I'd Do Differently

I Chose Chi Over Gin for 60 API Endpoints. Here's Why I Don't Regret It.

The Decision Point

How the Router Actually Looks

1. Middleware scoping is visual

2. r.With() for one-off middleware

3. The Deps pattern

The Dual Auth Middleware

What I Miss From Gin

When I'd Choose Gin Instead

When Chi Wins

I Detect 26 Frameworks Without AI. Here's How Deterministic File-Based Detection Works.

Why Not an LLM?

The Detection System: Three Phases

Phase 1: Root File Markers (High Confidence)

Phase 2: Dependency Markers (Medium Confidence)

Phase 3: Deep Scan (.NET Projects)

The Priority System: Resolving Ambiguity

Metadata Enrichment: Framework-Specific Intelligence

Next.js: Standalone Detection

Rust: Binary Name from Cargo.toml

Everything Lives in the Database

From Detection to Dockerfile

The Dashboard Integration: Real-Time Detection

The Fallback Strategy

The Numbers

Try It

Kaniko Is Dead. Here's How I Build Tenant Images in Kubernetes Now.

The Problem: Building Tenant Images From Source

Why Buildah

The Architecture

The Build Job

The Build Script

Image Tagging Strategy

GitHub Token Resolution

Registry Credentials

Polling, Failure Handling, and Cleanup

The Builder Interface Pattern

What I'd Do Differently

The Numbers

Try It

There Are 5 Ways to Do Multi-Tenancy. I Automated the Hardest One.

The Multi-Tenancy Spectrum (And Why Every Option Hurts)

Level 1: Shared Database, Shared Schema

Level 2: Shared Database, Separate Schemas

Level 3: Database Per Tenant

Level 4: Namespace or Container Isolation

Level 5: Fully Siloed, Dedicated Infrastructure Per Customer

Staxa: Full Tenant Isolation in 60 Seconds

How It Works Under the Hood

Why Go, Why Kubernetes, Why a $10 Server?

Design Decisions That Matter

2. `r.With()` for one-off middleware

3. The `Deps` pattern