DEV Community: Rk

Heroku Alternatives Worth Trying in 2026

Rk — Tue, 18 Aug 2026 14:43:41 +0000

Heroku defined a whole generation of application deployment. git push heroku main set the bar for what a good deploy loop feels like, buildpacks made runtimes disappear into the background, and add-ons turned integrating a database into a two-minute chore rather than a two-day one. Even now, teams building brand new applications quietly copy the operational model Heroku popularised.

What changed over the last few years is the price of that convenience and the surface area around it. Dyno costs climb quickly once traffic grows, the free tier that made Heroku famous for side projects is long gone, add-on pricing stacks on top of the dyno bill, and the ecosystem around Postgres, workers, and background jobs no longer feels as central as it once did. A generation of newer platforms has picked up the same developer experience and rebuilt it on top of containers, edge networks, and pricing models that match how modern applications actually consume resources.

This post walks through nine platforms that we see teams choose when they move away from Heroku, why each one exists, what they are best at, and the trade-offs to know before you start migrating. Some are fully managed, some are self-hosted, and one of them is ours.

What to look for when you leave Heroku

Before running through the list, it helps to name the axes that usually decide the choice, because most of the time the "right" answer changes depending on which of these matters most for the workload in question.

The runtime model is the first fork in the road, since some platforms run your code as always-on containers with steady memory footprints and others run it as short-lived function invocations that spin up on demand. Container platforms feel closer to Heroku dynos and suit long-running processes, background workers, and stateful protocols like WebSockets, while function platforms bill by request and shine for spiky, stateless traffic.

Pricing structure matters more than the sticker number. A flat monthly workspace fee plus per-second compute metering looks very different on a bill from a purely usage-based model, especially once you factor in bandwidth, storage, and add-ons like managed Postgres. Look at how the platform charges for the traffic patterns you actually have, not the ones the pricing page is optimised to make look cheap.

Regional coverage decides how close your services live to your users, and how close they live to any managed databases you already run on AWS, GCP, or Azure. If your Postgres is in us-east-1, running your app anywhere else adds latency you cannot make up for with faster code.

The edge tier in front of your app changes what you have to build yourself. Platforms fronted by a real CDN and WAF cover DDoS protection, TLS termination, and bot mitigation without you thinking about it, while platforms that only expose raw TCP leave those layers as your problem or as another vendor line on the bill.

Whose infrastructure your workload runs on is a question that used to have one answer (the vendor's) but increasingly has two. Bring-your-own-cloud and bring-your-own-cluster options let you use a managed control plane while keeping your workloads on infrastructure you already own or are contractually committed to.

With those in mind, here are the nine.

1. Suga

Suga is the platform we build, so consider this section the founder's pitch, but the honest one: we started Suga because we wanted the Heroku-style loop back with the parts that scale properly. You design your application on a visual canvas, connect services to databases and networking, and deploy every change with a commit message and an audit trail. Deployments finish in seconds, environments can be forked whole for testing, and rolling back to any previous state is a single click.

Under the hood, workloads run as containers on managed infrastructure fronted by Cloudflare's global edge, so DDoS protection, WAF, and TLS termination apply to every service by default, with no separate integration to configure. Pricing is per-seat with hosting credits bundled into every Pro seat, which suits small teams running real production workloads before they need enterprise governance features.

The interesting move on the top tier is bring-your-own-cloud. On Enterprise, the same control plane can also drive infrastructure you already own on any cloud or on-premises, so you get a single deployment surface across the managed environment and workloads that need to run inside your existing cloud commitments.

Best for: teams that want the discipline of Infrastructure as Code with fast deploys, that value environment forking for testing, and that want the edge tier included by default. Enterprises with existing cloud footprints who want a modern deployment experience on the cluster of record they already have.

Consider elsewhere if: you need very large per-instance memory ceilings today, you require published SOC 2 and ISO 27001 certifications (Suga is working toward these but has not published them yet), or you need a mature managed Postgres with point-in-time recovery as a first-party service.

Pricing: Free tier for evaluation, Pro is per-seat with $20 of hosting credits included per seat, Enterprise is custom with BYOC support.

2. Render

Render is the platform that most closely mirrors the Heroku operating model on modern infrastructure. You push a repo, Render detects the runtime or reads a Dockerfile, and the service comes up on managed infrastructure with automatic TLS, health checks, and zero-downtime deploys.

Where Render differentiates is on the range of instance sizes it offers, scaling up to 32 GB of RAM and 8 CPU per service on the top tier, and on its managed data services, which include a full Postgres product with point-in-time recovery, read replicas, and tiered compute. For teams whose main frustration with Heroku is dyno price-to-performance and add-on lock-in, Render often reads as the closest one-to-one replacement without the operational retraining.

Best for: teams who want the Heroku experience (push code, get a URL) with wider instance sizes, first-party managed Postgres, and published compliance certifications.

Consider elsewhere if: you need to run workloads on your own infrastructure, or your traffic pattern is spiky enough that flat workspace fees plus per-second compute metering does not fit as cleanly as pure usage-based billing.

Pricing: Free hobby tier with sleep, Pro from $25/month workspace fee plus compute metered by the second, Scale plan for larger teams.

3. Railway

Railway leans into the visual side of application composition. You drop services on a canvas, wire databases, workers, and crons together, and Railway takes care of stitching networking and secrets between them. Every project can spin up databases as containerised services, and per-service resources are set by picking a compute plan rather than an instance size.

For teams that liked Heroku's add-on marketplace but want the flexibility to run everything as containers, Railway hits a familiar note, and the developer loop of connecting to running containers over SSH is one of the more distinctive features on this list.

Best for: small teams and startups that want a single project canvas showing everything running in the application, plus straightforward per-minute billing on CPU and RAM.

Consider elsewhere if: you need L7 DDoS protection and a WAF at the edge without adding a separate provider, or you need to run on your own infrastructure.

Pricing: Hobby plan from $5/month with compute metered per minute, team plans add collaboration and higher limits.

4. Fly.io

Fly.io approaches the deployment problem from a very different angle. Fly runs your app on its own micro-VMs distributed across dozens of regions worldwide, so "run this container close to the user" is baked into how you design the deployment from the start, and the concepts you'd normally reach for on other platforms (a CDN in front of a single-region origin) are replaced by the workload actually living where the users are.

That model shines for latency-sensitive applications, multi-region databases, and anything with a genuinely global user base. It also introduces its own set of concerns around data locality, region failover, and how you design your topology, so Fly rewards teams who are willing to think about placement explicitly.

Best for: applications where round-trip latency to the user matters, multi-region topologies, and workloads that benefit from geographical distribution as a first-class primitive.

Consider elsewhere if: you have a single-region application backed by a single-region database, since much of Fly's distinctive value comes from the multi-region model.

Pricing: Usage-based across CPU, RAM, and bandwidth, with a small always-on hobby footprint free.

5. Vercel

Vercel is not a container platform, but it belongs on any honest list of Heroku alternatives because Heroku's Node.js and frontend workloads are exactly the traffic Vercel is optimised for. If your application is a Next.js frontend calling out to an API, Vercel gives you a build-and-deploy experience tuned for that pattern, with edge-cached static assets, serverless functions for the API layer, and preview environments on every pull request.

Where Vercel diverges from Heroku is in the runtime model. Instead of always-on dynos, Vercel runs your code as serverless functions that scale to zero between requests, which suits spiky HTTP workloads and hurts long-running background jobs, WebSocket connections, and anything that assumes a warm process.

Best for: Next.js frontends, marketing sites, JAMstack applications, and teams whose "backend" is mostly a thin API layer in front of managed services.

Consider elsewhere if: your workload is a long-running process, holds open WebSocket connections, runs background workers, or does heavy in-process caching that scale-to-zero would evict.

Pricing: Hobby free with usage limits, Pro from $20/user/month plus usage-based function invocations and bandwidth.

6. Porter

Porter takes the Heroku developer experience and puts it on top of Kubernetes. You get a push-to-deploy flow, sensible defaults for TLS and autoscaling, and the ability to graduate into raw Kubernetes primitives when you need them, all against a cluster Porter manages for you or one you connect from your own AWS, GCP, or Azure account.

The pitch is aimed at teams who see Kubernetes in their future but do not want to learn it as a prerequisite to shipping today. Porter is meaningfully more opinionated than raw Kubernetes and meaningfully less opinionated than a fully managed PaaS, so it lives in an interesting middle ground.

Best for: teams who expect to end up on Kubernetes eventually and want a gradient into it, or teams who already have cluster infrastructure they want to keep using with a nicer deploy loop on top.

Consider elsewhere if: you have no interest in Kubernetes concepts ever leaking into your day-to-day work, or you want a fully managed platform with no cluster-level decisions to make.

Pricing: Metered by compute usage on Porter Cloud, with self-hosted options for teams running on their own accounts.

7. Dokploy

Dokploy is a self-hosted PaaS that sits on top of Docker and Docker Swarm, giving you an application dashboard, database provisioning, and deploy flows on infrastructure you rent yourself. Because the platform itself is open source, you can run it on a single VPS and treat that VPS as your entire production footprint, which is a very different economic profile than any managed service.

The obvious appeal is cost control: for small side projects and internal tools, running everything on a single $5-$20 VPS with a self-managed platform on top can be dramatically cheaper than the equivalent workload on a managed PaaS, especially once bandwidth and add-ons stack up.

Best for: developers who are comfortable managing a VPS, want maximum control over the underlying host, and have workloads whose economics benefit from a fixed monthly VM bill rather than usage-based metering.

Consider elsewhere if: you would rather not manage host-level concerns like OS updates, backups, and VPS-level networking, or you want a platform team's support contract behind your production infrastructure.

Pricing: Open source and free to self-host, with a paid cloud option for teams that want the same platform without running the host.

8. Coolify

Coolify occupies the same self-hosted PaaS space as Dokploy and has grown a substantial community over the last few years. The user experience is broadly similar: connect a Git repository or a Docker image, provision databases as containers on the same host, and get automatic TLS through Let's Encrypt on top of your own server.

Coolify's differentiators are its plugin ecosystem, breadth of one-click databases and services, and a mature web UI that a lot of solo developers and small teams find pleasant to live in day to day.

Best for: solo developers and small teams who want a self-hosted PaaS with an active community, first-class support for a wide range of databases as containers, and no per-service pricing to worry about.

Consider elsewhere if: the workload really needs multi-region deployment, managed Postgres with point-in-time recovery, or a support contract with an SLA behind the platform itself.

Pricing: Open source and free to self-host, with paid cloud tiers available.

9. Dokku

Dokku is the original self-hosted "mini-Heroku" and still one of the leanest options on this list. Built around Docker and Heroku buildpacks, it gives you git push deploys onto a single server with almost no ceremony, and its plugin ecosystem covers Postgres, Redis, and the rest of the classic Heroku add-on lineup.

Where Dokku differs from the newer self-hosted PaaS options is in its philosophy: it is command-line first, opinionated in the Heroku direction (buildpacks, procfiles, twelve-factor conventions), and comfortable running on a single small server for years without much attention.

Best for: developers who want the original Heroku-style deploy loop reproduced on a single VPS, and teams who appreciate a CLI-first, minimal-web-UI approach to platform administration.

Consider elsewhere if: you prefer a web UI as the primary interface, or you need built-in multi-host orchestration without stepping up to Kubernetes.

Pricing: Open source and free to self-host.

How to choose

The question that decides most of these choices is not "which platform is best" but "which platform matches the character of the workload and the operational appetite of the team".

If the workload is a straightforward web application with a Postgres database, a managed platform like Suga, Render, or Railway will remove the largest amount of undifferentiated infrastructure work with the smallest amount of ceremony. Among those, Suga leads with an included edge tier and hosting credits per seat, Render leads on managed Postgres depth and published compliance, and Railway leads on the canvas-first project experience with SSH into running containers.

If latency to end users is central to the product, Fly.io's multi-region model is genuinely differentiated and worth investing in.

If the frontend is Next.js and the backend is mostly a thin API in front of managed services, Vercel is the natural fit even though the runtime model is different from Heroku's dynos.

If you are already on the road to Kubernetes or need to run on your own cluster, Porter is the smoother on-ramp than raw kubectl, and Suga's Enterprise plan can also run against clusters you already own.

If cost control is the primary motivation and you are comfortable managing a host, Dokploy, Coolify, and Dokku each solve the same problem with different flavours: Dokploy leans into a modern web UI on Swarm, Coolify offers the largest community and plugin ecosystem, and Dokku offers the closest reproduction of the original Heroku loop.

Migrating away from Heroku

Whichever destination you pick, the migration itself tends to follow the same rough pattern. Start by containerising the application, since every platform on this list can accept a Dockerfile and this is the deploy artefact most likely to survive any future platform change too. Move the database next by taking a fresh dump from Heroku Postgres and restoring it into the new managed Postgres or container-based database in your target platform, ideally cutting over during a low-traffic window with DNS pointed at both endpoints during the switch. Environment variables are usually the fiddliest part, since Heroku's config vars often accumulate over years, so treat this as an opportunity to audit and remove the ones nobody remembers adding.

Two decisions make the difference between a clean migration and a painful one. First, decide the runtime model deliberately: if you are moving to a serverless platform like Vercel, budget time to rework anything that assumed a warm long-running process, and if you are staying on containers, take the opportunity to consolidate any background workers and web dynos into a single application topology. Second, mirror your Heroku environment layout into whatever the target platform calls its environments (workspaces, projects, environments, spaces), and use the migration as a chance to formalise dev, staging, and production with clean isolation between them.

Frequently asked questions

Is Heroku itself still viable? Yes, Heroku is still a working product and still fits some teams well, particularly those that have been on it for years and have deep integration with its add-on ecosystem. The reason teams look elsewhere is usually a combination of cost curves that no longer match the workload, a preference for containers over buildpacks, or a desire to run on infrastructure the team already owns.

Can I combine platforms? Absolutely, and this is more common than the vendor pages suggest. A typical modern stack might run the frontend on Vercel, the backend on Suga or Render, and background jobs on Fly.io close to the database, all glued together with the same managed Postgres. Pick per workload rather than trying to fit everything onto one platform.

What about AWS, GCP, and Azure directly? They are always an option, and for large enterprises with dedicated platform teams they are often the right answer. For everyone else, the PaaS platforms on this list exist because the raw cloud consoles carry too much undifferentiated infrastructure work for a small team to absorb, so the sensible move is usually a managed PaaS with a bring-your-own-cloud escape hatch for the day the workload outgrows the vendor's own infrastructure.

How should I evaluate free tiers? As evaluation environments, not as long-term hosting. Every free tier on this list has constraints (sleep after inactivity, low resource ceilings, or usage caps) that make sense for kicking the tyres and stop making sense once you have real traffic. Budget for the paid tier of whichever platform you pick, and use the free tier to make sure the deploy loop and operational model actually suit your team before you commit.

What if I want to change platforms again later? Container-based platforms make this dramatically easier than the previous generation of buildpack-based PaaS, since the deploy artefact is a Docker image that any container platform can accept. If portability is a first-order concern, favour platforms that use Dockerfiles as the primary input and treat any platform-specific configuration as a thin wrapper around the container.

I handed Claude a docker-compose.yml and it deployed my app instantly onto Suga

Rk — Thu, 13 Aug 2026 14:38:46 +0000

It turns out you can hand a docker-compose.yml to Claude, point it at a container platform's MCP server, and get a working managed deploy back.

This is really cool, because moving a local compose stack onto a managed platform normally means one of three things: rewriting the deploy shape in whatever config language your target platform speaks, running a converter tool that only handles the parts compose and the platform both understand, or staying on a VM and running docker compose up directly.

Each of those requires an intervention, because compose describes a container topology while modern managed platforms need more than that to run your app well, including managed builds from a git source, auto-provisioned public URLs, cross-service secret references, and per-container scaling and resource specs.

The MCP-and-agent path skips all three: expose a platform's configuration API through an MCP server, and let an AI coding agent do the translation on demand from your compose file, without your compose file ever having to become anything else.

MCP (the Model Context Protocol) is an open standard for giving LLMs structured, permissioned access to external tools. If a platform ships MCP tools like create_environment, add_container, add_volume, and set_secret, an agent can read a compose file, translate the topology, generate the secrets it assumes are external, feed auto-allocated URLs back into env vars, flag missing durability settings, and stop short of actually deploying so a human still presses ship.

To see how it plays out, I pointed Claude (via Claude Code) at Suga's MCP server and asked it to stand up a real Laravel app's compose.yaml in a fresh environment.

The compose file

The example is taken from a Laravel app I run live as a side project to my main gig which is a hosting platform named Suga (https://suga.app), and since I work on Suga and I wanted to see if this would just work.

Here's the compose.yaml so you have the source material inline as we step through the translation:

x-app-env: &app-env
  APP_NAME: myapp
  APP_ENV: production
  APP_KEY: ${APP_KEY:?APP_KEY must be set (e.g. in .env)}
  APP_DEBUG: "false"
  APP_URL: http://localhost:8080
  LOG_CHANNEL: stderr
  LOG_LEVEL: info
  DB_CONNECTION: pgsql
  DB_HOST: postgres
  DB_PORT: "5432"
  DB_DATABASE: myapp
  DB_USERNAME: myapp
  DB_PASSWORD: myapp
  REDIS_HOST: redis
  REDIS_PORT: "6379"
  SESSION_DRIVER: redis
  CACHE_STORE: redis
  QUEUE_CONNECTION: redis

x-depends: &app-depends
  postgres:
    condition: service_healthy
  redis:
    condition: service_healthy

services:
  app:
    image: myapp:latest
    environment: *app-env
    depends_on: *app-depends
    ports:
      - "8080:8080"

  worker:
    image: myapp:latest
    command: ["php", "artisan", "queue:work", "--tries=3"]
    environment: *app-env
    depends_on: *app-depends

  scheduler:
    image: myapp:latest
    command: ["php", "artisan", "schedule:work"]
    environment: *app-env
    depends_on: *app-depends

  postgres:
    image: postgres:16-alpine
    environment:
      POSTGRES_DB: myapp
      POSTGRES_USER: myapp
      POSTGRES_PASSWORD: myapp
    volumes:
      - postgres-data:/var/lib/postgresql/data

  redis:
    image: redis:7-alpine

volumes:
  postgres-data:

The three PHP services share an image built from a repo-root Dockerfile and pull in a large block of shared env vars via the x-app-env YAML anchor, while postgres gets a named volume for its data directory so writes survive a container restart. That's a fairly standard Laravel-plus-Redis-plus-Postgres shape, and anyone who has run compose in dev will recognize the topology immediately.

Suga's model, briefly

To follow along you need a quick sketch of how Suga models an environment, since it doesn't map one to one to compose services:

An environment contains a draft, which is a canvas of containers and volumes.
Each container has either an image reference or a build source (a connected GitHub repo), plus env vars, private networking, and optional public HTTPS.
Containers can reference each other's values e.g. {{<id>.variables.KEY}} where <id> is a short nanoid that Suga assigns when the container is created.
The MCP server exposes tools for shaping the draft, including create_environment, add_container, add_volume, set_env_variable, set_secret, connect_build_repository, and update_build_repository.

The MCP has no deploy tool by design, which is the safety rail: the agent crafts the draft and a human approves and deploys.

Wiring Claude Code up to Suga

I registered Suga's MCP server with Claude Code:

claude mcp add --transport http suga https://dashboard.suga.app/api/mcp

Then I ran /mcp inside Claude Code to authenticate against my Suga account, and every Suga tool referenced in the rest of this post was callable directly from a Claude Code conversation. If you're using Cursor, Cline, or another MCP-capable coding agent instead of Claude Code, the Suga MCP connection guide has the equivalent setup instructions.

The prompt

The prompt I gave Claude was essentially

Deploy the compose.yaml at the repo root to Suga as a new environment on my existing project

with no accompanying instructions or hints, and that direction was enough for it to come back with a plan and a stream of MCP calls.

It did have access to the Laravel project

Translation, decision by decision

services: becomes five add_container calls

Each compose service becomes one add_container call, where the service name doubles as both the container's displayName and its networking.private.hostname. Keeping the hostname aligned with the compose service name matters because the app's env vars refer to services by name (DB_HOST=postgres, REDIS_HOST=redis), and cross-container DNS on Suga's private network resolves off the container's hostname, so the app reaches postgres and redis by the same names it uses locally.

image: myapp:latest becomes a GitHub build source

The myapp:latest tag in compose is really shorthand for "the image you get when you run docker build in this repo," so the natural translation on Suga is to give each PHP service a build source that actually produces that image. The agent did this by connecting app, worker, and scheduler to my GitHub repo on the main branch via connect_build_repository, pointing all three at the same repo-root Dockerfile.

There's a design choice worth calling out here: each Suga container has its own build source, which means you can point three containers at the same repo and Dockerfile for a monorepo setup, or at completely different repos and Dockerfiles for a multi-repo setup, depending on how your services are laid out. Compose's implicit "one image reused across services" pattern via the shared image field is the odd one out in that comparison, and if you want that same shared-image behaviour on Suga the pattern is to build once, push to a registry, and reference the tag from each container.

The agent picked the same-repo path here because it was the most direct translation of what the compose file was already expressing (three services sharing one Dockerfile, one build, one image), and because it stayed inside Suga (three connect_build_repository calls) rather than pulling in a separate registry push as a side quest.

ports: "8080:8080" becomes public HTTPS

The app service's port publish became networking.public.https[{port: 8080}] in the draft, at which point Suga auto-allocated a hostname on the cluster and returned a publicUrl immediately, before the deploy even happened. The response payload looked something like this: https://--..suga.run

Because that URL was available in the same MCP call that created the container, the agent could feed it back into the app's APP_URL env var in a follow-up call, wiring it up in the same pass rather than needing a second step to fill in the value later.

${APP_KEY:?...} becomes a generated secret

Compose's ${APP_KEY:?APP_KEY must be set (e.g. in .env)} syntax means "this variable must come from somewhere, fail loudly if unset." The agent recognized it as a Laravel APP_KEY, generated one with openssl rand -base64 32, and stored it as a sensitive variable in the draft so the value is write-only from that point on.

DB_PASSWORD: myapp becomes a sensitive cross-container reference

The compose file hardcodes the DB password on both the postgres service and the app services, which is convenient for local dev but worth improving on for a real deploy. The agent had a choice between hardcoding the literal password on both sides in Suga too or referencing postgres's secret from the app, and it picked the reference form: DB_PASSWORD = {{.variables.POSTGRES_PASSWORD}}

The <postgres-id> placeholder in that expression is where the postgres container's resource id goes, a nanoid that Suga assigned when the container was created (something in the shape of k3n8vpqm0xrs), and this reference form has no compose analogue at all, which is what makes it interesting: rotating the postgres password now updates every consuming container automatically on the next deploy, without the agent having to remember to keep two literal values in sync.

depends_on: {condition: service_healthy} gets dropped

Modern services are generally expected to retry their dependencies at startup rather than crash on the first failed connection, so the agent handled the compose depends_on gate by trusting that behaviour and letting the app come up naturally. That keeps the draft clean of compose-specific boot orchestration, and the retry logic built into most modern frameworks (Laravel included) does the rest at runtime.

Volumes

postgres-data:/var/lib/postgresql/data became a 1GB Suga volume mounted at the same path on the postgres container, which is essentially the same shape as the compose declaration with a size added.

What the agent noticed, that wasn't in the compose

The compose file doesn't say anything about Redis durability, which is a subtle gap worth flagging: Redis in production usually runs with --appendonly yes and a mounted volume so that a container restart doesn't erase the cache, session store, and queue state Laravel is putting there. The agent didn't quietly add either to the draft, staying faithful to the compose file as instructed, but it did flag the gap in its summary and noted that a real deploy of this shape would probably want durability configured before it ships. That's the kind of observation you get from an agent that has both the compose file in hand and general knowledge of how services like Redis are typically run in production, since a rules-based converter can only see what compose has spelled out.

The result

A couple of minutes and a few MCP calls later, the draft was populated and ready to review:

create_environment call created the new environment
add_container calls added app, worker, scheduler, postgres, and redis
add_volume call added postgres-data mounted on postgres
set_env_variable call fed the auto-allocated public URL back into APP_URL on the app container
connect_build_repository calls wired app, worker, and scheduler to the GitHub repo on the main branch

I opened the Suga link, logged in, reviewed the canvas, and clicked Deploy.

About ninety seconds later the deploy came back failed, with a clean and specific error surfaced right in the deployment record:

Build failed for service "worker": failed to solve: failed to read
dockerfile: open Dockerfile: no such file or directory

That same message showed up once for each of the three source-built services, and it was the branch guess catching up with me. My main branch on this repo happens to be nearly empty, since all of the real code (including the Dockerfile) lives on develop, but the agent had no way to know that and had picked main as a reasonable default.

I told the agent the correct branch name in one sentence, and it made three update_build_repository calls (one per source-built container) to patch each build config in place, leaving the rest of the environment untouched.

I clicked Deploy on the same review link, and the second attempt went through end to end.

The app came up and started responding to requests on the same public URL Suga had allocated back during add_container, which meant the URL I'd already fed into APP_URL on the app container was still the correct one, exactly as the pre-allocation flow implied it would be.

One thing worth noting from the logs: Suga starts all five containers in parallel, which is the fastest way to get a deploy up. During the first thirty seconds the scheduler's Laravel bootstrap tried to talk to postgres:5432 twice while postgres was still finishing initdb, both attempts were logged as expected connection errors, and the scheduler was healthy on subsequent connection attempts once postgres was ready (whether by Laravel retrying internally or by the container restarting and coming back up cleanly). Either path is a normal way for a modern app to handle a downstream service that takes a moment to warm up. For an app that wouldn't recover on its own (a one-shot init container or a strict migration job, for example), adding a container-level retry loop is a small tune-up worth doing in the second pass, once the first deploy is live and you can see everything talking.

What this is really about

For app devs, the reason this pattern matters is iteration speed. docker compose up gets your stack running locally in seconds, keeps every dependency described in one file, and lets you rebuild your mental model of the system quickly whenever you need to, which is why compose is still the default local dev setup for most multi-service apps in 2026. The friction hits the moment you want that same stack running on a managed platform rather than a raw VM, because your compose.yaml only describes half of what the platform needs to know, and you either translate the missing half by hand into whatever config language your platform expects or spend real build cycles pushing to CI just to iterate on the deploy shape.

An MCP-driven flow removes almost all of that friction. Your compose file stays the source of truth for how the app runs locally, and every time you change it, the same one-sentence prompt gets you a fresh production draft with the secrets generated, the build source wired up, the public URL allocated, and the cross-service references in place. The loop between "changed the compose file locally" and "reviewed a production draft" collapses to about the time it takes to type the prompt, which starts to feel a lot closer to the speed of local dev than the speed of a normal deploy pipeline.

Try it

If you want to point Claude at your own compose.yaml, the setup is short and free to try end to end. You'll need a Suga account first, which you can create at suga.app on the free tier, and once you're signed in you can follow the MCP connection guide to wire the Suga MCP server into Claude Code (or whichever MCP-capable agent you prefer). With the server connected, handing your compose file to the agent is a one-sentence prompt away, and you'll step through the same translation flow this post walked through, right up until you're the human clicking Deploy.

Fixing observability across async events (like queues)

Rk — Mon, 20 Apr 2026 18:14:04 +0000

Last week we had a deploy on Suga that was taking longer than usual. It was hard to reproduce, so the only obvious thing to do was try and trace it end to end. We immediately ran into roadblocks because the trace only covered the first 140 milliseconds.

For context: the bulk of our deployment code runs in a background worker and is triggered via a queue, and the worker's trace had no connection back to the request that kicked it off.

Without a complete trace, we were forced to look at everything related to our deployments and burnt time combing through the worker looking for oddities. At this point we were stuck trying to build a trace manually by correlating by timestamps in logs - not a lot of fun.

It turns out, a lot of tracing docs spend very little time on the edge cases like handling asynchronous workloads. Instead, they're more foundational, with a focus on synchronous HTTP requests between services which are a little more straight forward.

When one service calls another the headers carry the trace, and you get a trace log that flows like a waterfall which is the exact behavior we wanted for queued workloads.

Queues don't have headers

HTTP tracing works because requests have headers, a sanctioned place to attach arbitrary metadata that both sides know how to read. Queues generally don't, or at least not in the same obvious way.

They do have payloads, which is the data the job works and the first instinct when you hit this problem might be to cram trace context in there. This might work briefly, but its not a good solution because mixing makes it hard to debug and maintain in the future.

The trick is to find whatever metadata channel the queue already gives you and use that and thankfully, almost every mature queue has one (probably because of this scenario). SQS has message attributes, Temporal has context propagators built into the SDK, and Hatchet (which we use to run our workflows) has a metadata field called additionalMetadata.

What Hatchet gave us

additionalMetadata is a Record<string, string>, which turns out to be exactly the right shape for our trace data. It is structured enough to be useful and unstructured enough to carry whatever you want without Hatchet needing to understand it, but more importantly - no database or schema changes.

This is a big deal, because we were retroactively fitting traceability into our workflows, our requirements were to minimize the impact on all of our callers/workers.

Attaching the headers alongside the existing metadata looks like this:

await workflow.runNoWait(input, {
  additionalMetadata: {
    organizationId,
    ...injectTraceContext(),
  },
});

Anatomy of a trace header

The pattern we used leans on a standard called propagation: the way tracing SDKs serialize the current trace into a small set of string headers on one side of a boundary (like an enqueue event) and rehydrate it on the other (dequeue).

We use Sentry to handle the propagation, but, the pattern will work with any tracing SDK that supports an inject/continue pair.

The header names sentry-trace and baggage follow the W3C Trace Context spec, allowing any compliant SDK to work with each other.

A sentry-trace header is three dash-separated fields:

sentry-trace: 771a43a4192642f0b136d5159a501700-7d3efb1b173fecfa-1
              └──────── trace id ────────────┘ └── span id ───┘ └ sampled

The trace id is the identifier every span in this trace shares, which is how the worker-side spans end up stitched to the web request in the UI.
The span id is the parent the next span should hang off, so the worker's first span slots in as a child of whatever span called runNoWait.
The sampled flag tells the receiver whether this trace is being recorded, so both sides agree on whether to keep the data.

OTel calls the equivalent header traceparent and structures it almost identically.

The baggage field is a comma-separated list of key=value pairs carrying trace-level metadata like environment and release.

The handoff

In Sentry the two sides of propagation are getTraceData() on the way in and Sentry.continueTrace(...) on the way out. We've implemented the two helpers below, injectTraceContext and extractTraceContext, as thin adapters around that pair: one takes the SDK's output and hands it to the queue, the other takes the queue's input and hands it to the SDK.

injectTraceContext grabs the current trace's propagation headers, returning an empty object when there is no active trace data.

export function injectTraceContext(): Partial<TraceContext> {
  const data = Sentry.getTraceData();
  if (!data["sentry-trace"]) return {}; // no active trace on this request, nothing to propagate
  const result: Partial<TraceContext> = { "sentry-trace": data["sentry-trace"] };
  if (data.baggage) result.baggage = data.baggage;
  return result;
}

extractTraceContext takes the metadata coming in from the queue, and hands it to Sentry.continueTrace which wraps the callback so any spans created inside it become children of the original request.

export function extractTraceContext<T>(
  metadata: Record<string, string> | undefined,
  callback: () => T,
): T {
  const sentryTrace = metadata?.["sentry-trace"];
  if (!sentryTrace) return callback(); // upstream didn't propagate, run the task in its own trace
  return Sentry.continueTrace(
    { sentryTrace, baggage: metadata?.baggage },
    callback,
  );
}

Once continueTrace has fired, anything the task does after that including database queries or outbound HTTP calls hangs off the web request's trace automatically.

Making it invisible to the task author

From personal experience, developers aren't good at remembering things like tracing when writing new workers, so instead of exposing extractTraceContext directly, we put it alongside our structured logger inside the withTaskLogging wrapper that every task already uses.

return withTaskLogging(ctx, "deployment", "deploy", async ({ event }) => {
  event.set({ deploymentId: input.deploymentId });
  await deployEnvironment(input);
});

Behind the scenes, withTaskLogging handles the extraction before the task body runs:

return extractTraceContext(ctx.additionalMetadata(), () =>
  Sentry.startSpan(
    { name: `task:${workflowName}/${taskName}`, op: "hatchet.task" },
    async () => {
      // task body runs here, as a child of the original web trace
    },
  ),
);

The payoff

We now have a complete trace across synchronous and asynchronous events and the 30.2 seconds the job spent queued shows up as a visible gap rather than something you'd have to infer from timestamps.

POST /api/deploy                          60ms  web
 ├─ db.query: select env                  12ms  web
 └─ workflow.runNoWait: deploy            18ms  web
                                                        30.2s gap (queued)
 task:deployment/deploy                  8.4s   worker
 ├─ db.query: fetch deployment            45ms  worker
 ├─ k8s.apply: deployment                3.2s   worker
 └─ k8s.wait: service loadbalancer       4.8s   worker

By the way, that gap was the entire mystery of the deploy we opened with: the deploy wasn't actually slow; it was waiting in queue to execute and that gave us the information to know where the issue was.

Can we trust AI to deploy as fast as it generates code?

Rk — Mon, 30 Mar 2026 18:20:10 +0000

AI can generate Terraform, but generating it was never the hard part — trusting and deploying it is.

AI made me faster at writing code, not shipping it

The way I build software has changed more in the last year than in the decade before it. Take a decision like whether to process a new order with a background job or a webhook. Before AI tooling, the workflow looked something like this: identify the problem, spend time researching the tradeoffs, check how others had solved it in similar systems, form a plan, write the code, test it locally, then do everything needed to actually ship it. The whole thing, from identifying the problem to having it running in production, might take a couple of days or weeks depending on the feature.

Now I give an agent the context it needs, the codebase, the outcome, the constraints, and each option comes back as working code (usually) in minutes. The research, the planning, the implementation are compressed. We’re now looking at hours not days, and this is including the time it takes to review the result and wrestle with the AI to get to the end state that I had in mind.

That's not unique to me, I see it both in my organization and amongst my dev friends, AI is fundamentally changing the first half of the journey from problem to production. What hasn't changed much, is everything that happens after the code is written.

Every application has a shadow project

Behind every application in production there's a second project that nobody outside the immediate work ever sees. It's the infrastructure-as-code that provisions your databases and caches, the CI/CD pipeline config that builds and ships the containers, the Dockerfile, and sets the environment variables across staging and prod.

The two projects depend on each other but move independently. Add a service and the deployment project needs matching infrastructure. Add a queue and the pipeline config needs updating. Even something as trivial as changing a port can require an update to the infrastructure configuration.

During my review of the background job I found the performance wasn't good enough, fed that back to the agent, and it came back with a revised approach using a Redis cache. Once I had Redis running locally via Docker it worked perfectly, but from where the agent sits, the application code is the whole picture. It has no idea that Redis now needs to be provisioned, that the pipeline config needs an environment variable, or that staging and prod need to stay in sync.

Keeping the two projects in sync was already one of the harder parts of backend development. Teams hired platform engineers to manage it, but it remained and it was still a common source of production incidents. AI didn't create the problem, it just made it untenable. The problem now is that an agent can introduce new infrastructure requirements every time I prompt it, and it has no idea the shadow project exists.

Just generate the Terraform at the same time

The obvious response is to have the agent generate the deployment project alongside the application code. When the agent adds Redis, have it also write the Terraform module, the pipeline config, and the environment variables at the same time so both projects stay in sync.

The thing is, generating Terraform isn't the hard part. It's really not that different from leveraging an old project or selecting from a library of templates. The hard part is trusting it.

When an agent writes application code, there's a wealth of open source references, community patterns, and established conventions to validate against. IaC is different. It's privately held, idiosyncratic, and often tailored specifically for the application it's coupled to. There's no easy way to know whether what you're looking at reflects best practice or just how the last person happened to solve it. A misconfigured IAM policy or an incorrectly sized database doesn't throw an exception or print a stack trace. It either silently costs money or takes down production in a way that's genuinely difficult to debug.

Whether you write it yourself or generate it, someone has to review it before it touches production. That means verifying IAM policies aren't overly permissive, checking security groups are scoped correctly, confirming instance sizing makes sense for the workload, and running static analysis on the output before applying through staging and production. The work shifts from writing and reviewing Terraform to reviewing AI-generated Terraform, but you didn't write it, so you have to check everything with a fine-toothed comb.
Terraform wasn't built for this

I want to be clear that this isn't unique to Terraform. The same arguments apply to pretty much all IaC tools. The workflow Terraform enforces exists for good reason: plan, review, approve, apply, verify, repeat across environments. Infrastructure mistakes are expensive and sometimes irreversible, and when you're exposing your entire cloud provider's API surface including every IAM policy, every network rule, and every storage configuration, that level of scrutiny is genuinely warranted. The problem is that workflow was designed for infrastructure that changes infrequently and deliberately. It assumes the person writing the Terraform understands the full context of what they're configuring.

AI-assisted development breaks both of those assumptions at once. Infrastructure changes aren't just new resources, they're configuration updates, scaling adjustments, environment variables, and things being removed. During early development and every time a feature is added or removed, the rate of change spikes. Those are exactly the moments when you're moving fastest and leaning on an agent the most.

Think about what just happened with Redis, with just one performance issue during an iteration with your AI and the codebase had a new infrastructure dependency. That wasn't a deliberate decision made in a planning meeting, it was a side effect of a five minute conversation. The person reviewing is no longer reading decisions they made themselves, they're auditing a stream of generated output that has the potential to change the infrastructure in ways that are easy to miss and expensive to get wrong. The review process exists to catch exactly those mistakes, but it wasn't designed to run at this pace. So I'm either keeping up thorough reviews and becoming the bottleneck, or skipping them and accepting the risks.

I just want to show off my app

Frontend teams went through this transition years ago and it wasn't just that Vercel was more convenient. It's that the pace of frontend development made the old model impractical. Frameworks were proliferating, build tooling was changing constantly, and the effort required to keep Nginx configs, SSL certs, and CI/CD pipelines current with all of it had quietly become a job in itself.

When I first used Vercel for a frontend project, what struck me most was the ability to show someone what I'd built straight away, without having to configure a million things and hoping everything was right. Even if there were a few kinks to work out like bot protection or firewall rules, they could be resolved in minutes.

Teams that moved to Vercel didn't do it because they couldn't manage their own infrastructure. Many of them were very good at it. They did it because the return on that investment had collapsed.

Push to production

There are a few ways to close the gap between writing code and shipping it. Some teams that I have spoken to are adapting their review processes with lighter touch on low-risk changes and stricter gates before production. Others are looking to build or further adapt their internal platforms to encode guardrails and reduce per-change overhead. Both are reasonable, but neither gets you out of the review cycle; they just make it more manageable.

There's a specific feeling most backend developers rarely get to experience: finishing something and being able to share it immediately.

That's what we've been working towards with Suga, and it could be an interesting option for you if you aren’t looking to maintain a shadow project. The review, the auditability, the environment parity still matter, the question is whether they have to come bundled with a separate repo, a pile of YAML, and a review workflow that wasn't designed for the pace we're now working at.

Connect your repository, define your services on a canvas, and a push to your branch handles the rest.

Deploying a Sentiment Analysis API with Nitric and Python

Rk — Tue, 08 Apr 2025 18:58:47 +0000

Deploying a Sentiment Analysis API with Nitric and Python

In this guide, you’ll build a serverless API using Nitric and Python that performs sentiment analysis on text input using a pre-trained machine learning model. You'll use the transformers library from Hugging Face and keep your project lightweight by installing dependencies directly with uv.

What You'll Build

A POST /analyze HTTP endpoint
A Python function that loads and uses an ML model
A serverless app you can run locally and deploy to the cloud

Prerequisites

Python 3.10+
Nitric CLI Install using official instructions
uv Install from https://github.com/astral-sh/uv
- (optional) Your choice of an AWS, GCP or Azure account

Create a New Project

Create a new Nitric Python project:

nitric new ml-api py-starter
cd ml-api

Install Dependencies

Use uv to install your required libraries:

uv add torch transformers==4.39.3

Write the Sentiment API

Open or create the file functions/api.py and add the following:

from nitric.resources import api
from nitric.application import Nitric
from transformers import pipeline

# Load the model once globally
analyzer = pipeline("sentiment-analysis", model="ProsusAI/finbert")

# Define an API
ml_api = api("ml-api")

@ml_api.post("/analyze")
async def analyze(ctx):
    body = ctx.req.json
    text = body.get("text")

    if not text:
        ctx.res.status = 400
        ctx.res.body = {"error": "Missing 'text' field"}
        return

    result = analyzer(text)[0]

    ctx.res.body = {
        "sentiment": result["label"],
        "confidence": round(result["score"], 3)
    }

# Create the Nitric app and run it
Nitric.run()

This defines an API route that accepts a POST /analyze request and returns the sentiment and confidence score based on the input text.

Run Locally

Start your app locally using the Nitric CLI:

nitric start

Once running, test it using the dashboard - http://localhost:49152/

Or you can use cURL:

curl -X POST http://localhost:4001/analyze \
  -H "Content-Type: application/json" \
  -d '{"text": "The market is showing strong signs of recovery."}'

Example response:

{
  "sentiment": "positive",
  "confidence": 0.996
}

⚠️ The first request may take a few seconds as the model is downloaded.

Deploy to the Cloud

Step 1: Create a Stack

nitric stack new dev aws

Edit nitric.dev.yaml:

provider: nitric/aws@latest
region: us-east-1

You can also use gcp or azure depending on your cloud preference.

Step 2: Deploy

nitric up

Nitric will:

Package your function (including installed ML libraries)
Provision the API Gateway and runtime
Deploy to your cloud account

The CLI will print the public URL when deployment is complete.

Test the Deployed API

Replace <your-url> with the deployed API base URL:

curl -X POST https://<your-url>/analyze \
  -H "Content-Type: application/json" \
  -d '{"text": "Interest rates are expected to rise."}'

Clean Up

To destroy the cloud stack and avoid charges:

nitric down

Summary

You built a serverless ML inference API
Ran it locally with Nitric
Deployed it to the cloud with one command

You can now adapt this pattern for other use cases like classification, moderation, or lightweight recommendations.

Separating Concerns: Developers vs. Operations

Rk — Wed, 08 Jan 2025 16:16:22 +0000

When you’re building applications for the cloud, you naturally use a cloud provider’s SDK—AWS, Google Cloud, or Microsoft Azure, for example—to interact with services like storage buckets, queues, or databases. What seems straightforward at first can grow into a challenge. Before you know it, your application logic is tangled up in cloud-specific configuration, and two critical teams—Development and Operations—end up stepping on each other’s toes.

The Real Cost of Mixing Code and Cloud Config

Cloud SDKs were designed to make cloud resources easy to access from your codebase. But the convenience they bring also drags provisioning details into application code. In other words, a developer writes logic to “store a file” and ends up having to specify which S3 bucket, its region, and the permission settings. This bleeds infrastructure concerns into the application domain.

For developers, this means spending extra time wrangling IAM permissions and resource configurations instead of focusing on product features.

For operations, it forces them to decode the application’s code just to understand everything that needs provisioning. Suddenly, managing the application’s infrastructure is more about code archaeology than consistent, reliable deployment strategies.

Why Separating Concerns Matters

In well-functioning organizations, developers and operations each have distinct objectives:

Developers want to deliver business value quickly. They need to focus on crafting features and refining user experiences, not dealing with the specifics of IAM policies or message queue retention.
Operations want to ensure those applications run reliably, securely, and cost-effectively in production. They need the freedom to optimize and manage cloud services without rewriting application code.

When either side is forced to handle the other’s responsibilities, it creates friction. This friction can manifest as slower deployments, increased bugs, and a lot of confusion about who is responsible for what.

A Better Way

The key to restoring order is to isolate cloud resource details behind an abstraction. Instead of importing AWS S3 or Google Cloud Storage SDKs directly in your application code, you can use a framework like Nitric that exposes common operations—like creating an API route or storing a file—without tying you to a specific cloud provider.

Consider this code snippet (TypeScript):

import { schedule, bucket } from '@nitric/sdk'

const profiles = bucket('profiles').allow('write'); // Developer expresses "write" intent

schedule('process-often').every('1 days', async (ctx) => {
  const profile = 'image data';
  await profiles.file('image.png').write(profile); // Developer doesn't worry about S3 or GCS
});

Developer’s Perspective

Developers can now write application logic with minimal infrastructure distractions. They know there’s a storage bucket available, but they don’t need to specify which region it’s in or which encryption rules are in place. Their code focuses on creating user value—storing images, sending messages, processing events, etc.

Operations’ Perspective

Operations sees a clearly defined “bucket” resource. They decide whether it’s an S3 bucket, a GCS bucket, or something else. They can choose to enable cross-region replication or encryption, all within their own IaC configuration. Developers aren’t forced to refactor or even touch the code; the ops team simply updates the infrastructure modules as needed.

The Missing Link: Automating the Specification

Abstracting the cloud details is only half the battle. You also need a mechanism to document and communicate which resources each application needs.

Tools like Nitric scan your code to gather a specification of resources used at runtime:

Buckets
Schedules
Queues or Topics
Access policies
Services

That spec is automatically generated—no guesswork. As a result:

Developers don’t have to manually create or maintain a list of necessary resources.
Operations receives a clear map of what the application needs, making provisioning easier and more consistent.

Here’s an example of the kind of specification that might be generated:

{
  "resources": [
    {
      "id": { "type": "Bucket", "name": "profiles" },
      "bucket": {}
    },
    {
      "id": { "type": "Schedule", "name": "process-often" },
      "schedule": {
        "target": { "service": "demo_services-hello" },
        "every": { "rate": "1 days" }
      }
    },
    ...
  ]
}

Converting Specifications into IaC

Once you have a consolidated specification, you can “transpile” it into Terraform, CloudFormation, or other Infrastructure as Code (IaC) tools. For instance, Nitric can automatically generate a Terraform stack using the CDK for Terraform (CDKTF).

Developers remain blissfully unaware of the provider-specific config.
Operations is free to create or reuse modules that set up buckets, schedules, or messages exactly how they want. They can decide on replication strategies, naming conventions, and compliance rules—all outside the application code.

The Outcome: Clear Boundaries, Less Friction

By pushing cloud-specific details out of application code, teams can return to their core competencies:

Developers focus on creating rich features and experiences.
Operations ensures robust, secure, and efficient infrastructure.

This is a sustainable model that benefits both sides:

Less context-switching: Developers don’t wade into Ops territory.
Fewer misconfigurations: Operations controls provisioning details centrally.
Faster deployments: Changes no longer hinge on back-and-forth clarifications or digging into code to find resource dependencies.

Conclusion: Level Up Your Dev–Ops Collaboration

Separation of concerns isn’t a new concept, but it’s often overlooked in cloud development. Mixing infrastructure configurations directly into application code can weigh down both developers and operations, slowing innovation and complicating deployments. By abstracting away cloud SDKs and automating infrastructure specifications, each team can focus on what they do best.

Ready to see it in action? Explore Nitric to discover how you can simplify your application code, streamline provisioning, and enjoy smoother collaboration between Dev and Ops. It’s a win for developer creativity, a win for operational control—and a big win for your entire organization.

Llama 3.2 on Lambda

Rk — Mon, 25 Nov 2024 14:08:21 +0000

Llama 3.2 1B is a lightweight AI model that makes it interesting for serverless applications since it can be run relatively quickly without requiring GPU acceleration.

We’ll use models from Hugging Face and Nitric to demonstrate using it and manage the surrounding infrastructure, such as API routes and deployments.

Prerequisites

uv — for Python dependency management
The Nitric CLI
(optional) An AWS account

Project Setup

Let’s start by creating a new project using Nitric’s Python starter template.

nitric new llama py-starter
cd llama

Next, let’s install the base dependencies, then add the extra dependencies we need specifically for loading the language model.

# Install the base dependencies
uv sync
# Add the llama-cpp-python dependency
uv add llama-cpp-python

Choose a Llama Model

Llama 3.2 is available in different sizes and configurations, each with its own trade-offs in terms of performance, accuracy and resource requirements. For serverless applications without GPU acceleration, such as AWS Lambda, it’s important to choose a model that is lightweight and efficient to ensure it runs within the constraints of that environment.

We’ll use a quantized version of the lightweight Llama 1B model, specifically Llama-3.2-1B-Instruct-Q4_K_M.gguf.

If you’re not familiar with quantization, it’s a technique that reduces a model’s size and resource requirements, which, in our case, makes it suitable for serverless applications but may affect the accuracy of the model.

The LM Studio team provides several quantized versions of Llama 3.2 1B on Hugging Face. Consider trying different versions to find one that best fits your needs, such as Q5_K_M, which is slightly larger but of higher quality.

Let’s download the chosen model and save it in a models directory in your project.

Download link for Llama-3.2-1B-Instruct-Q4_K_M.gguf:

mkdir models
cd models
# This model is 0.81GB, it may take a little while to download
curl -OL https://huggingface.co/bartowski/Llama-3.2-1B-Instruct-GGUF/resolve/main/Llama-3.2-1B-Instruct-Q4_K_M.gguf
cd ..

Your folder structure should look like this:

/llama
    /models
        Llama-3.2-1B-Instruct-Q4_K_M.gguf
    /services
        api.py
    nitric.yaml
    pyproject.toml
    python.dockerfile
    python.dockerfile.ignore
    README.md
    uv.lock

Create a Service to Run the Model

Next, we’ll use Nitric to create an HTTP API that allows you to send prompts to the Llama model and receive the output in a response. The API will return the raw output from the model, but you can adjust this as you see fit.

Replace the contents of services/api.py with the following code, which loads the Llama model and implements the prompt functionality. Take a little time to understand the code. It defines an API with a single endpoint /prompt that accepts a POST request with a prompt in the body. The process_prompt function sends the prompt to the Llama model and returns the response.

from nitric.resources import api
from nitric.application import Nitric
from nitric.context import HttpContext
from llama_cpp import Llama

# Load the locally stored Llama model
llm = Llama(model_path="./models/Llama-3.2-1B-Instruct-Q4_K_M.gguf")

# Function to execute a prompt using the Llama model
def process_prompt(user_prompt):
    system_prompt = "You are a helpful assistant."

    # See https://www.llama.com/docs/model-cards-and-prompt-formats/llama3_1/ for details about prompt format
    prompt = (
        # System Prompt
        f'<|start_header_id|>system<|end_header_id|>{system_prompt}<|eot_id|>'
        # User Prompt
        f'<|start_header_id|>user<|end_header_id|>{user_prompt}<|eot_id|>'
        # Start assistants turn (we leave this open ended as the assistant hasn't started its turn)
        f'<|start_header_id|>assistant<|end_header_id|>'
    )

    response = llm(
        prompt=prompt,
        # Unlimited, consider setting a token limit
        max_tokens=-1,
        temperature=0.7,
    )

    return response

# Define an API for the prompt service
main = api("main")

@main.post("/prompt")
async def handle_prompt(ctx: HttpContext):
    # assume the input is text/plain
    prompt = ctx.req.data

    try:
        ctx.res.body = process_prompt(prompt)
    except Exception as e:
        print(f"Error processing prompt: {e}")
        ctx.res.body = {"error": str(e)}
        ctx.res.status = 500

Nitric.run()

OK, Let’s Run This Thing!

Now that you have an API defined, we can test it locally. The Python starter template uses python3.11-bookworm-slim as its basic container image, which doesn’t have the right dependencies to load the Llama model; let’s update the Dockerfile to use python3.11-bookworm (the non-slim version) instead.

Update line 2:

FROM python:3.11-bookworm
1
FROM python:3.11-bookworm

Update line 19:

FROM python:3.11-bookworm
1
FROM python:3.11-bookworm

Now we can run our services locally:

nitric run

nitric run will start your application in a container that includes the dependencies to use llama_cpp. If you’d rather use nitric start you’ll need to install dependencies for llama-cpp-python such as CMake and LLVM.

Once it starts, you can test it with the Nitric Dashboard.

You can find the URL to the dashboard in the terminal running the Nitric CLI. By default it’s http://localhost:49152. Add a prompt to the body of the request and send it to the /prompt endpoint.

IMAGE GOES HERE

Deploying to AWS

When you’re ready to deploy the project, we can create a new Nitric stack file that will target AWS:

nitric stack new dev aws

Update the stack file nitric.dev.yaml with the appropriate AWS region and memory allocation to handle the model.

provider: nitric/aws@1.14.0
region: us-east-1
config:
  # How services will be deployed by default, if you have other services not running models
  # you can add them here too so they don't use the same configuration
  default:
    lambda:
      # Set the memory to 6GB to handle the model, this automatically sets additional CPU allocation
      memory: 6144
      # Set a timeout of 30 seconds (this is the most API Gateway will wait for a response)
      timeout: 30
      # We add more storage to the lambda function, so it can store the model
      ephemeral-storage: 1024

Since we’ll use Nitric’s default Pulumi AWS Provider, make sure you’re set up to deploy using it. You can find more information on how to set up the AWS Provider in the Nitric AWS Provider documentation

If you’d like to deploy with Terraform or to another cloud provider, that’s also possible. You can find more information about how Nitric can deploy to other platforms in the Nitric Providers documentation.

You can then deploy using the following command:

nitric up

Take note of the API endpoint URL that is output after the deployment is complete.

If you’re done with the project later, tear it down with nitric down.

Testing on AWS

To test the service, you can use any API testing tool you like, such as cURL, Postman, etc. Here’s an example using cURL:

curl -X POST {your endpoint URL here}/prompt -d "Hello, how are you?"

Example Response

The response will include the results, plus other metadata. The output can be found in the choices array.

{
  "id": "cmpl-61064b38-45f9-496d-86d6-fdae4bc3db97",
  "object": "text_completion",
  "created": 1729655327,
  "model": "./models/Llama-3.2-1B-Instruct-Q4_K_M.gguf",
  "choices": [
    {
      "text": "\"I'm doing well, thank you for asking. I'm here and ready to assist you, so that's a good start! How can I help you today?\"",
      "index": 0,
      "logprobs": null,
      "finish_reason": "stop"
    }
  ],
  "usage": {
    "prompt_tokens": 26,
    "completion_tokens": 33,
    "total_tokens": 59
  }
}

Summary

As you’ve seen in the code example, we’ve set up a fairly basic prompt structure, but you can expand on this to include more complex prompts, including system prompts that help restrict/guide the model’s responses or even more complex interactions with the model. Also, in this example, we expose the model directly as an API, but this limits the response time to 30 seconds on AWS with API Gateway.

In future guides, we’ll show how you can go beyond simple one-time responses to more complex interactions, such as maintaining context between requests. We can also include Websockets and streamed responses to provide a better user experience for larger responses.

Bringing declarative infrastructure to developers

Rk — Mon, 14 Oct 2024 15:13:30 +0000

Over the past decade, Terraform has revolutionized how operations (Ops) teams manage cloud infrastructure by providing a simple, declarative way to define and provision resources.
Terraform allows us to declare cloud resources using a simple configuration language (HCL), which can then be executed to create or modify resources.

Here’s a simple example of declaring a bucket:

resource "aws_s3_bucket" "example_bucket" {
  bucket = "my-example-bucket"
  acl    = "private"

  tags = {
    Name        = "MyExampleBucket"
    Environment = "Dev"
  }

  versioning {
    enabled = true
  }
}

When you run terraform apply, terraform relies on providers to parse and translate the configuration into API requests. The provider acts as a bridge between Terraform and the AWS API, it constructs the necessary API calls to fulfill the requirements specification and sends them to AWS, where the actual creation of the bucket, tagging, and versioning happens.

Terraform handles all the communication with cloud providers, ensuring that resources are created or updated according to the configuration file, without requiring direct interaction with cloud APIs. But what about developers who also need to work with cloud infrastructure?

Our current workflow generally leads us towards building an application first, then handling the infrastructure.

This means that developers have a few choices:

Create variables for all unknown configuration and throw the ball over the fence to an ops team
Work intimately with the ops team to detail resource names, configuration etc.
Do it themselves with tools they aren’t in love with

None of these options are particularly desirable as they rely on a lot of manual steps which often lead to communication breakdowns and runtime/provision time errors. We need a way of shifting the declaration of resources forward in the workflow without introducing additional complexity to our teams.

Declarative infrastructure for developers

I’m not suggesting that developers don’t need to know how storage buckets, databases or APIs work, they just don’t need to know the specifics of how they will be provisioned while they are writing their application code.

With Infrastructure from Code (IfC), developers can declaratively define and request cloud resources such as APIs, storage buckets, and queues directly within their application logic. This declarative approach allows developers to specify what they want without worrying about how they will be provisioned.

Here’s an example of how this might work, and, actually, this should look extremely familiar, the only addition a declaration of the developers intent - to allow reading and writing on the bucket and this information is the missing link to assign the right permissions for the resources to interact with each other.

import * as nitric from '@nitric/sdk'

// Define a bucket for storing images
const imagesBucket = nitric.bucket('images').allow('read', 'write')

// Define an API for serving images
const mainApi = nitric.api('main')
mainApi.get('/images/:key', async (ctx) => {
  // Your image-serving logic here
})

From this example it's simple enough to recognize that we need the following cloud resources:

bucket('images').allow('read', 'write'): Creates a cloud storage bucket named "images" and sets access permissions for reading and writing.
api('main'): Creates an API named "main" with a route to serve images. The developer can then define application logic for handling image requests.
OpenAPI specification: From the routes configured we can automatically create the configuration required for our API gateways.

We can automatically generate the requirements specification -

{
  "resources": [
    {
      "id": {
        "type": "Bucket",
        "name": "images"
      },
      "bucket": {}
    },
    {
      "id": {
        "name": "main"
      },
      "api": {
        "openapi": "{\"components\":{},\"info\":{\"title\":\"main\",\"version\":\"v1\"},\"openapi\":\"3.0.1\",\"paths\":{\"/images/{key}\":{\"get\":{\"operationId\":\"imageskeyget\",\"responses\":{\"default\":{\"description\":\"\"}},\"x-nitric-target\":{\"name\":\"api-testing_services-hello\",\"type\":\"function\"}},\"parameters\":[{\"in\":\"path\",\"name\":\"key\",\"required\":true,\"schema\":{\"type\":\"string\"}}]}}}"
      }
    }
  ]
}

Bridging Runtime and Provisioning Gaps

One of the major challenges developers face is ensuring that the resources their application depends on are correctly provisioned at runtime. The generated specification ensures that the correct resources (storage buckets, APIs, etc.) are provisioned with the appropriate settings (permissions, paths, etc.). This removes the potential for mismatches between what the application requires and what is provisioned, ensuring a smoother deployment process.

We can now use it to orchestrate the necessary IaC, like the Terraform module above, to provision the required resources. This approach minimizes runtime and provision-time errors, ensuring that resources are automatically provisioned in line with application requirements.

It also means that the same application can be deployed across different cloud providers, which means you can pivot at any time to take advantage of the latest technologies and disruptors like AI.

Nitric is Terraform for Developers

Rk — Mon, 19 Aug 2024 14:45:40 +0000

As of the 29th of July this year, Terraform has been around for an entire decade! That’s right a one and a zero, Happy Birthday Terraform 🎉.

Terraform is best known for its declarative approach to infrastructure engineering, the product of which is executable documentation. Not only do we get clean documents that allow a reader to quickly reason about the architecture of a deployment, but that documentation can also be executed by Terraform to produce and execute a plan for that deployment.

Unsurprisingly, after a full decade of modern Infrastructure as Code (IaC) tooling we’re starting to see a whole new category of software emerge around it, Infrastructure from Code (IfC).

The core goal of the IfC paradigm is to bring to application development the same capability that tools like Terraform have allowed infrastructure engineers to enjoy for the past 10 years: the ability to document the requirements of our applications in a clear, concise, and executable fashion.

Using Nitric as an example:

import * as nitric from "@nitric/sdk";

// Document we need a bucket
const imagesBucket = nitric.bucket('images').allow('read', 'write');

// Document we need an API
const mainApi = nitric.api('main');
mainApi.get('/images/:key', async (ctx) => {
  // Code here
});

The Nitric SDKs allow us to not only express our requirements but also allow interaction with those requirements as well.

While these requirements can be spread out across a number of different services as files, it’s simple for us to capture this in a single highly declarative format using the Nitric CLI if we run:

nitric spec -o spec.json

Against our project, your Nitric infrastructure specification can be viewed as a simple JSON document, detailing your application’s requirements.

{
  "resources": [
    {
      "id": {
        "type": "Bucket",
        "name": "images"
      },
      "bucket": {}
    },
    {
      "id": {
        "name": "main"
      },
      "api": {
        "openapi": "{\"components\":{},\"info\":{\"title\":\"main\",\"version\":\"v1\"},\"openapi\":\"3.0.1\",\"paths\":{\"/images/{key}\":{\"get\":{\"operationId\":\"imageskeyget\",\"responses\":{\"default\":{\"description\":\"\"}},\"x-nitric-target\":{\"name\":\"api-testing_services-hello\",\"type\":\"function\"}},\"parameters\":[{\"in\":\"path\",\"name\":\"key\",\"required\":true,\"schema\":{\"type\":\"string\"}}]}}}"
      }
    },
    {
      "id": {
        "type": "Policy",
        "name": "3ed1c501e832fef808a464f42959858c"
      },
      "policy": {
        "principals": [
          {
            "id": {
              "type": "Service",
              "name": "api-testing_services-hello"
            }
          }
        ],
        "actions": [
          "BucketFileList",
          "BucketFileGet",
          "BucketFilePut"
        ],
        "resources": [
          {
            "id": {
              "type": "Bucket",
              "name": "images"
            }
          }
        ]
      }
    },
    {
      "id": {
        "type": "Service",
        "name": "api-testing_services-hello"
      },
      "service": {
        "image": {
          "uri": "api-testing_services-hello"
        },
        "workers": 1,
        "type": "default"
      }
    }
  ]
}

Behind the scenes, Nitric uses this infrastructure specification to provide support for multiple clouds and IaC technologies.

If you check out this recent blog, Putting Terraform behind an API you can see how the above JSON could be provided as a payload. Suddenly you would have an API that outputs Terraform directly from application code.

The point here is that while Terraform and similar technologies have existed for the past decade, providing Ops and Infrastructure teams with the tools they need to cleanly document their infrastructure, application development teams have had access to no such tools in this time. This is what the emergence of IfC is all about.

If you’re someone who works with infrastructure to help organizations deliver value by getting software out into the world, and care about a smooth and cohesive development experience, I would urge you to take a closer look at IfC as a whole and how you could leverage existing technology in this space or even build your own version of it.

If you'd like to support us, please check out the framework and give it a try, also help us spread the word by 🌟 starring us on GitHub!

Secure Upload URLs Buckets with Nitric in Python

Rk — Mon, 11 Mar 2024 18:54:15 +0000

Creating a secure method for users to upload files directly to an S3 bucket is a common requirement in modern web applications.

By using pre-signed URLs, you can allow a client to upload a file directly to S3 without exposing your AWS credentials, they also provide a secure mechanism to upload files directly to S3 from a client, avoiding the need to pass the file data through your server.

In this tutorial, we will leverage the Nitric SDK in Python to generate secure upload URLs for an S3 bucket, which can then be used from your front-end application.

If you haven't used the Nitric SDK before, then start with this tutorial.

Step 1: Initialize Nitric Bucket Instance

Inside the main function, initialize a new Nitric bucket instance with the name of your S3 bucket.

from nitric.resources import api, bucket
from nitric.application import Nitric
from nitric.context import HttpContext

main_api = api('main')
images = bucket('images').allow('reading')

Step 2: Generate a Secure Upload URL

Now, generate a secure upload URL for a specific file. In this example, we are creating a URL for uploading a file named cat.png. We also specify a time-to-live (TTL) for the URL of 300 seconds (5 minutes).

from nitric.resources import bucket
from nitric.application import Nitric

assets = bucket('assets').allow('reading')

logo = assets.file('images/logo.png')

logo_url = await logo.download_url(expiry=300)

Nitric.run()

Now, you have a secure upload URL for your cat.png file. You can use this URL in your front-end application to securely upload the cat.png file to your S3 bucket. The URL will expire after 1 hour, ensuring that the upload URL cannot be misused after a reasonable amount of time.

This setup abstracts much of the boilerplate and complexity involved in generating pre-signed URLs for S3, allowing developers to focus more on building their applications.

In the case that you want to get an image url for rendering, then simply redirect the URL with a 303 status and a location.

from nitric.resources import api, bucket
from nitric.application import Nitric
from nitric.context import HttpContext

main_api = api('main')
images = bucket('images').allow('reading')

@main_api.get('/images/:id')
async def get_image(ctx: HttpContext):
  id = ctx.req.params['id']
  url = await images.file(id).download_url()
  ctx.res.status = 303
  ctx.res.headers['Location'] = [url]

Nitric.run()

The Nitric SDK offers a streamlined approach to interacting with AWS S3, among other cloud services, in a cloud-agnostic manner.

Running a scheduled task with Nitric in Python

Rk — Mon, 11 Mar 2024 18:09:32 +0000

Setting up a scheduled job is a common requirement for many modern applications.

With the Nitric SDK, this task is simplified, and you can even test your scheduled jobs offline without needing Terraform.

In this tutorial, we'll walk through setting up a scheduled job that aggregates data every three days using the Nitric SDK in Python.

If you haven't used the Nitric SDK before, then start with this tutorial.

Step 1: Import Necessary Libraries

Start by importing the necessary libraries.

from nitric.resources import schedule
from nitric.application import Nitric
from nitric.context import IntervalContext

Step 2: Define Your Scheduled Job

Next, define a function for your scheduled job. In this example, we're creating a new scheduled job named "aggregate-data" that will run every three days.

Inside the function, we've added a simple print statement "aggregating data" to demonstrate the job's activity.

from nitric.resources import schedule
from nitric.application import Nitric
from nitric.context import IntervalContext

report_schedule = schedule('run-a-report')

@report_schedule.every('1 days')
async def process_transactions(ctx: IntervalContext):
  # do some processing
  fmt.Println("aggregating data")

Nitric.run()

Test Your Scheduled Job Offline

nitric start

One of the advantages of using the Nitric SDK is the ability to test your scheduled jobs offline. This is particularly useful for debugging and ensuring your jobs are working as expected before deploying them to the cloud.

In just a few steps, you've now created a scheduled job that aggregates data every three days, here are a few more examples showing you how to set up other frequencies.

description	example schedule
Every day	@schedule("work").every("day")
Every 14 hours	@schedule("work").every("14 hours")
Every 30 minutes	@schedule("work").every("30 minutes")
Every day (cron)	@schedule("work").cron("0 0 * * *")

Creating Bucket (S3) Notifications on AWS with Python

Rk — Mon, 11 Mar 2024 17:59:11 +0000

In this tutorial, we’ll explore how to set up bucket notifications in Python, enabling your applications to react to changes in file storage buckets. This is particularly useful in scenarios where you need to monitor a bucket for new file uploads or deletions.

If you haven't used the Nitric SDK before, then start with this tutorial.

1. Create a Bucket Instance

Import the necessary packages and instantiate a new bucket object. In this tutorial, we'll name our bucket 'assets'.

from nitric.resources import bucket
from nitric.application import Nitric

assets = bucket("assets")

accessible_assets = bucket("assets").allow("reading")

2. Set Up Notifications for File Write Events

To monitor for file write events, specifically for files starting with a certain prefix, use the On method on the bucket object. Let’s trigger notifications for files starting with '/users/images'.

from nitric.resources import bucket
from nitric.application import Nitric

assets = bucket("assets")

accessible_assets = bucket("assets").allow("reading")

# The request will contain the name of the file `key` and the type of event `type`
@assets.on("delete", "*")
async def delete_anything(ctx):
  print(f"a file named {ctx.req.key} was deleted")

@assets.on("write", "/images/cat")
async def create_cat_image(ctx):
  print(f"A cat image was written")

# If `on` is called with a permissioned bucket, a file will also be provided with the request
@accessible_assets.on("write", "/images/dog")
async def access_dog_file(ctx):
  dog_image = await ctx.req.file.read()

  print(dog_image)

Nitric.run()

3. Set Up Notifications for File Delete Events

Similarly, to monitor for file deletion events for any file, set up a notification like so:

from nitric.resources import bucket
from nitric.application import Nitric

# Create a reference to an 'assets' bucket with permissions to delete
assets = bucket('assets').allow('deleting')

logo = assets.file('images/logo.png')

await logo.delete()

Nitric.run()

Avoid writing or deleting to the bucket from within a notification as this can trigger the notification again, potentially causing an infinite loop which can be costly.

Let's run this thing!

nitric start

Conclusion

Now you have set up bucket notifications in Python using the Nitric SDK.

This setup will help you monitor file write and delete events, allowing your application to react to changes in your bucket accordingly.