- Repository: github.com/hrodrig/gfire
- Release: v1.0.0
- Site: gfire.net
Imagine a high-end sports car engine. Beautiful engineering. Everything packed tight under one hood — every gear meshing within millimeters. One piston fails, or a tiny oil leak starts in a corner you cannot see… and the whole car stops cold. Everything shared the same space. Tragically, the same fate.
That is also the drama behind a thousand 3 a.m. pages. A “secondary” task — resize a catalog image, generate a PDF, fan out emails — shares RAM, CPU, and process lifetime with the app that serves users. The OOM killer does not care which thread was “just a worker.” The storefront dies with the job.
We like things glued together. It feels efficient. One package to deploy. The real cost of that hyper-integration is fragility: when your whole machinery shares one runtime, an isolated failure becomes a systemic collapse.
GFire v1.0.0 (production-ready, MIT, Go) proposes the opposite move: pull the combustion engine out of the chassis. A headless, standalone job service. Your app never imports it. You enqueue over HTTP. Handlers are external processes. Storage is PostgreSQL, Redis, or ValKey. Language-agnostic by design — Python, Rust, Java, shell; GFire does not ask you to marry its ecosystem.
This article walks that argument the way a deep-dive conversation would: objections included.
The problem with “just a library”
Celery, Sidekiq, Asynq, River — excellent tools. They also usually live inside your process:
- Same memory space as the HTTP API
- Same deploy unit
- Same blast radius
Picture an online store. The main app serves checkout. In the background, a job starts resizing a massive image catalog. The job eats RAM without bound. The OS panics and kills the process. Customers see errors because checkout and the resize job shared one fate.
That is the sports-car trap in software form.
Separate the kitchen from the dining room
GFire runs as its own binary (or container). It enqueues and orchestrates. Business logic stays in handlers you already trust.
Think of a packed restaurant:
| Old model | GFire |
|---|---|
| The waiter cooks the steak mid-floor — smoke, chaos, elbows | A kitchen at the other end of the building; the waiter only passes tickets through a window |
The dining room seats guests. The kitchen absorbs the heat. If the kitchen catches fire, the dining room can still take reservations.
Enqueue is deliberately boring HTTP:
curl -sS -X POST http://127.0.0.1:8080/v1/jobs/enqueue \
-H 'Content-Type: application/json' \
-d '{"name":"resize_image","args":{"object_key":"s3://bucket/cat.jpg"}}'
No shared library. Node today, Rust tomorrow, C# next week — any producer that can POST JSON can hand a ticket through the window. Scale a viral marketing spike by spinning more GFire pods — without cloning the entire storefront fleet.
“But HTTP is slower than a function call”
Here is where performance-minded readers raise a hand. Passing a ticket through a window — even on localhost — is orders of magnitude slower than calling a function already in memory. Nanoseconds versus microseconds (or a full millisecond across a pod boundary). That penalty is real.
At production scale, that fraction of a millisecond is a tiny price for failure isolation. If a job collapses the kitchen, the dining room stays open and keeps charging cards. You also buy independent scale and polyglot producers without hunting for a compatible queue library in every language.
If your bottleneck is enqueue RTT, you are already in a good place.
Instruction cards, not moving vans
GFire’s design leans on thin instruction cards — roughly a kilobyte in spirit (there is a hard upper limit; do not abuse it). Do not treat the queue as a moving van. A 3 GB CSV in the HTTP body saturates the network and the job store.
Heavy data belongs in object storage (S3 and friends) or your primary database. The card is the ticket: user id, object URL, “handle this.” When a worker claims the card, it does not “understand” your business language. It reads YAML that maps job name → a system command, spawns a subprocess, passes args, waits, and records the OS exit status.
# conceptual — see gfire.example.yaml
handlers:
resize_image:
cmd: ["/usr/local/bin/resize", "--job"]
That handler can be Python, a Rust binary, or a bash script. Continuations chain the same way: on success / failure / any, enqueue the next card (onSucceeded, onFailed, onAny in the spec).
“Isn’t spawning a process killing a fly with a cannon?”
Another fair objection. Starting a Python or Node interpreter from scratch for a one-kilobyte ticket is expensive compared with a free goroutine or thread.
GFire trades raw spawn speed for a property distributed systems care about: predictability. Orchestrator memory stays flat because the OS owns the handler’s heap. If a script leaks ten gigabytes, the OOM killer annihilates that subprocess. GFire notices the unexpected death, marks the job failed, and takes the next card. In a thread-shared pool, the same leak often takes every coworker with it.
Trust the OS to isolate processes. That is what it is good at.
How the pieces fit
No Raft — storage is the referee
In real production you will not run one box. Five, ten, twenty GFire nodes may claim cards at once. How do you stop three pods from sending the same customer email three times?
GFire does not run Raft (or any peer gossip election) between nodes. Peers barely know each other exist. The only source of truth is the storage layer.
Redis / ValKey. Instead of polling “any jobs? any jobs?” every millisecond, workers use blocking pop (BRPOP-style): sleep until work arrives, then wake one worker. Atomic claim uses Lua so a job cannot be stolen mid-hand-off. Schedules lean on sorted sets keyed by Unix time.
PostgreSQL. Many people blink when they see Postgres on a “queue” list. Older relational patterns really did convoy: ten workers fight for one row; nine wait; deadlocks appear under load. Modern Postgres has FOR UPDATE SKIP LOCKED: “give me the oldest available row — if someone else holds it, skip it and give me the next.” Hundreds of workers can hit the same table and receive distinct rows. Pair that with fast natural ordering and time-sortable ids (UUIDv7 in the design), and Postgres becomes a serious job store — often chosen for ops simplicity: one less Redis cluster to patch at 3 a.m. when you already run Postgres for product data.
Redis/ValKey stay first-class when you want memory speed. Pick the referee that matches your ops story.
Production-shaped details (why “1.0” is a big word)
A cool GitHub repo is not the same as “ship this Monday.” The boring essentials matter. Behavior lives in SPECIFICATIONS.md — that document is the contract, not this post.
Recurring jobs (cron). Five nodes at midnight must not email the sales director five times. Distributed locks in storage: one winner holds the baton; the rest go back to sleep.
Bulk enqueue. POST /v1/jobs/enqueue/batch with partial acceptance — five bad rows in a ten-thousand-job payload do not trash the other 9,995. You get a detailed reject list instead of retry hell.
Idempotency keys. Classic terror path: server wrote the job, network died before your client saw 200, you panic-retry. Same key → same job id → no duplicate side effects. Vital for payment-shaped workflows.
Lifecycle → dead letter. Enqueued → processing → terminal (succeeded / failed / canceled). Retries use exponential backoff; exhausted jobs go DEAD (DLQ-style) — still visible, not mysteriously gone. CLI to list, inspect, requeue. Prometheus metrics for the Grafana you already run.
Honest picks (without losing the plot)
GFire is not the only way to run background work. Celery, Sidekiq, River, and Asynq remain excellent when you want ultra-cheap in-process fan-out and deep language idioms. Temporal/Cadence own long-running workflow history. Kubernetes Jobs own cluster-native one-shots. A raw broker (NATS, RabbitMQ, SQS) is plumbing — you still build claim, retry, and handler lifecycle.
Pick GFire when isolation and language-agnostic enqueue are the job you are hiring for.
Pick the library you already trust when shared-memory speed wins.
Philosophy from the deep dive, in one line: heavy work and the user-facing app never should have shared a bed.
Try GFire
git clone https://github.com/hrodrig/gfire.git
cd gfire
make build
cp gfire.example.yaml gfire.yaml
./bin/gfire server --config gfire.yaml
Enqueue with the curl snippet above. Health: curl -sS http://127.0.0.1:8080/healthz and /readyz. Binaries: GitHub Releases — v1.0.0. Site: gfire.net. License: MIT.
Closing
GFire 1.0.0 is the release where that separation — HTTP tickets, storage-backed peers, external handlers — stops being a sketch and becomes a baseline you can build runbooks on.
If the kitchen-window model clicks for you, star the repo or open an issue with the handler or storage edge case I have not thought of yet. Contributions welcome: examples, docs clarity, honest production war stories. That feedback shapes what comes after 1.0.
Clone traffic for this repo lives on gghstats — hrodrig/gfire — raw GitHub traffic, no vanity smoothing. Full index: gghstats.hermesrodriguez.com.
Thank you for reading — and if you already run GFire (or anything else in this little OSS family) somewhere that matters, thank you for trusting it with real workloads. Open source only works when people show up: issues, stars, runbooks, and honest feedback. Let's keep building dependable OSS together — see you in the next post.
Disclosure
Written by Hermes Rodríguez. This piece was shaped from a long-form architecture conversation about GFire, then drafted and edited with AI assistance. Technical claims were checked against SPECIFICATIONS.md and the v1.0.0 tag. Illustrations were AI-generated for this post. Treat the article as an introduction, not a substitute for the specification or the code.
Always verify behavior on the release you install.
Top comments (0)