Replacing Queues with gRPC in PikoCI
When I started building PikoCI, I kept coming back to a simplification: at its core, a CI/CD system is just jobs and workers waiting to be told to do them. Strip away everything else and what you have is a notification problem: the server knows a job is ready, a worker needs to find out.
That's a textbook pub/sub pattern. So I built PikoCI on queues from the start.
Why queues were the right call
I used go-cloud's pubsub abstraction, which let me support multiple backends (NATS, Kafka, RabbitMQ, or an in-memory queue for local development) without rewriting the worker logic for each one. People could plug in whatever they already run, or use nothing at all for local setups.
Queues also solve worker registration cleanly. A worker just subscribes to a topic. It doesn't need to be reachable by the server, doesn't need a stable address, doesn't need anything beyond outbound access to the queue. Adding a worker anywhere is trivial.
But there was a recurring question I kept arguing with myself about: was the queue actually buying me anything over a direct connection? Running a queue is one more thing to deploy, one more thing to operate, for self-hosters who just want a binary. I kept using it anyway because few CI/CD tools use this pattern and I wanted to see it through.
Where it started breaking down
The cracks showed up gradually, mostly while implementing features that needed more from the queue than "deliver this message."
Ordering wasn't guaranteed. go-cloud's in-memory pubsub doesn't guarantee FIFO delivery. I found this out while building worker tagging, when jobs started arriving out of order in ways that broke the scheduler's assumptions.
Retrying was painful. If a worker needed to re-queue a job (say, after a transient failure) there was no guarantee it would come back to the front of the line. It could end up anywhere, with no way to reason about when it would actually run.
Tagging didn't map across backends. Worker tagging needed dynamic routing: a worker with tags like gpu and linux should only receive matching jobs. NATS does this naturally with subject-based routing. Kafka and SQS don't, not without real workarounds.
Cancellation was slow. Workers had no way to learn that a build had been cancelled other than polling the server. That meant a cancelled build could keep running for seconds before the worker noticed, wasteful and visibly wrong in the UI.
Log streaming was a separate problem entirely. Build logs needed their own mechanism to get from worker to server, completely unrelated to the job dispatch queue. Two communication paths to build and maintain for what is fundamentally one relationship: server talks to worker.
Each of these pushed me toward solving the problem outside the queue rather than with it. The message stopped carrying the job payload entirely and became just a ping, a doorbell. The worker would get notified, then immediately turn around and ask the server over HTTP: "what do I need to do?" The server's scheduler held all the actual logic about what was next, regardless of which queue backend was configured underneath.
Considering NATS-only
Before dropping queues entirely, I seriously considered the opposite move: drop every backend except NATS and the in-memory option. NATS could actually do what I needed. Subject-based routing handled tagging naturally, and it's about as lightweight as a queue gets, which fit PikoCI's "easy to deploy" goal better than Kafka or RabbitMQ ever would.
But getting there required NATS-specific configuration: JetStream settings, subject hierarchies tuned for tagging, ordering guarantees that the generic abstraction didn't give me for free. That's the part I didn't like. It stopped being "bring whatever queue you already run" and became "run NATS, configured exactly this way, or tagging won't work." That's not a pluggable abstraction anymore, that's a hard dependency wearing an abstraction's clothes.
And even with NATS configured perfectly, I'd still need the worker to ask the scheduler what to do next, because the scheduler is the only thing that actually knows the answer. The queue would just be there to make the connection happen. At that point I was about to add an entire piece of required infrastructure whose only job was to say "hey, ping."
The moment it clicked
By the time I was deep into the tagging work, the pattern was obvious: the worker was already asking the scheduler directly for its next job. The queue's only remaining job was to say "hey, something happened, go ask." That's not nothing, but it's also not worth running a separate piece of infrastructure for.
If the server already knows which workers exist, it can just tell them directly. No doorbell needed.
Why gRPC
gRPC gives me a persistent bidirectional stream between server and worker. The worker opens one connection and keeps it open. Over that single connection:
- The server pushes jobs to the worker the moment they're ready. No asking, no doorbell
- The server pushes a cancellation message instantly when a build is cancelled, milliseconds instead of the multi-second polling interval cancellation needed before
- The worker streams build logs back in real time
- Heartbeats flow both ways, so the server always knows which workers are alive
One connection, one mental model, no external infrastructure to run or reason about. Workers still only need outbound network access; that part of the queue model's appeal didn't have to be sacrificed.
What this replaced
Before:
go-cloud pubsub (NATS / Kafka / RabbitMQ / in-memory)
--pubsub-system flag
Queue message as a doorbell, worker asks server what's next
Worker polls GET /api/builds/:id/status for cancellation
Separate mechanism for log streaming
After:
gRPC bidirectional stream on a dedicated port
No --pubsub-system flag, removed entirely
Server pushes jobs directly on the open stream
Server pushes CancelJob message on the same stream
Logs stream on the same connection
The architecture today
Two ports. Port 8080 serves the HTTP JSON API: everything the frontend and CLI talk to, unchanged. Port 9090 serves gRPC, workers only. The two transports never mix.
Multi-server deployments still use Postgres LISTEN/NOTIFY to coordinate which server instance dispatches a job; that part didn't change. What changed is how the chosen server instance gets the job to the worker: a direct push on an open stream instead of a queue message.
The migration was a clean break. v0.5.0 dropped the --pubsub-system flag entirely. Workers needed to update to the matching version, but since PikoCI ships as a single binary that's both server and worker, upgrading one means upgrading the other. No transition period with both protocols running in parallel.
The result
Removed an entire category of optional infrastructure. Self-hosted PikoCI users no longer need to think about which queue backend to run, or whether to run one at all. Cancellation now takes effect in milliseconds instead of seconds. Log streaming and job dispatch share one connection instead of two separate mechanisms.
Shipped in v0.5.0.
github.com/pikoci/pikoci · pikoci.com · Apache 2.0
Top comments (0)