DEV Community: Fahim Faisaal

How VarMQ Processes Millions of Messages with an Event Loop and Worker Pool

Fahim Faisaal — Sat, 01 Aug 2026 01:31:56 +0000

Processing a million messages is easy if time and memory don't matter. The hard part is doing it without paying a big dispatch cost per message, without letting concurrency run wild, and without spinning up a fresh goroutine or a fresh allocation every time something arrives.

VarMQ tackles that with two pieces that cooperate:

One event loop decides when the next job may start.
A reusable worker pool runs jobs concurrently, up to a limit you configure.

The queue sits between the producers and these two pieces. Producers can add work as fast as they want, but the event loop controls how quickly that work reaches the pool. That separation keeps the submission path cheap, and it means a burst of messages never becomes an equally large burst of goroutines.

The processing path

Here's what a message goes through inside an in-memory VarMQ worker:

Adding a message never creates a goroutine. It builds a job, enqueues it, and sends a wake-up notification to the worker. Only the event loop can admit queued jobs into execution inside the worker pool.

The event loop dispatches; it does not execute

The core loop is small enough to understand in one pass. This is a shortened version of the event loop

for {
    select {
    case <-ctx.Done():
        // Wait for in-flight work, clean up the pool, and stop.
        return

    case <-signal:
        for w.IsActive() &&
            w.curProcessing.Load() < w.concurrency.Load() &&
            w.queues.Len() > 0 {
            cont, err := w.processNextJob()
            if err != nil {
                w.sendError(err)
            }
            if !cont {
                break
            }
        }
    }
}

When there's nothing to do, the loop just blocks in select. It doesn't poll the queue, and it doesn't burn a CPU core while idle.

Once signaled, it keeps dispatching until one of three conditions changes:

The worker is paused or stopping.
The configured concurrency limit has been reached.
Every bound queue is empty.

So a single wake-up can dispatch many jobs. There's no one-to-one relationship between messages and event-loop wake-ups.

The signal channel has a capacity of one, and notifications use a non-blocking send:

select {
case w.eventLoopSignal <- struct{}{}:
default:
}

If 10,000 messages arrive while a wake-up is already waiting, the extra notifications collapse into the pending signal. The event loop wakes up, inspects the real queue length, and fills every available processing slot. That's how a burst of submissions avoids becoming thousands of blocked signal senders.

Workers send the same notification when they finish. Releasing a processing slot decrements the in-flight counter and wakes the loop if more capacity is available. The result is a short feedback cycle:

message arrives -> event loop fills pool -> worker finishes -> slot released -> event loop fills slot

The pool bounds concurrency

The event loop dispatches serially, but handlers run concurrently inside pool nodes. Each node owns a small channel and a goroutine that serves it. When the event loop dispatches a job, it removes an idle node from the pool and sends the job to it.

If no idle node exists, VarMQ creates another one. The event loop's curProcessing < concurrency check keeps this from running past the configured limit. A queue may hold a million messages, but a worker configured with concurrency 32 will execute at most 32 at a time.

When a handler returns, its node follows one of two paths:

It goes back into the idle list when more work is waiting or the configured idle-worker policy wants to keep it warm.
It stops, and its linked-list node goes into a sync.Pool cache for later reuse.

You can tune concurrency while the worker is running. Increasing it wakes the event loop immediately. Decreasing it changes admission for new jobs and lets excess idle workers disappear without interrupting jobs already in flight.

This is where VarMQ's concurrency comes from. The event loop does a small amount of coordination; the pool spends its time inside application handlers.

Where the memory savings come from

"Low memory usage" here means low overhead per message and bounded execution resources. It does not mean an unlimited backlog somehow consumes constant memory.

VarMQ trims overhead in a few places.

Chunked queue storage (Default standard Queue)

The standard FIFO queue stores jobs in linked buffer chunks instead of repeatedly growing and copying one large slice or a linked list node for each message. It starts with space for 1,024 entries. When a chunk fills, the next chunk grows by 50 percent, up to the configured maximum chunk size.

Enqueue and dequeue remain amortized O(1). Crossing a chunk boundary advances a pointer rather than copying unread messages into a new array. A dequeue also clears the consumed slot, so the queue doesn't keep a reference to a completed job and its payload.

The payoff: no large copy spikes when a single backing array grows under a heavy backlog.

Reused worker nodes

Pool nodes move between the busy state, the idle linked list, and a sync.Pool cache. Reusing those nodes means the bookkeeping around a worker doesn't have to be rebuilt every time traffic rises after an idle period.

The optional idle expiry policy also lets an application pick its memory and latency tradeoff. Keeping idle workers warm reduces dispatch latency. Expiring excess workers reduces retained pool state during quiet periods. VarMQ always keeps at least the configured minimum idle ratio, with one idle worker as the default minimum.

Small coordination state

The worker tracks its status, concurrency, and current processing count with atomics. Locks still protect queue chunks, pool lists, and condition-variable transitions, so the design is not lock-free. But the hot checks in the event loop don't need a large shared scheduler object or a goroutine for every queued message.

VarMQ also offers different job types for different needs. A fire-and-forget Worker carries less result state than ErrWorker or ResultWorker. Choosing the smallest useful abstraction shows up directly in the allocation numbers.

VarMQ under load: a head-to-head benchmark against PondV2

The repository's own microbenchmark - a trivial handler, one CPU, concurrency one - measures VarMQ's scheduler overhead in isolation (roughly 1.01 million round trips per second for single adds, 1.54 million for batches). To see how the design behaves under realistic load - many producers, different handler shapes, and different CPU counts - we ran a head-to-head comparison against PondV2, a popular Go worker pool with a feature set close to VarMQ's. VarMQ's design was partly inspired by PondV2, which makes the comparison fair and informative.

The full benchmark suite lives in the varmq-benchmarks repository. Interactive charts for every result, grouped by workload and CPU configuration, are published at goptics.github.io/varmq-benchmarks.

Each run exercises the complete path from submitting tasks to waiting for completion. The suite covers:

three task shapes - Sleep10ms (I/O simulation), Loop10 (CPU-bound math), and Sleep5Loop5 (mixed);
five producer patterns - from 1 user submitting 1M tasks (1u-1Mt) to 1M users submitting 1 task each (1Mu-1t);
pool sizes of 50k, 100k, and 300k workers;
three CPU configurations - 4, 8, and 24 (the i7-13700's default GOMAXPROCS).

All runs used the same machine as the microbenchmark: an Intel Core i7-13700 on Linux/amd64. The tables below show representative results; the charts hold the full set.

4 CPUs

Workload	VarMQ	PondV2	Speedup	Memory (VarMQ / Pond)	Allocations (VarMQ / Pond)
`Sleep10ms`, 50k workers, `1u-1Mt`	1,414,893 ops/s	639,325 ops/s	2.2x	150 MB / 365 MB	2.38 M / 8.23 M
`Loop10`, 50k workers, `1u-1Mt`	1,894,515 ops/s	831,844 ops/s	2.3x	117 MB / 241 MB	2.09 M / 7.00 M
`Loop10`, 300k workers, `1Mu-1t`	1,532,306 ops/s	472,661 ops/s	3.2x	165 MB / 333 MB	3.60 M / 8.48 M
`Sleep5Loop5`, 50k workers, `1u-1Mt`	1,581,041 ops/s	689,866 ops/s	2.3x	127 MB / 340 MB	2.22 M / 8.02 M

8 CPUs

Workload	VarMQ	PondV2	Speedup	Memory (VarMQ / Pond)	Allocations (VarMQ / Pond)
`Sleep10ms`, 50k workers, `1u-1Mt`	1,398,627 ops/s	684,068 ops/s	2.0x	148 MB / 368 MB	2.37 M / 8.26 M
`Loop10`, 50k workers, `1u-1Mt`	1,710,220 ops/s	891,813 ops/s	1.9x	117 MB / 241 MB	2.10 M / 7.00 M
`Loop10`, 100k workers, `1Mu-1t`	1,361,620 ops/s	219,070 ops/s	6.2x	152 MB / 345 MB	3.18 M / 8.18 M
`Sleep10ms`, 300k workers, `1Mu-1t`	1,110,899 ops/s	341,835 ops/s	3.3x	202 MB / 482 MB	3.99 M / 9.28 M

24 CPUs

Workload	VarMQ	PondV2	Speedup	Memory (VarMQ / Pond)	Allocations (VarMQ / Pond)
`Sleep10ms`, 50k workers, `1u-1Mt`	871,903 ops/s	357,154 ops/s	2.4x	150 MB / 361 MB	2.38 M / 8.19 M
`Loop10`, 50k workers, `1u-1Mt`	936,156 ops/s	226,394 ops/s	4.1x	116 MB / 240 MB	2.08 M / 7.00 M
`Loop10`, 300k workers, `1Mu-1t`	671,072 ops/s	158,539 ops/s	4.2x	160 MB / 367 MB	3.44 M / 8.72 M
`Sleep5Loop5`, 50k workers, `1Mu-1t`	577,518 ops/s	593,402 ops/s	~1.0x	160 MB / 349 MB	3.22 M / 7.40 M

What the numbers show:

VarMQ won every workload at every CPU count. The typical throughput advantage is roughly 2–2.5x, and the widest gap is 6.2x - the CPU-bound Loop10 workload with one million producers at 8 CPUs.
Memory use stays about 2.4–2.5x lower and allocations about 3.4–3.5x lower regardless of CPU count, which matches the queue chunking and node reuse described above.
The 1Mu-1t pattern many producers, one task each - is where PondV2 degrades most. That is exactly the case VarMQ's design targets: Add never creates a goroutine, and producer wake-ups coalesce into one event-loop signal.
Both libraries slow down at 24 CPUs - the i7-13700 has 16 physical cores and 24 threads, so hyper-threading adds scheduling pressure - but VarMQ keeps its lead.
The one near-parity result is Sleep5Loop5 at 24 CPUs with a million producers, where the two libraries land within a few percent of each other. Everything else is a clear VarMQ win.

As with the microbenchmark, these are derived rates from ns/op on one machine, with the benchmark harness overhead included. Throughput on your hardware and handler shape will differ. The charts let you compare any workload across all three CPU configurations, and the repository documents how to re-run the suite yourself.

Backlogs still need a memory budget

Bounded concurrency protects execution, not queue length. If producers submit work faster than handlers complete it, pending messages accumulate - and every pending job and its payload has to live somewhere.

For a service that may receive sustained bursts, set a queue capacity and decide what the producer should do when Add returns false. Depending on the application, that may mean retrying with backoff, shedding low-priority work, writing to a durable queue, or applying backpressure earlier in the request path.

That's how low per-message overhead turns into predictable memory usage. Without a capacity, even an efficient queue can exhaust memory when the arrival rate stays above the completion rate long enough.

Graceful shutdown keeps concurrency manageable

The same event loop owns shutdown coordination. Cancelling the worker context moves the worker to stopping, waits for in-flight jobs, removes pool workers, broadcasts the final state, and exits the event-loop goroutine.

Pause works differently. It stops admission of new jobs while allowing current handlers to finish. Resume returns the worker to idle and signals the event loop so queued work starts again. These state transitions let an application control traffic without abandoning jobs that are already running.

The useful idea is separation

VarMQ gets high throughput by keeping responsibilities narrow:

Producers enqueue and send a cheap, coalesced notification.
The event loop admits work only while capacity exists.
Pool workers run handlers concurrently and report completion.
Chunked queues and reusable pool nodes keep allocation pressure under control.

The event loop doesn't make handlers faster; it makes dispatch predictable. The pool doesn't make the queue smaller; it keeps active concurrency bounded. Together, they let VarMQ process large message volumes without creating execution resources in proportion to the backlog.

That is how a small Go scheduler can move more than a million trivial messages per second in a microbenchmark while still giving production applications explicit control over concurrency, memory, and shutdown behavior.

Visualize Your Entire GitHub Contribution History as a Stunning 3D Skyline — With Just One Command

Fahim Faisaal — Fri, 24 Jul 2026 14:28:41 +0000

Have you ever wanted to see your full GitHub activity — every contribution across years — as an interactive 3D bar chart? With Vizb, you can turn the entire contribution history of any GitHub user into a beautiful, self-contained 3D “skyline” in a single terminal command.

No complex setup. No external servers. Just one pipeline that fetches the data and generates a fully interactive HTML visualization.

What is Vizb?

Vizb is a lightweight CLI tool that transforms tabular data (CSV, JSON, or benchmark output) into interactive charts. It supports multi-dimensional grouping, so dates can be split into Year / Month / Day axes — perfect for turning contribution calendars into 3D skyline charts.

Quick Install

Linux / MacOS

curl -fsSL https://vizb.goptics.org/install.sh | bash

Windows Powershell

irm https://vizb.goptics.org/install.ps1 | iex

The Magic Command

curl -s "https://github-contributions-api.jogruber.de/v4/<username>" \
  | vizb bar \
      --group date \
      --group-pattern '[z{Year}-y{Month}-x{Date}]' \
      --json-path '.contributions' \
      --select 'count{Contributions}' \
      --stat \
      --output index.html

Example: Linus Torvalds

Let’s use the creator of Linux and Git himself. Torvalds has made more than 37,000 contributions across roughly 16 years on GitHub.

Replace <username> with torvalds:

Linux / MacOS

curl -s "https://github-contributions-api.jogruber.de/v4/torvalds" \
  | vizb bar \
      --group date \
      --group-pattern '[z{Year}-y{Month}-x{Date}]' \
      --json-path '.contributions' \
      --select 'count{Contributions}' \
      --stat \
      --output index.html

Windows Powershell

(iwr "https://github-contributions-api.jogruber.de/v4/torvalds").Content `
  | vizb bar `
      --group date `
      --group-pattern '[z{Year}-y{Month}-x{Date}]' `
      --json-path '.contributions' `
      --select 'count{Contributions}' `
      --stat `
      --output index.html

Open the generated index.html in any browser. You’ll get an interactive 3D skyline like below:

Each year becomes a new layer (z-axis)
Months form the depth (y-axis)
Individual days become the bars (x-axis)
Bar height = contribution count that day

You can rotate, zoom, hover for exact numbers, toggle the statistics panel, and even export the view.

Why This Works So Well

The public GitHub Contributions API returns every day’s count in clean JSON
Vizb’s smart grouping pattern [z{Year}-y{Month}-x{Date}] automatically builds the 3D axes
The entire visualization is a single self-contained HTML file — no dependencies, no complexity

Try it with your own username (or any public profile). In under 10 seconds you’ll have a personal 3D contribution skyline you can share, embed, or just admire.

One command. Your entire GitHub story — visualized.

How to Verify Your Git Commits and Tags with GPG: A Step-by-Step Guide

Fahim Faisaal — Mon, 29 Dec 2025 05:37:50 +0000

In the world of open source and collaborative development, identity is everything. When you see a commit from torvalds on the Linux kernel, how do you know it's actually Linus Torvalds and not an impersonator?

The answer is GPG signing.

By signing your commits with a GPG (GNU Privacy Guard) key, you cryptographically verify that the code came from you and hasn't been altered. GitHub (and GitLab) rewards this with a shiny green "Verified" badge.

Here is a straightforward guide to setting this up.

1. Install GPG

First, you need the GPG tool installed on your machine.

macOS (Homebrew):

brew install gnupg

Linux (Debian/Ubuntu/Pop!_OS):

sudo apt install gnupg

Windows:
Download and install Gpg4win.

2. Generate a GPG Key

Run the following command to generate a new key pair.

gpg --batch --passphrase "" --quick-gen-key "Your Name <your_email@example.com>" rsa4096 default never

3. Get Your Key ID

Once generated, list your keys to find the Key ID:

gpg --list-secret-keys --keyid-format LONG

Output example:

sec   rsa4096/3AA5C34371567BD2 2024-01-01 [SC]
      ...
uid                 [ultimate] John Doe <john@example.com>

In this example, 3AA5C34371567BD2 is your Key ID.

4. Tell Git About Your Key

Now configure Git to use this key for signing.

Set the key:

git config --global user.signingkey 3AA5C34371567BD2

(Replace 3AA5C34371567BD2 with your actual Key ID)

Enable automatic signing (Optional but recommended):
This ensures every commit you make is signed by default.

git config --global commit.gpgsign true

5. Add Your Public Key to GitHub

Git knows about your key, but GitHub doesn't yet.

1. Export your public key:

gpg --armor --export 3AA5C34371567BD2

2. Copy the output:
Copy the entire block (including -----BEGIN PGP PUBLIC KEY BLOCK----- and -----END...).

3. Add to GitHub:

Go to Settings > SSH and GPG keys.
Click New GPG key.
Paste your key and save.

6. Verify It Works

Make a commit in any repository:

git commit -m "My first signed commit"

Push it to GitHub. You should now see the verified badge next to your commit in the history!

Troubleshooting

If you see "Unverified":

Check that the email in git config user.email matches the email in your GPG key.
Ensure that exact email is added and verified in your GitHub account settings.

Try vzib to visualize the go ugly benchmarks to interactive charts

Fahim Faisaal — Tue, 24 Jun 2025 15:19:55 +0000

Fahim Faisaal

Jun 24 '25

Vizb: Beautiful Go Benchmark Visualization Made Simple

#showdev #webdev #go #cli

2 min read

Vizb: Beautiful Go Benchmark Visualization Made Simple

Fahim Faisaal — Tue, 24 Jun 2025 14:32:37 +0000

Benchmarking in Go is straightforward, but visualizing those results? That's where things get messy. When you're comparing different libraries or implementations across various workloads, staring at raw benchmark numbers just doesn't cut it.

I found myself in this exact situation while working with my message queue library, varmq. I needed to compare performance across different scenarios, but existing visualization tools either didn't exist or didn't fit my needs.

So I built Vizb – a CLI tool that transforms Go benchmark results into beautiful, interactive HTML charts with a single command.

The Magic is in the Simplicity

go test -bench . -run=^$ -json | vizb -o output.html

That's it. One command, and you get:

Interactive HTML charts showing execution time, memory usage, and allocations
Responsive design that works on any device
Export capability – download charts as PNG images directly from your browser
Smart grouping that automatically organizes your benchmarks by workload and subject

Key Features That Actually Matter

Multiple Metrics in One View: Compare execution time, memory usage, and allocation counts side by side.

Customizable Units: Display metrics in your preferred units (ns/μs/ms/s for time, B/KB/MB/GB for memory).

Flexible Input: Works with files or piped input – whatever fits your workflow.

JSON Output: Need the data in a different format? Use --format json to get structured data instead of HTML.

Real-World Example

Here's how Vizb automatically groups your benchmarks. If you have:

BenchmarkEncoder/small/json
BenchmarkEncoder/small/msgpack
BenchmarkEncoder/large/json
BenchmarkEncoder/large/msgpack

Vizb creates charts where:

Test: Encoder
Workloads: small, large
Subjects: json, msgpack

The result? Clean, organized charts that make performance differences immediately obvious.

Why It Matters

When you're optimizing code or choosing between libraries, you need insights, not raw numbers. Vizb turns your benchmark data into actionable visualizations that help you make better decisions faster.

Check it out: github.com/goptics/vizb
Check out the live chart benchmarks: varmq-benchmark

Install it with: go install github.com/goptics/vizb

Happy benchmarking! 🚀

sync.Cond in Go: Efficient Goroutine Signaling Without Channels

Fahim Faisaal — Wed, 04 Jun 2025 12:50:51 +0000

Concurrency in Go often brings us to channels, but there’s another synchronization primitive that may be exactly what you need in some scenarios: sync.Cond. If you’ve ever wondered why you’d reach for a sync.Cond instead of using channels alone, this article is for you. By the end, you’ll see a simple custom implementation, understand how the real sync.Cond works under the hood, and know when to choose it in your own projects.

Why Use `sync.Cond`?

Most Go developers instinctively reach for channels to coordinate goroutines: sending values, waiting for results, and so on. However, channels also carry data. What if all you need is a simple “wake-up” signal, without any payload? That’s exactly where sync.Cond shines. It’s a lightweight way to block one or more goroutines until a condition becomes true, without transferring actual data.

Think of it like a broadcast system: goroutines can call Wait() and suspend until somebody calls Signal() (wake a single waiter) or Broadcast() (wake all waiters). Underneath, sync.Cond doesn’t allocate a channel for each goroutine; instead, it maintains a small linked list of waiting goroutines, making it more memory-efficient when you just need signaling.

To illustrate, let’s build our own “poor man’s” condition variable using channels. Once you see the analogy, switching to sync.Cond becomes straightforward.

Building a Custom `Cond` with Channels

Here’s a minimal struct that mimics sync.Cond by using a slice of channels and a mutex:

type MyCond struct {
    chs []chan struct{}
    mu  sync.Mutex
}

chs holds one channel per waiting goroutine.
mu ensures that appending or removing from chs is safe.

Below are three methods Wait(), Signal(), and Broadcast() that emulate the core behavior of sync.Cond.

func (c *MyCond) Wait() {
    c.mu.Lock()
    ch := make(chan struct{})
    c.chs = append(c.chs, ch)
    c.mu.Unlock()

    // wait for a signal
    <-ch
}

func (c *MyCond) Signal() {
    c.mu.Lock()
    defer c.mu.Unlock()

    if len(c.chs) == 0 {
        return
    }

    // pick the first channel and send signal
    ch := c.chs[0]
    ch <- struct{}{}
    close(ch)

    // remove that channel from the slice
    c.chs = c.chs[1:]
}

func (c *MyCond) Broadcast() {
    c.mu.Lock()
    defer c.mu.Unlock()

    for _, ch := range c.chs {
        ch <- struct{}{}
        close(ch)
    }

    // reset the slice so no stale channels remain
    c.chs = make([]chan struct{}, 0)
}

What’s happening here?

Wait()

Lock the mutex.
Create a new “signal” channel (ch).
Append it to c.chs.
Unlock, then block on <-ch.
When someone calls Signal() or Broadcast(), that channel is closed (and a value is sent), letting this goroutine resume.

Signal()

Lock the mutex.
If there’s at least one waiting channel, pick the first.
Send a dummy struct{}{} onto it, then close(ch) so that any extra <-ch receives don’t hang.
Remove that channel from the slice.

Broadcast()

Lock the mutex.
Loop over every waiting channel: send a dummy signal and close it.
Reset the slice to empty, so future waiters start fresh.

This simple approach shows how condition variables signal “ready to go” without passing actual payloads just signals.

Testing Our Custom `MyCond`

To see MyCond in action, imagine spawning multiple worker goroutines that all wait for a signal. Then, from another goroutine, send one signal at a time. Finally, switch to broadcasting to wake everyone at once.

func main() {
    cond := &MyCond{}
    wg := sync.WaitGroup{}
    tasks := 5
    wg.Add(tasks) // add tasks count to the wait group

    for id := range tasks {
        // create separate go routine for each task
        go func() {
            defer wg.Done()

            fmt.Println("Waiting", id)
            cond.Wait()
            fmt.Println("Done", id)
        }()
    }

    go func() {
        for range tasks {
            time.Sleep(1 * time.Second)
            cond.Signal() // send signal to one routine in every 1 second
        }
    }()

    // wait for all routines to finish
    wg.Wait()
}

The output

When you run that, each goroutine hangs on cond.Wait(). Every second, Signal() wakes exactly one goroutine, until all 5 finish.

Switching to Broadcast

Instead of signaling one by one, you can broadcast after a delay to wake all of them at once:

// just change the signal to broadcast go routine
go func() {
    // - for range tasks {
    // -    time.Sleep(1 * time.Second)
    // -    cond.Signal() // send signal to one routine in every 1 second
    // - }

    time.Sleep(2 * time.Second)
    cond.Broadcast() // send signal to all routines at once after 2 seconds
}()

The output

With this modification, all 5 goroutines sleep in cond.Wait(). After two seconds, a single Broadcast() wakes everybody, and you’ll see all “Done” messages in rapid succession.

Replacing `MyCond` with `sync.Cond`

Once you’ve verified the custom behavior, swapping in the real sync.Cond is straightforward. Anywhere you wrote:

//  cond := &MyCond{}
cond := sync.NewCond(&sync.Mutex{})

// cond.Wait()
cond.L.Lock()
cond.Wait()
cond.L.Unlock()

You’ll get the same “Waiting … Done” behavior as before, but now backed by the official, optimized implementation.

Under the hood, sync.Cond doesn’t spin up a channel per waiter. Instead it uses an internal notifyList a small doubly linked list of waiting goroutines ) and uses low‐level runtime primitives to park and wake goroutines. Each call to Wait() enqueues the goroutine on that list. Signal() removes one link and wakes its goroutine; Broadcast() traverses the whole list and wakes every waiter. Memory-wise, this is much cheaper than allocating a channel per waiter, especially if you have hundreds or thousands of goroutines occasionally blocking on the same condition.

For a deeper dive, check out the source code for sync.Cond.

When to Choose `sync.Cond` Over Channels

Here are a few scenarios where sync.Cond makes sense:

Simple Signaling
If goroutines only need a “go now” notification no data passed sync.Cond provides a clearer, more intent-expressive API than channels filled with dummy values.
Broadcast Semantics
Channels lack a built-in “wake everyone” primitive. You could loop over a list of channels, but managing that list is extra boilerplate. sync.Cond.Broadcast() does exactly what it says: wake all waiters at once.
Lower Memory Overhead
Each Go channel has internal buffers, mutexes, and so on. If you merely need a “signal,” channels allocate more than necessary. A sync.Cond maintains a minimal linked list of waiters, which is especially noticeable if you have thousands of goroutines waiting occasionally.
Condition-Based Waiting
Often you combine sync.Cond with a separate shared value. For example:

   mu.Lock()
   for !conditionMet {
       cond.Wait()
   }
   // now the condition is true; proceed
   mu.Unlock()

This “wait in a loop” pattern is common in concurrent structures like pools, queues, or buffered buffers. Channels alone can’t you’d have to juggle extra variables or use select, which can get messy.

In short, if your goroutines coordinate purely on a boolean or numerical condition and you want to wake either one waiter or all waiters sync.Cond shines. If you need to send actual data, channels remain the more idiomatic choice.

Open-Source Promotions

VarMQ - A Simplest Storage-Agnostic and Zero-dep Message Queue for Your Concurrent Go Program
Vizb - An interactive go benchmarks visualizer

VarMQ Tuning Worker Pool

Fahim Faisaal — Sun, 18 May 2025 03:34:15 +0000

I've created a "tune API" for the next version of VarMQ. Essentially, "Tune" allows you to increase or decrease the size of the worker/thread pool at runtime.

For example, when the load on your server is high, you'll need to process more concurrent jobs. Conversely, when the load is low, you don't need as many workers, because workers consume resources.

Therefore, based on your custom logic, you can dynamically change the worker pool size using this tune API.

In this video, I've enqueued 1000 jobs into VarMQ, and I've set the initial worker pool size to 10 (the concurrency value).

Every second, using the tune API, I'm increasing the worker pool size by 10 until it reaches 100.

Once it reaches a size of 100, then I start removing 10 workers at a time from the pool.

This way, I'm decreasing and then increasing the worker pool size.

Cool, right?

VarMQ primarily uses an Event-Loop internally to handle this concurrency.

This event loop checks if there are any pending jobs in the queue and if any workers are available in the worker pool. If there are, it distributes jobs to all available workers and then goes back into sleep mode.

When a worker becomes free, it then tells the event loop, "Hey, I'm free now; if you have any jobs, you can give them to me."

The event loop then checks again if there are any pending jobs in the queue. If there are, it continues to distribute them to the workers.

This is VarMQ's concurrency model.

A Story of Building a Storage-Agnostic Message Queue

Fahim Faisaal — Fri, 09 May 2025 13:41:34 +0000

A year ago, I was knee-deep in Golang, trying to build a simple concurrent queue as a learning project. Coming from a Node.js background, where I’d spent years working with tools like BullMQ and RabbitMQ, Go’s concurrency model felt like a puzzle. My first attempt a minimal queue with round-robin channel selection was, well, buggy. Let’s just say it worked until it didn’t.

But that’s how learning goes, right?

The Spark of an Idea

In my professional work, I’ve used tools like BullMQ and RabbitMQ for event-driven solutions, and p-queue and p-limit for handling concurrency. Naturally, I began wondering if there were similar tools in Go. I found packages like asynq, ants, and various worker pools solid, battle-tested options. But suddenly, a thought struck me: what if I built something different? A package with zero dependencies, high concurrency control, and designed as a message queue rather than submitting functions?

With that spark, I started building my first Go package, released it, and named it Gocq (Go Concurrent Queue). The core API was straightforward, as you can see here:

// Create a queue with 2 concurrent workers
queue := gocq.NewQueue(2, func(data int) int {
    time.Sleep(500 * time.Millisecond)
    return data * 2
})
defer queue.Close()

// Add a single job
result := <-queue.Add(5)
fmt.Println(result) // Output: 10

// Add multiple jobs
for result := range queue.AddAll(1, 2, 3, 4, 5) {
    fmt.Println(result) // Output: 2, 4, 6, 8, 10 (unordered)
}

From the excitement, I posted it on Reddit. To my surprise, it got traction upvotes, comments, and appreciations. Here’s the fun part: coming from the Node.js ecosystem, I totally messed up Go’s package system at first.

Within a week, I released the next version with a few major changes and shared it on Reddit again. More feedback rolled in, and one person asked for "persistence abstractions support".

The Missing Piece

That hit home, I’d felt this gap before, Persistence. It’s the backbone of any reliable queue system. Without persistence, the package wouldn’t be complete. But then a question is: if I add persistence, would I have to tie it to a specific tool like Redis or another database?

I didn’t want to lock users into Redis, SQLite, or any specific storage. What if the queue could adapt to any database?

So I tore gocq apart.

I rewrote most of it, splitting the core into two parts: a worker pool and a queue interface. The worker would pull jobs from the queue without caring where those jobs lived.

The result? VarMQ, a queue system that doesn’t care if your storage is Redis, SQLite, or even in-memory.

How It Works Now

Imagine you need a simple, in-memory queue:

w := varmq.NewWorker(func(j varmq.Job[any])  {
   // your heavy work
}, 2)
q := w.BindQueue() // Done. No setup, no dependencies.

if you want persistence, just plug in an adapter. Let’s say SQLite:

import "github.com/goptics/sqliteq"

db := sqliteq.New("test.db")
pq, _ := db.NewQueue("orders")
q := w.WithPersistentQueue(pq) // Now your jobs survive restarts.

Or Redis for distributed workloads:

import "github.com/goptics/redisq"

rdb := redisq.New("redis://localhost:6379")
pq := rdb.NewDistributedQueue("transactions")
q := w.WithDistributedQueue(pq) // Scale across servers.

The magic? The worker doesn’t know—or care—what’s behind the queue. It just processes jobs.

Lessons from the Trenches

Building this taught me two big things:

Simplicity is hard.
Feedback is gold.

Why This Matters

Message queues are everywhere—order processing, notifications, data pipelines. But not every project needs Redis. Sometimes you just want SQLite for simplicity, or to switch databases later without rewriting code.

With Varmq, you’re not boxed in. Need persistence? Add it. Need scale? Swap adapters. It’s like LEGO for queues.

What’s Next?

The next step is to integrate the PostgreSQL adapter and a monitoring system.

If you’re curious, check out Varmq on GitHub. Feel free to share your thoughts and opinions in the comments below, and let's make this Better together.

DEV Community: Fahim Faisaal

How VarMQ Processes Millions of Messages with an Event Loop and Worker Pool

The processing path

The event loop dispatches; it does not execute

The pool bounds concurrency

Where the memory savings come from

Chunked queue storage (Default standard Queue)

Reused worker nodes

Small coordination state

VarMQ under load: a head-to-head benchmark against PondV2

4 CPUs

8 CPUs

24 CPUs

Backlogs still need a memory budget

Graceful shutdown keeps concurrency manageable

The useful idea is separation

Visualize Your Entire GitHub Contribution History as a Stunning 3D Skyline — With Just One Command

What is Vizb?

Quick Install

Linux / MacOS

Windows Powershell

The Magic Command

Linux / MacOS

Windows Powershell

Why This Works So Well

How to Verify Your Git Commits and Tags with GPG: A Step-by-Step Guide

1. Install GPG

2. Generate a GPG Key

3. Get Your Key ID

4. Tell Git About Your Key

5. Add Your Public Key to GitHub

6. Verify It Works

Troubleshooting

Try vzib to visualize the go ugly benchmarks to interactive charts

Vizb: Beautiful Go Benchmark Visualization Made Simple

Vizb: Beautiful Go Benchmark Visualization Made Simple

The Magic is in the Simplicity

Key Features That Actually Matter

Real-World Example

Why It Matters

sync.Cond in Go: Efficient Goroutine Signaling Without Channels

Why Use sync.Cond?

Building a Custom Cond with Channels

What’s happening here?

Testing Our Custom MyCond

The output

Switching to Broadcast

The output

Replacing MyCond with sync.Cond

When to Choose sync.Cond Over Channels

Open-Source Promotions

VarMQ Tuning Worker Pool

A Story of Building a Storage-Agnostic Message Queue

The Spark of an Idea

The Missing Piece

How It Works Now

Lessons from the Trenches

Why This Matters

What’s Next?

Why Use `sync.Cond`?

Building a Custom `Cond` with Channels

Testing Our Custom `MyCond`

Replacing `MyCond` with `sync.Cond`

When to Choose `sync.Cond` Over Channels