Feroz

Posted on Aug 13

Concurrent Resource Scheduler v1.2.3: Sharded Priority Heaps Under Concurrent Load

#go #concurrency #performance #opensource

GitHub: https://github.com/phero20/concurrent-resource-scheduler

Go documentation: https://pkg.go.dev/github.com/phero20/concurrent-resource-scheduler

Concurrent Resource Scheduler v1.2.3: Sharded Priority Heaps Under Concurrent Load

A resource scheduler sounds simple until many goroutines start competing for a small number of resources.

At that point, the problem stops being:

"How do I pick the next item?"

and becomes:

"How do I pick the right item while thousands of goroutines are acquiring, releasing, updating, and observing resources concurrently?"

That is the problem I built Concurrent Resource Scheduler (CRS) to solve.

CRS is a domain-agnostic Go library for managing reusable resources with:

sharded priority heaps
concurrent-safe resource lookup
configurable acquisition strategies
shared and exclusive acquisition
affinity routing
resource lifecycle management
cooldown extensions
asynchronous events
optional Prometheus telemetry

The current release is v1.2.3.

This post is a release-focused technical deep dive into the architecture, the reasoning behind it, and the actual measurements from the current release.

First: What Problem Are We Actually Solving?

Imagine a service with a pool of reusable resources.

Those resources could be:

API keys
database connections
GPU workers
proxy endpoints
backend workers
LLM providers
service instances
connection pools
rate-limited accounts

Now imagine:

             10,000 concurrent requests
                       │
                       ▼
              ┌─────────────────┐
              │    Scheduler    │
              └────────┬────────┘
                       │
          ┌────────────┼────────────┐
          ▼            ▼            ▼
       Resource A   Resource B   Resource C

The scheduler needs to answer several questions at once:

Which resource should this request receive?
Is that resource active?
Is it already exclusively acquired?
Can it be shared?
Which resource has the best priority?
Should affinity influence the selection?
Has a resource entered cooldown?
What happens when it is released?
What happens when its priority changes?
How do we observe all of this without putting telemetry directly on the hot path?

A simple slice and mutex can answer some of these questions.

The difficult part is doing all of them concurrently and predictably.

The Obvious Design: One Slice, One Mutex

The first design most people naturally reach for is something like:

type Scheduler struct {
    mu        sync.Mutex
    resources []*Resource
}

Then acquisition becomes:

func (s *Scheduler) Acquire() (*Resource, error) {
    s.mu.Lock()
    defer s.mu.Unlock()

    // Scan resources.
    // Find the best available resource.
    // Mark it acquired.
    // Return it.

    return resource, nil
}

For a small pool, this is completely reasonable.

The problem appears when concurrency and resource count grow.

Suppose:

Resources:           10,000
Concurrent goroutines: 10,000

Every operation now enters the same synchronization boundary:

                 GLOBAL MUTEX
                      │
       ┌──────────────┼──────────────┐
       ▼              ▼              ▼
   goroutine       goroutine      goroutine
       │              │              │
       └──────────────┼──────────────┘
                      ▼
                    WAIT

Even if the actual operation only concerns a small part of the resource pool, the lock protects everything.

There is another problem.

If resources are stored in a slice and the scheduler has to find the best candidate, acquisition can require a linear scan:

O(N)

Now the same global lock is protecting an operation whose amount of work grows with the number of resources.

That is the combination I wanted to avoid.

The Main Architectural Decision

The core idea behind CRS is:

Partition the active resource pool into independently synchronized shards.

Instead of:

                    ONE LOCK
                       │
                 ONE BIG POOL

CRS uses:

                         Scheduler
                             │
              ┌──────────────┼──────────────┐
              ▼              ▼              ▼
           Shard 0        Shard 1        Shard N
              │              │              │
            Heap           Heap           Heap
              │              │              │
            Mutex          Mutex          Mutex

Each shard owns a priority heap.

Each heap has its own synchronization boundary.

That gives us a much more useful concurrency model:

goroutine A ──► shard 0 ──► heap 0
goroutine B ──► shard 1 ──► heap 1
goroutine C ──► shard 2 ──► heap 2

The goal is not to claim that sharding magically eliminates contention.

It doesn't.

The goal is to reduce the amount of unrelated work competing for the same lock.

That distinction matters.

Why a Priority Heap?

Sharding solves one problem:

How do we reduce contention?

We still need to solve another:

How do we efficiently maintain resource priority?

A slice makes this straightforward but potentially expensive:

resources:
[ A, B, C, D, E, F, G, ... ]

Find minimum priority:
scan everything

A priority heap gives us an ordered structure where the highest-priority candidate can be accessed from the heap root.

Conceptually:

             best resource
                  │
                  ▼
               [ 10 ]
              /      \
           [ 20 ]   [ 30 ]
           /   \
        [40]   [50]

So CRS combines the two ideas:

                 RESOURCE POOL
                       │
             ┌─────────┴─────────┐
             ▼                   ▼
          Sharding            Priority
             │                   │
             ▼                   ▼
        Lower lock           Heap ordering
         contention          / fast candidate

This is the central design tradeoff of the project.

The Scheduler Is More Than a Heap

One thing I learned while building CRS is that the heap is actually only one part of the problem.

The scheduler has several independent concerns:

                         CRS
                          │
       ┌──────────────────┼──────────────────┐
       │                  │                  │
       ▼                  ▼                  ▼
   Acquisition          State             Lookup
    Strategy          Lifecycle             Map
       │                  │                  │
       ▼                  ▼                  ▼
    Adaptive          Active/Inactive       O(1)
    Weighted          Acquire/Release       lookup
    Round Robin       Include/Exclude
    Affinity

Then there are extensions:

                Event Dispatcher
                       │
             ┌─────────┴─────────┐
             ▼                   ▼
          Cooldown           Telemetry
                                 │
                                 ▼
                             Prometheus

Keeping these responsibilities separated makes the implementation easier to reason about and lets the core remain domain-agnostic.

Resource Lookup Is a Separate Problem

Priority ordering tells us:

"Which resource should be considered first?"

But APIs also need direct resource lookup.

For example:

resource, err := sched.Get("worker-42")

We don't want to search every heap for that.

CRS therefore maintains a lookup structure alongside the active heap shards.

Conceptually:

                resource ID
                     │
                     ▼
              ┌─────────────┐
              │ Lookup Map  │
              └──────┬──────┘
                     │
                     ▼
                  node
                     │
             ┌───────┴───────┐
             ▼               ▼
          resource          shard

This gives the scheduler a useful separation:

heap → candidate ordering
lookup map → direct identity lookup

These are different access patterns, so they shouldn't be forced into the same data structure.

Acquisition Strategies Are Pluggable

Another design decision was not to hard-code one definition of "best resource."

Different systems want different behavior.

For example:

Adaptive

Choose resources based on observed scheduling state.

Weighted

Some resources should receive more traffic than others.

Round Robin

Distribute acquisitions sequentially.

Affinity

Prefer a resource or shard associated with some identifier.

That means the scheduler's core doesn't need to know why an application considers one resource better than another.

The application provides the policy.

The scheduler provides the concurrency-safe machinery around that policy.

Shared vs Exclusive Acquisition

This distinction is especially important.

A resource might support:

Shared

where multiple callers can use it simultaneously.

Or:

Exclusive

where only one caller can own it at a time.

These are not just two API names.

They produce different synchronization behavior.

For example, under shared acquisition, the scheduler can inspect the best candidate without removing it from the heap.

Conceptually:

Shared:

heap
  │
  ▼
peek candidate
  │
  ▼
use resource

Whereas exclusive acquisition may need to remove the candidate from the active scheduling position:

Exclusive:

heap
  │
  ▼
pop candidate
  │
  ▼
exclusive owner

This distinction is also reflected in the documentation and complexity model.

Resource Lifecycle

Resources don't simply exist or disappear.

They move through states.

A simplified lifecycle looks like:

              ┌──────────────┐
              │    ACTIVE    │
              └──────┬───────┘
                     │
            acquire / exclude
                     │
                     ▼
              ┌──────────────┐
              │   INACTIVE   │
              └──────┬───────┘
                     │
               include / release
                     │
                     ▼
              ┌──────────────┐
              │    ACTIVE    │
              └──────────────┘

There are also terminal operations such as removal and shutdown.

The important part is that the scheduler must maintain consistent state while these operations happen concurrently.

That's why lifecycle management is part of the scheduler design rather than an afterthought.

Cooldown Is an Extension, Not Core Scheduling Logic

Cooldown is a good example of why the architecture is modular.

Suppose a resource fails or needs temporary exclusion:

Resource
   │
   ▼
Cooldown
   │
   ├── removed from active scheduling
   │
   └── restored after duration

The core scheduler shouldn't need to understand every possible reason a resource temporarily leaves the pool.

The cooldown extension coordinates with the scheduler's lifecycle controller.

This was also one of the areas tightened in v1.2.3: the cooldown documentation and example now use the correct LifecycleController wrapper pattern instead of trying to initialize the cooldown manager before the scheduler exists.

That ordering matters because the scheduler is the object that ultimately owns the resource lifecycle.

Why v1.2.3 Exists

v1.2.3 is not a giant feature release.

It is a correctness, consistency, and release-hardening release.

The main fixes include:

Cooldown initialization documentation

The cooldown extension's initialization example was corrected to use the proper lifecycle-controller wrapper pattern.

This avoids the circular initialization problem of trying to construct a component that needs the scheduler before the scheduler has been created.

Cooldown example timing

The cooldown example was adjusted so it does not immediately race an asynchronous exclusion event.

The example now demonstrates the intended lifecycle deterministically.

Benchmark package boundary

The scheduler benchmark file was moved to the black-box package boundary:

package scheduler_test

because it only relies on exported APIs.

That makes the benchmark follow the same testing convention as the rest of the package.

Documentation consistency

README and API documentation were updated to match the implementation, including error documentation and complexity behavior.

Prometheus extension

The optional Prometheus module is also aligned to:

v1.2.3

while remaining a separate nested Go module.

The Numbers: Actual v1.2.3 Microbenchmarks

Architecture diagrams are useful.

Numbers are better.

I ran the current benchmark suite with:

go test -run="^$" -bench="." -benchmem ./...

The measurements below were collected locally on:

OS:   Windows
Arch: amd64
CPU:  AMD Ryzen 5 6600H with Radeon Graphics

These are local measurements, not universal performance guarantees.

Acquire Strategy Benchmarks

The current release produced the following verified measurements:

Strategy	Select/GetShard Cost
`ConsistentHashRing.GetShard`	7.2 ns
`WeightedStrategy.Select`	19.1 ns
`AdaptiveStrategy.Select`	27.0 ns

The strategy selection costs are all in the low-nanosecond range on this machine. The scheduler benchmarks below separately report allocation counts for the larger operations.

Scheduler Hot-Path Benchmarks

The scheduler benchmarks are more representative of actual scheduler operations.

Operation	HeapCount=1	HeapCount=8	HeapCount=32	Allocs/op
`Add`	591.9 ns	836.2 ns	718.5 ns	3
`Update`	280.6 ns	228.9 ns	200.4 ns	1
`BatchAdd` (1,000 resources)	308.1 µs	351.3 µs	331.1 µs	~1,100
`Acquire` (Shared, Sequential)	12.36 ns	11.52 ns	11.23 ns	0
`Acquire` (Shared, Parallel)	63.14 ns	16.80 ns	17.99 ns	0
`Acquire` + `Release` (Exclusive)	241.2 ns	249.2 ns	208.8 ns	0

There is also a parallel acquisition benchmark:

Benchmark	Heap count	ns/op
AcquireSharedParallel	1	63.14
AcquireSharedParallel	8	16.80
AcquireSharedParallel	32	17.99

The parallel benchmark is particularly useful when thinking about the sharding architecture.

On this machine and workload, moving from a single heap to multiple heaps reduced the measured AcquireSharedParallel time substantially:

HeapCount=1   63.14 ns/op
HeapCount=8   16.80 ns/op
HeapCount=32  17.99 ns/op

That is one workload on one machine, not a universal scaling law.

But it is exactly the kind of behavior the architecture is intended to make measurable.

Why We Benchmark Multiple Heap Counts

It would be easy to benchmark only:

HeapCount = 1

and stop there.

That wouldn't tell us much about the reason for sharding.

Instead, the scheduler benchmarks compare:

HeapCount = 1
HeapCount = 8
HeapCount = 32

This gives us a way to observe how the synchronization structure behaves as the number of shards changes.

The expected tradeoff is not:

"More shards are always faster."

It is:

"The right number of shards depends on workload, resource count, contention, and scheduling strategy."

More shards also mean more structures to manage.

So sharding is a tuning dimension, not a magic constant.

Batch Operations Have a Different Cost Profile

The benchmark suite also measures batch insertion:

Benchmark	Heap count	ns/op	B/op	allocs/op
BatchAdd	1	308.1 µs	283,715	1,042
BatchAdd	8	351.3 µs	283,530	1,095
BatchAdd	32	331.1 µs	282,314	1,223

This is important because it prevents cherry-picking only the fastest numbers.

AcquireShared is extremely cheap in the measured workload.

BatchAdd is much more expensive.

That's expected.

Adding a large batch involves substantially more work than peeking at an already-populated scheduling structure.

A useful benchmark suite should expose those differences instead of presenting one "magic" performance number.

Validation Is More Important Than a Single Benchmark

A concurrency library can produce beautiful microbenchmarks and still be broken.

That's why v1.2.3 was validated with multiple layers.

Normal tests

go test ./...

Result:

PASS

All tested packages completed successfully.

Race detector

go test -race ./...

Result:

PASS

No data races were reported.

Static analysis

go vet ./...

Result:

PASS

Formatting

gofmt -l .

Result:

(no output)

The working tree is therefore clean according to gofmt.

Why the Race Detector Matters

For a concurrency-heavy library, this command is not optional validation theater:

go test -race ./...

A normal test can pass while a race exists.

The race detector instruments memory accesses and can expose unsafe concurrent access that ordinary functional assertions don't catch.

It does not prove that a concurrent system is mathematically bug-free.

But it is one of the most important tools available for catching a class of real concurrency bugs.

For v1.2.3, the race-enabled test suite completed without reported races.

The Load-Test Question

Microbenchmarks answer:

"How quickly does this small operation execute?"

A load test asks a different question:

"What happens when the whole scheduler is placed inside a realistic concurrent workload?"

For CRS, the dedicated load-test harness models things such as:

concurrent workers
resource pools
backend processing delay
request duration
failures
cancellation
resource utilization
acquisition latency
total latency
throughput
burst traffic
cooldown behavior

The distinction is important.

A benchmark like:

25 ns/op

does not mean a real request takes 25 ns.

A real request also includes:

scheduler
   +
application work
   +
network
   +
backend
   +
serialization
   +
other system costs

So I treat microbenchmarks and load tests as different measurements.

10,000 Concurrent Workers

One of the release validation workloads uses:

10,000 concurrent workers

This is useful because it puts the scheduler under a very different kind of pressure from a small benchmark.

The important question isn't:

"Can Go start 10,000 goroutines?"

Of course it can.

The question is:

"Can the resource-management layer maintain correct lifecycle and acquisition behavior while thousands of workers continuously compete for a small pool?"

That is where the sharded architecture, lookup synchronization, acquisition strategies, and lifecycle rules interact.

For the load-test results themselves, I treat the numbers as workload-specific measurements rather than claiming they represent every deployment.

The load-test harness also separates scheduler-side failures from simulated backend failures, because those are operationally different failure modes.

A Failure Is Not Necessarily a Scheduler Failure

This distinction is easy to lose in a load test.

Imagine:

10,000 workers
       │
       ▼
    Scheduler
       │
       ▼
    Resource
       │
       ▼
 Backend request
       │
       ▼
    FAILURE

The backend failed.

That doesn't mean:

scheduler failed

Similarly:

scheduler acquire failure

is not the same thing as:

backend failure

A serious load test should keep these categories separate.

That's why the harness tracks acquisition behavior, backend behavior, release behavior, and timeout/cancellation behavior independently.

Why This Matters for Resource Pools

Consider four backend resources:

backend-1
backend-2
backend-3
backend-4

and:

10,000 concurrent workers

You cannot turn four resources into 10,000 concurrent backend operations just because there are 10,000 goroutines.

If the acquisition policy is exclusive, the resource pool remains the bottleneck:

10,000 workers
      │
      ▼
 ┌───────────┐
 │ Scheduler │
 └─────┬─────┘
       │
       ▼
  4 resources

That is not a failure.

That's exactly what resource scheduling is supposed to enforce.

Domain-Agnostic by Design

The scheduler doesn't know whether a resource is:

type APIKey struct {
    ID string
}

or:

type GPUWorker struct {
    ID string
}

or:

type DatabaseReplica struct {
    ID string
}

The application provides the resource type and the functions needed to identify and compare it.

For example:

compare := func(a, b *Worker) int {
    if a.Priority < b.Priority {
        return -1
    }

    if a.Priority > b.Priority {
        return 1
    }

    return 0
}

keyFunc := func(w *Worker) string {
    return w.ID
}

The scheduler doesn't need to understand what Priority means.

It only needs a consistent comparison function.

That's what makes the library reusable across domains.

A Minimal v1.2.3 Example

The basic flow remains intentionally small:

package main

import (
    "fmt"
    "log"

    "github.com/phero20/concurrent-resource-scheduler/config"
    "github.com/phero20/concurrent-resource-scheduler/scheduler"
)

type Worker struct {
    ID       string
    Priority int
}

func main() {
    compare := func(a, b *Worker) int {
        if a.Priority < b.Priority {
            return -1
        }

        if a.Priority > b.Priority {
            return 1
        }

        return 0
    }

    keyFunc := func(w *Worker) string {
        return w.ID
    }

    cfg := config.Config[*Worker, string]{
        HeapCount: 8,
        Comparator: compare,
        KeyFunc:    keyFunc,
    }

    sched, err := scheduler.New(cfg)
    if err != nil {
        log.Fatal(err)
    }

    defer sched.Shutdown()

    _ = sched.Add(&Worker{
        ID:       "worker-1",
        Priority: 10,
    })

    _ = sched.Add(&Worker{
        ID:       "worker-2",
        Priority: 20,
    })

    resource, err := sched.Acquire()
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println("Acquired:", resource.ID)
}

The important thing is how little domain logic is inside the scheduler.

The application defines:

Resource type
Key function
Comparator
Configuration

CRS manages the rest.

What v1.2.3 Changed Technically

The release isn't about replacing the architecture.

It is about tightening the implementation and making the public surface accurately represent what the implementation already does.

The release includes:

1. Cooldown initialization correction

The cooldown manager now has documentation and examples that correctly reflect its dependency on a lifecycle controller.

2. Cooldown example stabilization

The example accounts for asynchronous event dispatch so the demonstration is deterministic instead of racing an event-driven state transition.

3. Benchmark boundary cleanup

The scheduler benchmark uses:

package scheduler_test

rather than relying on internal package access.

4. Documentation corrections

The README and API documentation were aligned with the actual implementation, including exported errors and shared/exclusive complexity behavior.

5. Prometheus module alignment

The optional Prometheus extension is tagged:

extensions/prometheus/v1.2.3

and references the corresponding core release.

What I Don't Want These Numbers to Mean

This is probably the most important disclaimer in the entire post.

I don't want to say:

"CRS is 10x faster than every mutex-based scheduler."

I haven't proven that.

I don't want to say:

"CRS handles 37,000 requests/sec in production."

A workload-specific test is not a production guarantee.

I don't want to say:

"32 shards is always optimal."

It isn't.

The benchmarks show what happened under a particular machine and workload.

The architecture explains why those measurements are interesting.

That's a much more useful claim.

When Sharding Helps

Sharding is most interesting when:

many concurrent operations
          +
multiple independent resource groups
          +
shared scheduling state

It can reduce contention by allowing unrelated operations to work against different synchronization boundaries.

But it introduces its own tradeoffs:

more heaps
more bookkeeping
more complex selection logic
shard-distribution decisions
potentially uneven workloads if the routing strategy is poor

So the design isn't:

"Sharding is always better."

It's:

"Sharding is a useful way to control contention when the workload benefits from partitioning."

Why I Chose This Architecture

The design can be summarized as four decisions:

1. Heap
   ↓
efficient priority ordering

2. Sharding
   ↓
reduce shared lock contention

3. Lookup map
   ↓
fast direct resource access

4. Pluggable strategies
   ↓
separate scheduling policy from
resource-management mechanics

Then the lifecycle/event system sits around those primitives:

                Scheduler Core
                      │
       ┌──────────────┼──────────────┐
       ▼              ▼              ▼
     Heap           Lookup         Policy
       │              │              │
       └──────────────┼──────────────┘
                      │
                      ▼
                  Lifecycle
                      │
              ┌───────┴───────┐
              ▼               ▼
           Cooldown        Events
                              │
                              ▼
                          Telemetry

That separation is the part I'm most interested in continuing to improve.

v1.2.3 Release Checklist

Before publishing this release, I wanted the project to pass more than just a version bump.

The current validation includes:

go test ./...

go test -race ./...

go vet ./...

gofmt -l .

The benchmark suite was also run with:

go test -run="^$" -bench="." -benchmem ./...

And the release artifacts are available through the Go module ecosystem:

github.com/phero20/concurrent-resource-scheduler@v1.2.3

github.com/phero20/concurrent-resource-scheduler/extensions/prometheus@v1.2.3

The core module targets Go 1.22+.

The optional Prometheus extension is a separate module targeting the newer Go toolchain used by that extension.

Where This Project Goes Next

v1.2.3 is a good point to stop and evaluate the architecture under more workloads.

The areas I'm particularly interested in are:

larger resource pools
different shard counts
different acquire strategies
mixed shared/exclusive traffic
heavier update workloads
affinity-heavy workloads
more aggressive cooldown behavior
longer-running stress tests
different CPU architectures

The goal isn't simply to produce a bigger benchmark number.

The goal is to understand where the scheduler's architecture helps, where it doesn't, and what tradeoffs become visible at scale.

Final Thoughts

Building a concurrent scheduler taught me that the difficult part isn't implementing:

Acquire()

The difficult part is everything around it.

You need to coordinate:

                 RESOURCE SCHEDULING

                      Priority
                         │
                         ▼
Concurrency ───────► Scheduler ◄────── Acquisition Policy
                         │
             ┌───────────┼───────────┐
             ▼           ▼           ▼
           Lookup     Lifecycle    Events
             │           │           │
             ▼           ▼           ▼
          O(1) map    Active/     Telemetry
                      Inactive        │
                                     ▼
                                 Prometheus

That is what CRS is trying to provide.

Not just a priority queue.

Not just a resource pool.

But a reusable concurrency layer where:

resource ordering
acquisition policy
shard synchronization
direct lookup
shared/exclusive semantics
lifecycle state
cooldowns
events
observability

can exist as separate pieces without forcing the application to rebuild the entire system.

v1.2.3 is the current release.

If you're building an LLM gateway, proxy pool, database router, GPU worker pool, API-key manager, or another system where many concurrent requests compete for reusable resources, I'd genuinely like to hear how you're solving it.