DEV Community: Feroz

Concurrent Resource Scheduler v1.2.3: Sharded Priority Heaps Under Concurrent Load

Feroz — Thu, 13 Aug 2026 13:09:03 +0000

GitHub: https://github.com/phero20/concurrent-resource-scheduler

Go documentation: https://pkg.go.dev/github.com/phero20/concurrent-resource-scheduler

Concurrent Resource Scheduler v1.2.3: Sharded Priority Heaps Under Concurrent Load

A resource scheduler sounds simple until many goroutines start competing for a small number of resources.

At that point, the problem stops being:

"How do I pick the next item?"

and becomes:

"How do I pick the right item while thousands of goroutines are acquiring, releasing, updating, and observing resources concurrently?"

That is the problem I built Concurrent Resource Scheduler (CRS) to solve.

CRS is a domain-agnostic Go library for managing reusable resources with:

sharded priority heaps
concurrent-safe resource lookup
configurable acquisition strategies
shared and exclusive acquisition
affinity routing
resource lifecycle management
cooldown extensions
asynchronous events
optional Prometheus telemetry

The current release is v1.2.3.

This post is a release-focused technical deep dive into the architecture, the reasoning behind it, and the actual measurements from the current release.

First: What Problem Are We Actually Solving?

Imagine a service with a pool of reusable resources.

Those resources could be:

API keys
database connections
GPU workers
proxy endpoints
backend workers
LLM providers
service instances
connection pools
rate-limited accounts

Now imagine:

             10,000 concurrent requests
                       │
                       ▼
              ┌─────────────────┐
              │    Scheduler    │
              └────────┬────────┘
                       │
          ┌────────────┼────────────┐
          ▼            ▼            ▼
       Resource A   Resource B   Resource C

The scheduler needs to answer several questions at once:

Which resource should this request receive?
Is that resource active?
Is it already exclusively acquired?
Can it be shared?
Which resource has the best priority?
Should affinity influence the selection?
Has a resource entered cooldown?
What happens when it is released?
What happens when its priority changes?
How do we observe all of this without putting telemetry directly on the hot path?

A simple slice and mutex can answer some of these questions.

The difficult part is doing all of them concurrently and predictably.

The Obvious Design: One Slice, One Mutex

The first design most people naturally reach for is something like:

type Scheduler struct {
    mu        sync.Mutex
    resources []*Resource
}

Then acquisition becomes:

func (s *Scheduler) Acquire() (*Resource, error) {
    s.mu.Lock()
    defer s.mu.Unlock()

    // Scan resources.
    // Find the best available resource.
    // Mark it acquired.
    // Return it.

    return resource, nil
}

For a small pool, this is completely reasonable.

The problem appears when concurrency and resource count grow.

Suppose:

Resources:           10,000
Concurrent goroutines: 10,000

Every operation now enters the same synchronization boundary:

                 GLOBAL MUTEX
                      │
       ┌──────────────┼──────────────┐
       ▼              ▼              ▼
   goroutine       goroutine      goroutine
       │              │              │
       └──────────────┼──────────────┘
                      ▼
                    WAIT

Even if the actual operation only concerns a small part of the resource pool, the lock protects everything.

There is another problem.

If resources are stored in a slice and the scheduler has to find the best candidate, acquisition can require a linear scan:

O(N)

Now the same global lock is protecting an operation whose amount of work grows with the number of resources.

That is the combination I wanted to avoid.

The Main Architectural Decision

The core idea behind CRS is:

Partition the active resource pool into independently synchronized shards.

Instead of:

                    ONE LOCK
                       │
                 ONE BIG POOL

CRS uses:

                         Scheduler
                             │
              ┌──────────────┼──────────────┐
              ▼              ▼              ▼
           Shard 0        Shard 1        Shard N
              │              │              │
            Heap           Heap           Heap
              │              │              │
            Mutex          Mutex          Mutex

Each shard owns a priority heap.

Each heap has its own synchronization boundary.

That gives us a much more useful concurrency model:

goroutine A ──► shard 0 ──► heap 0
goroutine B ──► shard 1 ──► heap 1
goroutine C ──► shard 2 ──► heap 2

The goal is not to claim that sharding magically eliminates contention.

It doesn't.

The goal is to reduce the amount of unrelated work competing for the same lock.

That distinction matters.

Why a Priority Heap?

Sharding solves one problem:

How do we reduce contention?

We still need to solve another:

How do we efficiently maintain resource priority?

A slice makes this straightforward but potentially expensive:

resources:
[ A, B, C, D, E, F, G, ... ]

Find minimum priority:
scan everything

A priority heap gives us an ordered structure where the highest-priority candidate can be accessed from the heap root.

Conceptually:

             best resource
                  │
                  ▼
               [ 10 ]
              /      \
           [ 20 ]   [ 30 ]
           /   \
        [40]   [50]

So CRS combines the two ideas:

                 RESOURCE POOL
                       │
             ┌─────────┴─────────┐
             ▼                   ▼
          Sharding            Priority
             │                   │
             ▼                   ▼
        Lower lock           Heap ordering
         contention          / fast candidate

This is the central design tradeoff of the project.

The Scheduler Is More Than a Heap

One thing I learned while building CRS is that the heap is actually only one part of the problem.

The scheduler has several independent concerns:

                         CRS
                          │
       ┌──────────────────┼──────────────────┐
       │                  │                  │
       ▼                  ▼                  ▼
   Acquisition          State             Lookup
    Strategy          Lifecycle             Map
       │                  │                  │
       ▼                  ▼                  ▼
    Adaptive          Active/Inactive       O(1)
    Weighted          Acquire/Release       lookup
    Round Robin       Include/Exclude
    Affinity

Then there are extensions:

                Event Dispatcher
                       │
             ┌─────────┴─────────┐
             ▼                   ▼
          Cooldown           Telemetry
                                 │
                                 ▼
                             Prometheus

Keeping these responsibilities separated makes the implementation easier to reason about and lets the core remain domain-agnostic.

Resource Lookup Is a Separate Problem

Priority ordering tells us:

"Which resource should be considered first?"

But APIs also need direct resource lookup.

For example:

resource, err := sched.Get("worker-42")

We don't want to search every heap for that.

CRS therefore maintains a lookup structure alongside the active heap shards.

Conceptually:

                resource ID
                     │
                     ▼
              ┌─────────────┐
              │ Lookup Map  │
              └──────┬──────┘
                     │
                     ▼
                  node
                     │
             ┌───────┴───────┐
             ▼               ▼
          resource          shard

This gives the scheduler a useful separation:

heap → candidate ordering
lookup map → direct identity lookup

These are different access patterns, so they shouldn't be forced into the same data structure.

Acquisition Strategies Are Pluggable

Another design decision was not to hard-code one definition of "best resource."

Different systems want different behavior.

For example:

Adaptive

Choose resources based on observed scheduling state.

Weighted

Some resources should receive more traffic than others.

Round Robin

Distribute acquisitions sequentially.

Affinity

Prefer a resource or shard associated with some identifier.

That means the scheduler's core doesn't need to know why an application considers one resource better than another.

The application provides the policy.

The scheduler provides the concurrency-safe machinery around that policy.

Shared vs Exclusive Acquisition

This distinction is especially important.

A resource might support:

Shared

where multiple callers can use it simultaneously.

Or:

Exclusive

where only one caller can own it at a time.

These are not just two API names.

They produce different synchronization behavior.

For example, under shared acquisition, the scheduler can inspect the best candidate without removing it from the heap.

Conceptually:

Shared:

heap
  │
  ▼
peek candidate
  │
  ▼
use resource

Whereas exclusive acquisition may need to remove the candidate from the active scheduling position:

Exclusive:

heap
  │
  ▼
pop candidate
  │
  ▼
exclusive owner

This distinction is also reflected in the documentation and complexity model.

Resource Lifecycle

Resources don't simply exist or disappear.

They move through states.

A simplified lifecycle looks like:

              ┌──────────────┐
              │    ACTIVE    │
              └──────┬───────┘
                     │
            acquire / exclude
                     │
                     ▼
              ┌──────────────┐
              │   INACTIVE   │
              └──────┬───────┘
                     │
               include / release
                     │
                     ▼
              ┌──────────────┐
              │    ACTIVE    │
              └──────────────┘

There are also terminal operations such as removal and shutdown.

The important part is that the scheduler must maintain consistent state while these operations happen concurrently.

That's why lifecycle management is part of the scheduler design rather than an afterthought.

Cooldown Is an Extension, Not Core Scheduling Logic

Cooldown is a good example of why the architecture is modular.

Suppose a resource fails or needs temporary exclusion:

Resource
   │
   ▼
Cooldown
   │
   ├── removed from active scheduling
   │
   └── restored after duration

The core scheduler shouldn't need to understand every possible reason a resource temporarily leaves the pool.

The cooldown extension coordinates with the scheduler's lifecycle controller.

This was also one of the areas tightened in v1.2.3: the cooldown documentation and example now use the correct LifecycleController wrapper pattern instead of trying to initialize the cooldown manager before the scheduler exists.

That ordering matters because the scheduler is the object that ultimately owns the resource lifecycle.

Why v1.2.3 Exists

v1.2.3 is not a giant feature release.

It is a correctness, consistency, and release-hardening release.

The main fixes include:

Cooldown initialization documentation

The cooldown extension's initialization example was corrected to use the proper lifecycle-controller wrapper pattern.

This avoids the circular initialization problem of trying to construct a component that needs the scheduler before the scheduler has been created.

Cooldown example timing

The cooldown example was adjusted so it does not immediately race an asynchronous exclusion event.

The example now demonstrates the intended lifecycle deterministically.

Benchmark package boundary

The scheduler benchmark file was moved to the black-box package boundary:

package scheduler_test

because it only relies on exported APIs.

That makes the benchmark follow the same testing convention as the rest of the package.

Documentation consistency

README and API documentation were updated to match the implementation, including error documentation and complexity behavior.

Prometheus extension

The optional Prometheus module is also aligned to:

v1.2.3

while remaining a separate nested Go module.

The Numbers: Actual v1.2.3 Microbenchmarks

Architecture diagrams are useful.

Numbers are better.

I ran the current benchmark suite with:

go test -run="^$" -bench="." -benchmem ./...

The measurements below were collected locally on:

OS:   Windows
Arch: amd64
CPU:  AMD Ryzen 5 6600H with Radeon Graphics

These are local measurements, not universal performance guarantees.

Acquire Strategy Benchmarks

The current release produced the following verified measurements:

Strategy	Select/GetShard Cost
`ConsistentHashRing.GetShard`	7.2 ns
`WeightedStrategy.Select`	19.1 ns
`AdaptiveStrategy.Select`	27.0 ns

The strategy selection costs are all in the low-nanosecond range on this machine. The scheduler benchmarks below separately report allocation counts for the larger operations.

Scheduler Hot-Path Benchmarks

The scheduler benchmarks are more representative of actual scheduler operations.

Operation	HeapCount=1	HeapCount=8	HeapCount=32	Allocs/op
`Add`	591.9 ns	836.2 ns	718.5 ns	3
`Update`	280.6 ns	228.9 ns	200.4 ns	1
`BatchAdd` (1,000 resources)	308.1 µs	351.3 µs	331.1 µs	~1,100
`Acquire` (Shared, Sequential)	12.36 ns	11.52 ns	11.23 ns	0
`Acquire` (Shared, Parallel)	63.14 ns	16.80 ns	17.99 ns	0
`Acquire` + `Release` (Exclusive)	241.2 ns	249.2 ns	208.8 ns	0

There is also a parallel acquisition benchmark:

Benchmark	Heap count	ns/op
AcquireSharedParallel	1	63.14
AcquireSharedParallel	8	16.80
AcquireSharedParallel	32	17.99

The parallel benchmark is particularly useful when thinking about the sharding architecture.

On this machine and workload, moving from a single heap to multiple heaps reduced the measured AcquireSharedParallel time substantially:

HeapCount=1   63.14 ns/op
HeapCount=8   16.80 ns/op
HeapCount=32  17.99 ns/op

That is one workload on one machine, not a universal scaling law.

But it is exactly the kind of behavior the architecture is intended to make measurable.

Why We Benchmark Multiple Heap Counts

It would be easy to benchmark only:

HeapCount = 1

and stop there.

That wouldn't tell us much about the reason for sharding.

Instead, the scheduler benchmarks compare:

HeapCount = 1
HeapCount = 8
HeapCount = 32

This gives us a way to observe how the synchronization structure behaves as the number of shards changes.

The expected tradeoff is not:

"More shards are always faster."

It is:

"The right number of shards depends on workload, resource count, contention, and scheduling strategy."

More shards also mean more structures to manage.

So sharding is a tuning dimension, not a magic constant.

Batch Operations Have a Different Cost Profile

The benchmark suite also measures batch insertion:

Benchmark	Heap count	ns/op	B/op	allocs/op
BatchAdd	1	308.1 µs	283,715	1,042
BatchAdd	8	351.3 µs	283,530	1,095
BatchAdd	32	331.1 µs	282,314	1,223

This is important because it prevents cherry-picking only the fastest numbers.

AcquireShared is extremely cheap in the measured workload.

BatchAdd is much more expensive.

That's expected.

Adding a large batch involves substantially more work than peeking at an already-populated scheduling structure.

A useful benchmark suite should expose those differences instead of presenting one "magic" performance number.

Validation Is More Important Than a Single Benchmark

A concurrency library can produce beautiful microbenchmarks and still be broken.

That's why v1.2.3 was validated with multiple layers.

Normal tests

go test ./...

Result:

PASS

All tested packages completed successfully.

Race detector

go test -race ./...

Result:

PASS

No data races were reported.

Static analysis

go vet ./...

Result:

PASS

Formatting

gofmt -l .

Result:

(no output)

The working tree is therefore clean according to gofmt.

Why the Race Detector Matters

For a concurrency-heavy library, this command is not optional validation theater:

go test -race ./...

A normal test can pass while a race exists.

The race detector instruments memory accesses and can expose unsafe concurrent access that ordinary functional assertions don't catch.

It does not prove that a concurrent system is mathematically bug-free.

But it is one of the most important tools available for catching a class of real concurrency bugs.

For v1.2.3, the race-enabled test suite completed without reported races.

The Load-Test Question

Microbenchmarks answer:

"How quickly does this small operation execute?"

A load test asks a different question:

"What happens when the whole scheduler is placed inside a realistic concurrent workload?"

For CRS, the dedicated load-test harness models things such as:

concurrent workers
resource pools
backend processing delay
request duration
failures
cancellation
resource utilization
acquisition latency
total latency
throughput
burst traffic
cooldown behavior

The distinction is important.

A benchmark like:

25 ns/op

does not mean a real request takes 25 ns.

A real request also includes:

scheduler
   +
application work
   +
network
   +
backend
   +
serialization
   +
other system costs

So I treat microbenchmarks and load tests as different measurements.

10,000 Concurrent Workers

One of the release validation workloads uses:

10,000 concurrent workers

This is useful because it puts the scheduler under a very different kind of pressure from a small benchmark.

The important question isn't:

"Can Go start 10,000 goroutines?"

Of course it can.

The question is:

"Can the resource-management layer maintain correct lifecycle and acquisition behavior while thousands of workers continuously compete for a small pool?"

That is where the sharded architecture, lookup synchronization, acquisition strategies, and lifecycle rules interact.

For the load-test results themselves, I treat the numbers as workload-specific measurements rather than claiming they represent every deployment.

The load-test harness also separates scheduler-side failures from simulated backend failures, because those are operationally different failure modes.

A Failure Is Not Necessarily a Scheduler Failure

This distinction is easy to lose in a load test.

Imagine:

10,000 workers
       │
       ▼
    Scheduler
       │
       ▼
    Resource
       │
       ▼
 Backend request
       │
       ▼
    FAILURE

The backend failed.

That doesn't mean:

scheduler failed

Similarly:

scheduler acquire failure

is not the same thing as:

backend failure

A serious load test should keep these categories separate.

That's why the harness tracks acquisition behavior, backend behavior, release behavior, and timeout/cancellation behavior independently.

Why This Matters for Resource Pools

Consider four backend resources:

backend-1
backend-2
backend-3
backend-4

and:

10,000 concurrent workers

You cannot turn four resources into 10,000 concurrent backend operations just because there are 10,000 goroutines.

If the acquisition policy is exclusive, the resource pool remains the bottleneck:

10,000 workers
      │
      ▼
 ┌───────────┐
 │ Scheduler │
 └─────┬─────┘
       │
       ▼
  4 resources

That is not a failure.

That's exactly what resource scheduling is supposed to enforce.

Domain-Agnostic by Design

The scheduler doesn't know whether a resource is:

type APIKey struct {
    ID string
}

or:

type GPUWorker struct {
    ID string
}

or:

type DatabaseReplica struct {
    ID string
}

The application provides the resource type and the functions needed to identify and compare it.

For example:

compare := func(a, b *Worker) int {
    if a.Priority < b.Priority {
        return -1
    }

    if a.Priority > b.Priority {
        return 1
    }

    return 0
}

keyFunc := func(w *Worker) string {
    return w.ID
}

The scheduler doesn't need to understand what Priority means.

It only needs a consistent comparison function.

That's what makes the library reusable across domains.

A Minimal v1.2.3 Example

The basic flow remains intentionally small:

package main

import (
    "fmt"
    "log"

    "github.com/phero20/concurrent-resource-scheduler/config"
    "github.com/phero20/concurrent-resource-scheduler/scheduler"
)

type Worker struct {
    ID       string
    Priority int
}

func main() {
    compare := func(a, b *Worker) int {
        if a.Priority < b.Priority {
            return -1
        }

        if a.Priority > b.Priority {
            return 1
        }

        return 0
    }

    keyFunc := func(w *Worker) string {
        return w.ID
    }

    cfg := config.Config[*Worker, string]{
        HeapCount: 8,
        Comparator: compare,
        KeyFunc:    keyFunc,
    }

    sched, err := scheduler.New(cfg)
    if err != nil {
        log.Fatal(err)
    }

    defer sched.Shutdown()

    _ = sched.Add(&Worker{
        ID:       "worker-1",
        Priority: 10,
    })

    _ = sched.Add(&Worker{
        ID:       "worker-2",
        Priority: 20,
    })

    resource, err := sched.Acquire()
    if err != nil {
        log.Fatal(err)
    }

    fmt.Println("Acquired:", resource.ID)
}

The important thing is how little domain logic is inside the scheduler.

The application defines:

Resource type
Key function
Comparator
Configuration

CRS manages the rest.

What v1.2.3 Changed Technically

The release isn't about replacing the architecture.

It is about tightening the implementation and making the public surface accurately represent what the implementation already does.

The release includes:

1. Cooldown initialization correction

The cooldown manager now has documentation and examples that correctly reflect its dependency on a lifecycle controller.

2. Cooldown example stabilization

The example accounts for asynchronous event dispatch so the demonstration is deterministic instead of racing an event-driven state transition.

3. Benchmark boundary cleanup

The scheduler benchmark uses:

package scheduler_test

rather than relying on internal package access.

4. Documentation corrections

The README and API documentation were aligned with the actual implementation, including exported errors and shared/exclusive complexity behavior.

5. Prometheus module alignment

The optional Prometheus extension is tagged:

extensions/prometheus/v1.2.3

and references the corresponding core release.

What I Don't Want These Numbers to Mean

This is probably the most important disclaimer in the entire post.

I don't want to say:

"CRS is 10x faster than every mutex-based scheduler."

I haven't proven that.

I don't want to say:

"CRS handles 37,000 requests/sec in production."

A workload-specific test is not a production guarantee.

I don't want to say:

"32 shards is always optimal."

It isn't.

The benchmarks show what happened under a particular machine and workload.

The architecture explains why those measurements are interesting.

That's a much more useful claim.

When Sharding Helps

Sharding is most interesting when:

many concurrent operations
          +
multiple independent resource groups
          +
shared scheduling state

It can reduce contention by allowing unrelated operations to work against different synchronization boundaries.

But it introduces its own tradeoffs:

more heaps
more bookkeeping
more complex selection logic
shard-distribution decisions
potentially uneven workloads if the routing strategy is poor

So the design isn't:

"Sharding is always better."

It's:

"Sharding is a useful way to control contention when the workload benefits from partitioning."

Why I Chose This Architecture

The design can be summarized as four decisions:

1. Heap
   ↓
efficient priority ordering

2. Sharding
   ↓
reduce shared lock contention

3. Lookup map
   ↓
fast direct resource access

4. Pluggable strategies
   ↓
separate scheduling policy from
resource-management mechanics

Then the lifecycle/event system sits around those primitives:

                Scheduler Core
                      │
       ┌──────────────┼──────────────┐
       ▼              ▼              ▼
     Heap           Lookup         Policy
       │              │              │
       └──────────────┼──────────────┘
                      │
                      ▼
                  Lifecycle
                      │
              ┌───────┴───────┐
              ▼               ▼
           Cooldown        Events
                              │
                              ▼
                          Telemetry

That separation is the part I'm most interested in continuing to improve.

v1.2.3 Release Checklist

Before publishing this release, I wanted the project to pass more than just a version bump.

The current validation includes:

go test ./...

go test -race ./...

go vet ./...

gofmt -l .

The benchmark suite was also run with:

go test -run="^$" -bench="." -benchmem ./...

And the release artifacts are available through the Go module ecosystem:

github.com/phero20/concurrent-resource-scheduler@v1.2.3

github.com/phero20/concurrent-resource-scheduler/extensions/prometheus@v1.2.3

The core module targets Go 1.22+.

The optional Prometheus extension is a separate module targeting the newer Go toolchain used by that extension.

Where This Project Goes Next

v1.2.3 is a good point to stop and evaluate the architecture under more workloads.

The areas I'm particularly interested in are:

larger resource pools
different shard counts
different acquire strategies
mixed shared/exclusive traffic
heavier update workloads
affinity-heavy workloads
more aggressive cooldown behavior
longer-running stress tests
different CPU architectures

The goal isn't simply to produce a bigger benchmark number.

The goal is to understand where the scheduler's architecture helps, where it doesn't, and what tradeoffs become visible at scale.

Final Thoughts

Building a concurrent scheduler taught me that the difficult part isn't implementing:

Acquire()

The difficult part is everything around it.

You need to coordinate:

                 RESOURCE SCHEDULING

                      Priority
                         │
                         ▼
Concurrency ───────► Scheduler ◄────── Acquisition Policy
                         │
             ┌───────────┼───────────┐
             ▼           ▼           ▼
           Lookup     Lifecycle    Events
             │           │           │
             ▼           ▼           ▼
          O(1) map    Active/     Telemetry
                      Inactive        │
                                     ▼
                                 Prometheus

That is what CRS is trying to provide.

Not just a priority queue.

Not just a resource pool.

But a reusable concurrency layer where:

resource ordering
acquisition policy
shard synchronization
direct lookup
shared/exclusive semantics
lifecycle state
cooldowns
events
observability

can exist as separate pieces without forcing the application to rebuild the entire system.

v1.2.3 is the current release.

If you're building an LLM gateway, proxy pool, database router, GPU worker pool, API-key manager, or another system where many concurrent requests compete for reusable resources, I'd genuinely like to hear how you're solving it.

Project

Concurrent Resource Scheduler

GitHub: https://github.com/phero20/concurrent-resource-scheduler

Go documentation: https://pkg.go.dev/github.com/phero20/concurrent-resource-scheduler

Current release:

v1.2.3

Optional Prometheus extension:

extensions/prometheus/v1.2.3

If you find the project useful, a GitHub star is appreciated.

If you find a concurrency bug, even better: open an issue.

The most useful validation for a concurrency library isn't another diagram.

It's putting it under a workload the author didn't design for.

I Built a Concurrent Resource Scheduler in Go with Sharded Priority Heaps

Feroz — Wed, 12 Aug 2026 15:53:56 +0000

Support on GitHub: github.com/phero20/concurrent-resource-scheduler (Give it a star if you find it useful!)

View Docs: pkg.go.dev/github.com/phero20/concurrent-resource-scheduler

What happens when thousands of concurrent requests compete for a small pool of reusable resources?

You can put a mutex around a slice and hope for the best.

Or you can design the scheduler around concurrency from the beginning.

I chose the second option.

I built Concurrent Resource Scheduler (CRS), a domain-agnostic Go library for selecting, prioritizing, routing, and maintaining reusable resources under heavy concurrent load.

It was designed from the ground up for production readiness. The core library supports Go 1.22+ and is intentionally built with zero third-party dependencies. Extended features like Prometheus telemetry are strictly separated into an optional nested Go module (Go 1.25+) to keep the core scheduler dependency graph perfectly empty.

The core idea is simple:

                  MANY CONCURRENT REQUESTS
                           │
                           ▼
                  ┌───────────────────┐
                  │ Resource Scheduler│
                  └─────────┬─────────┘
                            │
             ┌──────────────┼──────────────┐
             │              │              │
             ▼              ▼              ▼
        Priority         Acquire        State
          Heap           Strategy      Management
             │              │              │
             └──────────────┼──────────────┘
                            │
                            ▼
                    BEST AVAILABLE
                       RESOURCE

But making that work correctly under concurrency is where things get interesting.

CRS is designed for use cases such as:

LLM/API gateways
API key pools
proxy rotation
database replicas
GPU workers
backend pools
worker resources
connection pools
rate-limited providers
reusable compute resources

The scheduler itself does not know what a resource means.

It only knows:

"I have resources. I need to safely maintain them, prioritize them, and return an appropriate one to a concurrent caller."

The Problem
The Naive Approach
Why a Global Mutex Becomes a Problem
The Core Idea Behind CRS
Architecture at a Glance
Sharded Priority Heaps
Why Sharding Helps
The O(1) Lookup Map
Priority and Acquire Are Different Problems
Acquire Strategies
Round Robin
Weighted Acquire
Adaptive Acquire
Affinity Routing
Shared vs Exclusive Acquisition
Resource Lifecycle
Atomic State Transitions
The Inactive Store
Batch Operations
Updates Without Destroying Heap Ordering
Cooldowns
Asynchronous Events
Observability
Prometheus Integration
Concurrency Model
Complexity
Testing the Library
Race Detector Validation
Real-World Load Testing
10,000 Concurrent Workers
Burst Testing
Failure Testing
Cooldown Stress Testing
What the Load Tests Actually Tell Us
A Minimal Example
LLM Gateway Example
Why CRS Is Domain-Agnostic
Project Structure
Design Principles
Lessons Learned
When You Should NOT Use CRS
Future Directions
Installation
Final Thoughts

The Problem

Let's start with a realistic scenario.

Imagine an LLM gateway with 100 API keys.

Each key may have different:

rate-limit availability
priority
health
cooldown state
provider
capacity
temporary availability

Thousands of requests arrive concurrently.

A simplified system looks like:

                   ┌──────────────┐
Request 1 ────────►              │
Request 2 ────────►              │
Request 3 ────────►   Gateway    │
Request 4 ────────►              │
Request 5 ────────►              │
   ...             │              │
Request N ────────►              │
                   └──────┬───────┘
                          │
                          ▼
                  ┌───────────────┐
                  │ Resource Pool │
                  └───────┬───────┘
                          │
           ┌──────────────┼──────────────┐
           ▼              ▼              ▼
        API Key 1      API Key 2      API Key N

The scheduler now has to answer:

Which resource should this request use?
Which resource has the best priority?
Is the resource currently active?
Can multiple requests use it simultaneously?
Should this resource temporarily leave the pool?
Which shard should we search?
Should requests stick to the same shard?
What happens when the resource is released?
How do we update its priority?
How do we observe all of this without slowing down the hot path?

That is the problem CRS tries to solve.

The Naive Approach

The easiest implementation looks something like:

type Scheduler struct {
    mu        sync.Mutex
    resources []*Resource
}

Then:

func (s *Scheduler) Acquire() *Resource {
    s.mu.Lock()
    defer s.mu.Unlock()

    // Scan resources.
    // Find the best one.
    // Return it.

    return best
}

At first glance, this looks perfectly reasonable.

For 10 resources and 2 goroutines, it probably is.

But imagine:

Resources:           10,000
Concurrent requests: 5,000

Now every operation fights over one lock.

                GLOBAL MUTEX
                     │
       ┌─────────────┼─────────────┐
       │             │             │
       ▼             ▼             ▼
    Worker 1      Worker 2      Worker 3
       │             │             │
       └─────────────┼─────────────┘
                     │
                  WAITING

The scheduler becomes serialized around the lock.

Why a Global Mutex Becomes a Problem

There are several problems.

1. Lock contention

Only one goroutine can manipulate the pool at a time.

2. Linear scanning

If resources are stored in an array, finding the best resource can become:

O(N)

per acquisition.

3. Priority maintenance

If resources have priorities that change, the scheduler has to continuously maintain ordering.

4. State transitions

Resources can move between:

ACTIVE
INACTIVE
REMOVED

and those transitions must be synchronized.

5. Observability

Metrics and event callbacks should not block the scheduler.

The challenge is therefore not just:

"How do I build a priority queue?"

It is:

"How do I build a concurrent priority resource manager where priority, acquire, lifecycle, and observability coexist?"

The Core Idea Behind CRS

The central architectural decision was:

Don't put one global lock around the entire priority structure.

Instead, CRS partitions resources into independently locked shards.

Conceptually:

                  CRS
                   │
        ┌──────────┼──────────┐
        │          │          │
        ▼          ▼          ▼
     Shard 1    Shard 2    Shard N
        │          │          │
      Heap       Heap       Heap
        │          │          │
     Mutex      Mutex      Mutex

Each shard owns its own heap and its own lock.

This is the heart of CRS.

Architecture at a Glance

                        APPLICATION
                             │
               Add / Acquire / Release / Update
                             │
                             ▼
                  ┌────────────────────┐
                  │   CRS Scheduler    │
                  └─────────┬──────────┘
                            │
            ┌───────────────┼────────────────┐
            │               │                │
            ▼               ▼                ▼
      Acquire          Lookup          Inactive
       Strategy            Map             Store
            │               │                │
            ▼               ▼                │
     Candidate Shard     O(1) Node          │
            │                                │
            ▼                                │
    ┌─────────────────────────────────┐      │
    │         ACTIVE HEAP SHARDS       │      │
    │                                 │      │
    │  Heap 1     Heap 2     Heap N   │      │
    │  +Mutex     +Mutex     +Mutex   │      │
    └─────────────────────────────────┘      │
            │                                │
            └──────────────┬─────────────────┘
                           │
                           ▼
                    EVENT DISPATCHER
                           │
                 ┌─────────┴──────────┐
                 ▼                    ▼
             Telemetry            Cooldown
                 │                    │
                 ▼                    ▼
             Prometheus          Resource State

This separation is intentional.

Sharded Priority Heaps

Each shard maintains a priority heap.

For example:

Shard 1

        [Priority 10]
        /           \
 [Priority 20]    [Priority 30]
    /     \
[40]      [50]

Another shard:

Shard 2

        [Priority 5]
        /          \
 [Priority 15]   [Priority 25]

Every shard has its own synchronization boundary.

Shard 1                 Shard 2                 Shard 3

┌─────────────┐        ┌─────────────┐        ┌─────────────┐
│   Mutex     │        │   Mutex     │        │   Mutex     │
├─────────────┤        ├─────────────┤        ├─────────────┤
│ Priority    │        │ Priority    │        │ Priority    │
│ Heap        │        │ Heap        │        │ Heap        │
└─────────────┘        └─────────────┘        └─────────────┘

There is no global heap mutex.

Why Sharding Helps

Suppose we have 32 shards.

Instead of:

1 global lock

we have:

32 independently locked heaps

Different goroutines can operate on different shards simultaneously.

Goroutine A ─────► Shard 1 ─────► lock
Goroutine B ─────► Shard 7 ─────► lock
Goroutine C ─────► Shard 19 ────► lock
Goroutine D ─────► Shard 27 ────► lock

The locks are independent.

This doesn't magically eliminate contention.

If every request targets the same shard, that shard can still become contended.

That's why CRS also separates:

acquire strategy

from

priority ordering

This distinction is extremely important.

The O(1) Lookup Map

A heap is excellent at answering:

"What is the best resource?"

But a heap is not ideal for answering:

"Where is resource X?"

Searching a heap can require scanning.

CRS therefore maintains an additional lookup structure.

Conceptually:

ID
 │
 ▼
┌──────────────────────┐
│ Lookup Map            │
│                      │
│ "backend-01" ───────► Node
│ "backend-02" ───────► Node
│ "backend-03" ───────► Node
└──────────────────────┘

The lookup map is protected independently with a read/write mutex.

This gives the scheduler an O(1)-style membership/location lookup by application-defined key.

That is particularly useful for:

Get
Update
Remove
Release
Exclude
Include

without scanning every heap.

Priority and Acquire Are Different Problems

This is one of the most important design ideas in CRS.

A resource can have:

Priority = 10

but that doesn't necessarily tell us:

Which shard should we inspect first?

These are separate decisions.

CRS therefore separates:

                REQUEST
                   │
                   ▼
            ACQUIRE STRATEGY
                   │
                   ▼
              SHARD SELECTION
                   │
                   ▼
             PRIORITY HEAP
                   │
                   ▼
             BEST RESOURCE

This allows different routing strategies to be plugged into the scheduler without changing the underlying heap implementation.

Acquire Strategies

CRS provides several acquire approaches:

Round Robin
Weighted
Adaptive
Consistent Hashing for affinity routing

Each solves a different problem.

Round Robin

Round Robin is the simplest.

Request 1 → Shard 1
Request 2 → Shard 2
Request 3 → Shard 3
Request 4 → Shard 4
Request 5 → Shard 1
...

It is simple and predictable.

Use it when:

shards are roughly equivalent
you want even distribution
resource capacity is similar

Weighted Acquire

Not every shard is necessarily equal.

Imagine:

GPU 1 → 24 GB VRAM
GPU 2 → 24 GB VRAM
GPU 3 → 80 GB VRAM
GPU 4 → 80 GB VRAM

You may want larger resources to receive more work.

Weighted acquire lets you express relative capacity.

Conceptually:

Shard 1: weight 1
Shard 2: weight 1
Shard 3: weight 4
Shard 4: weight 4

Traffic can then be distributed proportionally.

This is useful for:

heterogeneous GPUs
backend instances with different capacity
API providers with different quotas
worker pools with different performance characteristics

Adaptive Acquire

Round Robin doesn't know anything about current load.

Adaptive acquire attempts to account for shard activity.

Conceptually:

             REQUEST
                │
                ▼
       ┌──────────────────┐
       │ Inspect shard    │
       │ load information  │
       └────────┬─────────┘
                │
       ┌────────┼────────┐
       ▼        ▼        ▼
    Shard A   Shard B   Shard C
      busy      low       busy
                │
                ▼
            choose B

The scheduler uses lightweight shard-level state to favor less-contended shards without introducing another global lock.

This is useful when the resource pool is dynamic and simple round-robin distribution isn't enough.

Affinity Routing

Sometimes you don't want random distribution.

You want:

user-123 → same shard
user-456 → same shard

CRS supports affinity routing through consistent hashing.

Conceptually:

                   HASH RING

             ┌───────────────────┐
             │                   │
        S1   │        S2         │
             │                   │
             │                   │
        S4   │        S3         │
             │                   │
             └───────────────────┘

              ▲
              │
        hash("user-123")

The same affinity identifier deterministically maps to the same shard.

This is useful for:

sticky sessions
tenant affinity
cache locality
connection locality
stateful workers

Shared vs Exclusive Acquisition

CRS supports two major acquisition semantics.

Shared

The resource remains active.

ACTIVE
  │
  │ Acquire
  ▼
ACTIVE

Multiple callers can acquire the same resource.

This is useful when resources represent things like:

API keys
read replicas
stateless endpoints
shared provider capacity

Exclusive

The resource temporarily leaves the active pool.

ACTIVE
   │
   │ Acquire
   ▼
INACTIVE
   │
   │ Release
   ▼
ACTIVE

This is useful when a resource represents something that cannot safely be used by multiple concurrent operations.

Examples:

GPU worker
exclusive connection
physical device
single-use worker
exclusive job executor

Resource Lifecycle

A CRS resource essentially moves through states.

                  ┌───────────┐
                  │   ADD     │
                  └─────┬─────┘
                        ▼
                 ┌─────────────┐
                 │   ACTIVE    │
                 └──────┬──────┘
                        │
              ┌─────────┼─────────┐
              │         │         │
           Acquire    Exclude   Remove
              │         │         │
              ▼         ▼         ▼
          INACTIVE   INACTIVE   DELETED
              │
            Release
              │
              ▼
           ACTIVE

The important invariant is:

A resource should exist in exactly one state/location at a time.

Atomic State Transitions

Imagine:

Goroutine A: Acquire(resource)
Goroutine B: Remove(resource)
Goroutine C: Update(resource)

all happening at almost the same time.

Without careful synchronization, you can get:

Resource exists in heap
AND
Resource exists in inactive store

or:

Lookup map says ACTIVE
but heap doesn't contain it

Those are catastrophic consistency bugs.

CRS therefore treats state transitions as carefully synchronized operations involving:

lookup state
heap state
inactive state
shard locks

The Inactive Store

The inactive store is particularly important for Exclusive acquisition.

Suppose:

backend-01

is acquired exclusively.

It is removed from the active heap and stored as inactive.

ACTIVE HEAP

backend-01
backend-02
backend-03

After acquisition:

ACTIVE HEAP

backend-02
backend-03


INACTIVE STORE

backend-01

When released:

INACTIVE STORE
      │
      │ Release
      ▼
ACTIVE HEAP

The lookup map allows CRS to locate the resource without scanning every heap.

Batch Operations

Adding resources one at a time is easy.

But imagine importing 10,000 resources.

You don't want partial state like:

Batch:
1 ✓
2 ✓
3 ✓
4 ✓
5 ✗
6 ?
7 ?
...

CRS provides BatchAdd with atomic insertion behavior.

Conceptually:

                BatchAdd
                   │
                   ▼
          ┌─────────────────┐
          │ Validate batch  │
          └────────┬────────┘
                   │
                   ▼
          ┌─────────────────┐
          │ Prepare changes │
          └────────┬────────┘
                   │
                   ▼
          ┌─────────────────┐
          │ Insert shards   │
          └────────┬────────┘
                   │
                   ▼
              COMPLETE

The goal is to avoid exposing a partially inserted batch.

Updates Without Destroying Heap Ordering

Suppose:

A = priority 10
B = priority 20
C = priority 30

Now:

C → priority 5

C should move toward the top.

Simply changing the value isn't enough.

The heap must be repaired.

Conceptually:

Before:

        A(10)
       /     \
    B(20)   C(30)


Update:

C(30) → C(5)


After:

        C(5)
       /    \
    A(10)  B(20)

This preserves the priority-queue invariant.

Cooldowns

Real resources sometimes need a cooldown period.

For example:

API key hits rate limit
        │
        ▼
   cooldown 5s
        │
        ▼
available again

CRS includes a cooldown extension.

The architecture is event-driven:

Acquire
   │
   ▼
Release
   │
   ▼
Event Dispatcher
   │
   ▼
Cooldown Manager
   │
   ▼
Exclude resource
   │
   ▼
wait
   │
   ▼
Include resource

The cooldown extension is asynchronous. That means there can be a small eventual-consistency window between a release event and the cooldown observer processing it.

That is intentional.

Asynchronous Events

Observability and extensions should not unnecessarily slow the scheduler's hot path.

CRS therefore uses an event dispatcher.

                 Scheduler
                     │
                     │ emit event
                     ▼
             ┌───────────────┐
             │ Buffered      │
             │ Event Stream  │
             └───────┬───────┘
                     │
              background worker
                     │
          ┌──────────┼──────────┐
          ▼          ▼          ▼
      Observer 1  Observer 2  Observer 3
          │          │          │
          ▼          ▼          ▼
       Metrics   Cooldown   Prometheus

The scheduler doesn't need to execute arbitrary observer logic while holding heap locks.

Observability

A production scheduler should answer:

How many acquisitions happened?
How many releases?
Which resources are being used?
How many failures?
What is the current resource count?

CRS includes telemetry support using atomic counters and asynchronous events.

The goal is:

Scheduler hot path
      │
      ▼
lightweight event
      │
      ▼
asynchronous telemetry

rather than doing expensive monitoring work while holding scheduler locks.

Prometheus Integration

CRS provides an optional Prometheus exporter extension.

In the new architecture, Prometheus integration lives entirely in its own nested Go module:
github.com/phero20/concurrent-resource-scheduler/extensions/prometheus

This separation exists so that:

The core scheduler remains compatible with Go 1.22+.
The core scheduler maintains zero third-party dependencies.
Prometheus remains completely optional.
Users who don't need Prometheus don't download its dependency graph.

Conceptually:

               CRS
                │
                ▼
          Telemetry
                │
                ▼
      ┌─────────────────┐
      │ Atomic Counters │
      └────────┬────────┘
               │
               ▼
       Prometheus Collector
               │
               ▼
       /metrics endpoint

This makes it possible to expose scheduler activity to an existing monitoring stack.

Concurrency Model

The concurrency model is based on multiple independent synchronization boundaries.

                   Scheduler
                       │
          ┌────────────┼────────────┐
          │            │            │
          ▼            ▼            ▼
      Shard 1      Shard 2       Shard N
       Mutex         Mutex         Mutex
          │            │            │
          ▼            ▼            ▼
        Heap          Heap          Heap

Separately:

Lookup Map
    │
sync.RWMutex

And:

Telemetry
    │
atomic counters / event channel

This is much more granular than one mutex around everything.

Complexity

The scheduler is designed around heap and lookup properties.

Operation	Complexity	Synchronization
Add	O(log N)	Single shard lock
BatchAdd	O(log N) per insertion	Shard locks
Acquire	Acquire + heap operation	Shard lock
AcquireByAffinity	Hash lookup + heap operation	Shard lock
Release	O(log N)	Single shard lock
Update	O(log N) active / O(1) inactive	Shard lock
Remove	O(log N) active / O(1) inactive	Shard lock
Get	O(1)-style lookup	Lookup synchronization
Stats	O(Shards)	Short shard reads

Where:

N = resources in a shard

Actual performance depends on workload, shard count, resource distribution, acquire policy, and contention.

Testing the Library

A concurrency library cannot be validated with only:

go test ./...

Unit tests answer:

"Does this operation behave correctly?"

Load tests answer:

"What happens when thousands of goroutines continuously hammer it?"

The CRS test strategy includes:

                  TESTING
                     │
       ┌─────────────┼─────────────┐
       │             │             │
       ▼             ▼             ▼
   Unit Tests    Race Tests    Load Tests
       │             │             │
       ▼             ▼             ▼
  correctness    data races    behavior

In the current multi-module architecture, testing includes:

Go 1.22 Core Test Validation: Ensuring the core scheduler works natively on Go 1.22 without third-party dependencies.
Go 1.25 Prometheus Tests: Validating the optional extension module separately.
Go Vet: Continuous static analysis.
Concurrency Stress Testing: Continuous validation of multi-shard locks and lifecycle state.

Tests cover:

scheduler construction
configuration validation
Add
BatchAdd
Get
Len
Stats
Acquire
Release
Update
Include
Exclude
Remove
Shutdown
acquire strategies
event observers
cooldown
telemetry
Prometheus collection
concurrent stress behavior

Race Detector Validation

One of the most important commands for a concurrent Go library is:

go test -count=1 -race ./...

The race detector doesn't prove that a library is bug-free.

But it can catch a class of extremely dangerous concurrency problems that ordinary tests may miss.

The final release was validated with the race detector across both Go 1.22 (for core) and Go 1.25 (for extensions) without reported data races.

Real-World Load Testing

For CRS, I created a dedicated load-test harness.

The harness models:

concurrent workers
backend processing delay
random request durations
backend failures
request cancellation
different scenarios
resource utilization
acquisition latency
backend latency
total latency
throughput
cooldown behavior
burst traffic

The important distinction is that these are load-test results, not universal benchmarks.

10,000 Concurrent Workers

One of the larger tests used:

Concurrency: 10,000 workers
Resources:   4 backends
Duration:    60 seconds
Scenario:    normal
Acquire:     adaptive
Policy:      shared
Race detector: enabled

The test produced approximately:

Total requests:        2,248,610
Successful:            2,226,165
Failed:                22,445

Throughput:            ~37,137.28 req/s

Peak simultaneous:     10,000

Resource utilization was extremely well-balanced:

backend-01    25.01%
backend-02    24.96%
backend-03    25.03%
backend-04    25.00%

The scheduler acquisition latency was remarkably low, and should be clearly distinguished from the simulated backend latency:

Acquire Latency:
mean: 7.59µs
max:  51.893ms

Backend Latency:
p50: 260.334ms
p95: 477.661ms
p99: 496.863ms

Total Request Latency:
p50: 262.430ms
p95: 479.425ms
p99: 498.559ms

Accounting check verified complete consistency:

success + backend failures = 2,248,610
backend requests = 2,248,610
total request attempts = 2,248,610
accounting: OK

Note: The 22,445 failures correspond exactly to the configured 1% simulated backend failure rate, not scheduler acquire failures (which remained at 0). Acquire timeouts and release failures were also 0.

The test environment was:

OS:           Windows
Architecture: amd64
CPU cores:    12
GOMAXPROCS:   12
Go:           1.25.5

The backend latency was simulated and should not be confused with CRS scheduler latency. These are workload-specific test results and should NOT be presented as universal benchmark claims.

Burst Testing

Real systems often receive bursts.

Traffic
  │
  │       ███████████
  │       ███████████
  │       ███████████
  │
  │  ███
  │  ███
  │  ███
  └────────────────────────► time

The burst scenario generated approximately:

Requests:        468,791
Successful:      464,109
Failed:          4,682

Throughput:      ~7,746 req/s

Peak simultaneous: 5,000

The success rate was approximately:

99%

Backend latency remained around:

p50 ≈ 260 ms
p95 ≈ 476 ms
p99 ≈ 495 ms

These backend numbers came from the simulated workload.

Failure Testing

A scheduler should also behave correctly when resources fail.

I tested a failure scenario with:

Concurrency: 1000
Resources:   4
Duration:    60 seconds
Failure rate: 10%

The workload produced approximately:

Requests:       231,443
Successful:     208,293
Failed:          23,150
Success rate:    90%

The scheduler reported:

Acquire failures: 0
Release failures: 0
Backend failures: 23,150

That distinction matters.

The scheduler successfully acquired resources while simulated backend operations failed at the expected rate.

Cooldown Stress Testing

Cooldown was one of the more interesting tests.

With exclusive acquisition, resources temporarily leave the active pool.

                4 resources
                     │
                     ▼
              ┌─────────────┐
              │   ACTIVE    │
              └──────┬──────┘
                     │
                Acquire
                     │
                     ▼
              ┌─────────────┐
              │  INACTIVE   │
              └──────┬──────┘
                     │
                  cooldown
                     │
                     ▼
              ┌─────────────┐
              │   ACTIVE    │
              └─────────────┘

At 1,000 concurrent workers, the scheduler generated a very large number of acquisition attempts while only a limited number of resources were available.

This demonstrated an important property:

High concurrency doesn't mean unlimited successful backend concurrency.

If only four resources exist and the policy is exclusive, four resources are still four resources.

What the Load Tests Actually Tell Us

The load tests gave several useful observations.

1. Sharding worked under heavy concurrency

The scheduler continued operating with thousands of concurrent workers without race-detector failures.

2. Acquire remained balanced

The adaptive workload distributed acquisitions across four backends at approximately 25% each.

3. Resource state remained consistent

The accounting checks showed that acquired resources were not silently lost.

4. Failures remained distinguishable

The harness separated:

Acquire failure
Backend failure
Release failure
Timeout

A scheduler failure and a backend failure are very different operational problems.

5. Cooldown changes the workload completely

With exclusive resources, the bottleneck becomes resource availability rather than CPU.

A Minimal Example

The basic usage pattern is:

package main

import (
    "fmt"
    "log"

    "github.com/phero20/concurrent-resource-scheduler/config"
    "github.com/phero20/concurrent-resource-scheduler/scheduler"
)

type Worker struct {
    ID       string
    Priority int
}

func main() {
    compare := func(a, b *Worker) int {
        if a.Priority < b.Priority {
            return -1
        }

        if a.Priority > b.Priority {
            return 1
        }

        return 0
    }

    keyFunc := func(w *Worker) string {
        return w.ID
    }

    cfg := config.Config[*Worker, string]{
        HeapCount: 1,
        Comparator: compare,
        KeyFunc:    keyFunc,
    }

    sched, err := scheduler.New(cfg)
    if err != nil {
        log.Fatal(err)
    }

    defer sched.Shutdown()

    sched.Add(&Worker{
        ID:       "worker-1",
        Priority: 50,
    })

    sched.Add(&Worker{
        ID:       "worker-2",
        Priority: 10,
    })

    resource, err := sched.Acquire()

    if err != nil {
        log.Fatal(err)
    }

    fmt.Println("Acquired:", resource.ID)
}

The application controls the resource type, key, and comparison logic.

The scheduler handles the concurrent resource management.

LLM Gateway Example

Imagine:

type APIKey struct {
    ID        string
    Provider  string
    Remaining int
}

Your application could define priority based on remaining quota.

Conceptually:

API Key
   │
   ├── Provider
   ├── Remaining quota
   └── Health

The comparator could make a key with more remaining capacity more desirable.

The scheduler doesn't need to know what those fields mean.

The application owns that logic.

CRS maintains the ordering and concurrent lifecycle.

This is the core idea behind being domain-agnostic.

Why CRS Is Domain-Agnostic

The scheduler doesn't contain logic like:

if resource.IsGPU() {
    ...
}

or:

if provider == "openai" {
    ...
}

Instead, the application provides:

Resource type
     +
Key function
     +
Comparator
     +
Acquire strategy

This makes the same scheduler applicable to many domains.

Example: GPU Workers

                Job
                 │
                 ▼
          ┌─────────────┐
          │ CRS         │
          └──────┬──────┘
                 │
       ┌─────────┼─────────┐
       ▼         ▼         ▼
      GPU 1     GPU 2     GPU 3
      24 GB     24 GB     80 GB

Weighted acquire could favor larger GPUs.

Exclusive acquisition could prevent two jobs from taking the same GPU.

Example: Database Replicas

                 Query
                   │
                   ▼
                 CRS
                   │
        ┌──────────┼──────────┐
        ▼          ▼          ▼
      DB-1       DB-2       DB-3
       20%        70%        35%

The application could define priority around current load.

The scheduler remains unaware that the resources happen to be databases.

Example: Proxy Pool

                  HTTP Request
                       │
                       ▼
                     CRS
                       │
        ┌──────────────┼──────────────┐
        ▼              ▼              ▼
    Proxy A         Proxy B        Proxy C
    healthy         cooldown       healthy

Cooldown can temporarily remove a failing proxy.

Affinity can keep a particular tenant or session mapped consistently.

Project Structure

concurrent-resource-scheduler/
│
├── go.mod               (Core module: Go 1.22+, Zero dependencies)
├── config/
├── scheduler/
├── acquire/
├── internal/
│   ├── heap/
│   ├── lookup/
│   └── node/
├── extensions/
│   ├── cooldown/
│   ├── metrics/
│   └── prometheus/
│       └── go.mod       (Prometheus module: Go 1.25+)
├── events/
├── errors/
├── stats/
├── tests/               (Core production-validation tests)
├── examples/
├── docs/
├── README.md
├── CHANGELOG.md
├── CONTRIBUTING.md
├── SECURITY.md
└── LICENSE

The main architectural boundary is:

Public API
    │
    ▼
Scheduler
    │
    ├── Acquire
    ├── Heap
    ├── Lookup
    ├── Events
    └── Extensions

Design Principles

Several principles shaped CRS.

1. No global heap lock

The heap state is partitioned.

2. Separate routing from priority

Acquire chooses where to look.

The heap chooses what is best.

3. Keep the scheduler domain-agnostic

The application owns business logic.

4. Keep observability outside the hot path

Telemetry should not become the scheduler bottleneck.

5. Make state transitions explicit

Resources are either active, inactive, or removed.

6. Optimize for concurrent workloads

Concurrency is not an afterthought.

It is part of the architecture.

Lessons Learned

Building a concurrent library taught me something important:

Concurrency problems are usually not caused by one complicated function.

They are caused by interactions between simple functions.

For example:

Acquire
   +
Release
   +
Update
   +
Remove
   +
Cooldown
   +
Events
   +
Concurrent callers

Each function can be correct individually while the combination is broken.

That's why concurrency testing needs to go beyond unit tests.

Another Lesson: Observability Is Part of the Design

When you have:

10,000 requests
4 resources
multiple shards
multiple goroutines
failures
cooldowns

you need to know what actually happened.

That's why CRS exposes events and telemetry around scheduler activity.

Without observability, debugging concurrent systems becomes guesswork.

Another Lesson: Load Tests Need Accounting

One of the most useful parts of the load harness was accounting.

For example:

total requests
      │
      ├── acquire failures
      │
      └── reached backend
              │
              ├── successful
              └── backend failure

If:

success + backend_failure

doesn't match:

backend_requests

something is wrong.

Accounting checks are powerful for detecting concurrency bugs.

When You Should NOT Use CRS

CRS isn't intended to replace every queue or pool implementation.

You probably don't need it if you simply have:

10 objects
+
one goroutine

A simple slice may be better.

You also probably don't need CRS if your problem is purely:

"I need a FIFO queue."

Use a simpler primitive.

CRS becomes interesting when you need combinations of:

concurrency
+
priority
+
acquire
+
resource lifecycle
+
affinity
+
cooldowns
+
observability

Future Directions

Potential future areas include:

deeper benchmarking
additional acquire strategies
richer scheduling policies
more advanced resource health models
better operational tooling
additional observability integrations
workload-specific tuning
broader real-world validation

The goal is not to optimize for feature count.

The goal is to keep the core scheduler:

small
predictable
composable
concurrent
domain-agnostic

Installation

Install the core library (zero third-party dependencies):

go get github.com/phero20/concurrent-resource-scheduler

If you plan to use the optional Prometheus metrics exporter, install the extension separately:

go get github.com/phero20/concurrent-resource-scheduler/extensions/prometheus

Then explore the examples and documentation in the repository.

Final Thoughts

The interesting part of building a scheduler isn't writing:

Acquire()

The interesting part is everything around it.

You need to think about:

                    RESOURCE SCHEDULING

                         ┌─────────┐
                         │ Priority│
                         └────┬────┘
                              │
        ┌─────────────────────┼─────────────────────┐
        │                     │                     │
        ▼                     ▼                     ▼
   Concurrency           Acquire              Lifecycle
        │                     │                     │
        ▼                     ▼                     ▼
   Sharded heaps       Adaptive/Weighted       Active/Inactive
        │               Round Robin             Release
        │                     │                     │
        └─────────────────────┼─────────────────────┘
                              │
                              ▼
                         Observability
                              │
                   ┌──────────┴──────────┐
                   ▼                     ▼
               Telemetry            Prometheus

CRS was built around that complete picture.

It is not just a priority queue.

It is a concurrent resource-management layer that combines:

sharded priority heaps
concurrent-safe lookup
configurable acquire
priority ordering
shared and exclusive acquisition
affinity routing
resource lifecycle management
cooldown extensions
asynchronous events
telemetry
optional Prometheus integration (separate module)

And the project has been tested beyond basic unit tests, including:

Race detector
       +
Concurrent stress tests
       +
1,000 workers
       +
2,000 workers
       +
5,000 workers
       +
10,000 workers
       +
Burst workloads
       +
Failure workloads
       +
Cooldown workloads

The most important result isn't a single throughput number.

It's that the architecture gives me a foundation where concurrency, scheduling policy, resource state, and observability can evolve independently.

That's what I wanted to build.

If You're Building Something Similar

I'd love to hear how you approach resource scheduling.

Especially if you're working on:

LLM gateways
GPU schedulers
proxy pools
database routing
distributed workers
API key rotation
connection pools
high-concurrency Go services

What would you change about this architecture?

What workloads should I test next?

What concurrency problems have you encountered in production?

I'd genuinely like to hear from people who have operated systems like these in the real world.

Project

Support on GitHub: github.com/phero20/concurrent-resource-scheduler (Give it a star if you find it useful!)

View Docs: pkg.go.dev/github.com/phero20/concurrent-resource-scheduler

Concurrent Resource Scheduler (CRS) (Currently v1.2.2)

A high-performance, domain-agnostic Go resource scheduler built around sharded priority heaps, concurrent-safe lookup, pluggable acquire strategies, affinity routing, lifecycle management, cooldowns, and observability.

If you find the project useful, consider giving it a star.

Found a bug? Open an issue.

Have an idea? Start a discussion.

The best validation for a concurrency library isn't another local test.

It's seeing it survive workloads you didn't design yourself.

I Built a Concurrent Resource Scheduler in Go Using Sharded Priority Heaps

Feroz — Tue, 11 Aug 2026 15:05:06 +0000

Support on GitHub: github.com/phero20/concurrent-resource-scheduler (Give it a star if you find it useful!)

View Docs: pkg.go.dev/github.com/phero20/concurrent-resource-scheduler

What happens when thousands of concurrent requests compete for a small pool of reusable resources?

You can put a mutex around a slice and hope for the best.

Or you can design the scheduler around concurrency from the beginning.

I chose the second option.

I built Concurrent Resource Scheduler (CRS), a domain-agnostic Go library for selecting, prioritizing, routing, and maintaining reusable resources under heavy concurrent load.

The core idea is simple:

                  MANY CONCURRENT REQUESTS
                           │
                           ▼
                  ┌───────────────────┐
                  │ Resource Scheduler│
                  └─────────┬─────────┘
                            │
             ┌──────────────┼──────────────┐
             │              │              │
             ▼              ▼              ▼
        Priority         Acquire        State
          Heap           Strategy      Management
             │              │              │
             └──────────────┼──────────────┘
                            │
                            ▼
                    BEST AVAILABLE
                       RESOURCE

But making that work correctly under concurrency is where things get interesting.

CRS is designed for use cases such as:

LLM/API gateways
API key pools
proxy rotation
database replicas
GPU workers
backend pools
worker resources
connection pools
rate-limited providers
reusable compute resources

The scheduler itself does not know what a resource means.

It only knows:

"I have resources. I need to safely maintain them, prioritize them, and return an appropriate one to a concurrent caller."

The Problem
The Naive Approach
Why a Global Mutex Becomes a Problem
The Core Idea Behind CRS
Architecture at a Glance
Sharded Priority Heaps
Why Sharding Helps
The O(1) Lookup Map
Priority and Acquire Are Different Problems
Acquire Strategies
Round Robin
Weighted Acquire
Adaptive Acquire
Affinity Routing
Shared vs Exclusive Acquisition
Resource Lifecycle
Atomic State Transitions
The Inactive Store
Batch Operations
Updates Without Destroying Heap Ordering
Cooldowns
Asynchronous Events
Observability
Prometheus Integration
Concurrency Model
Complexity
Testing the Library
Race Detector Validation
Real-World Load Testing
10,000 Concurrent Workers
Burst Testing
Failure Testing
Cooldown Stress Testing
What the Load Tests Actually Tell Us
A Minimal Example
LLM Gateway Example
Why CRS Is Domain-Agnostic
Project Structure
Design Principles
Lessons Learned
When You Should NOT Use CRS
Future Directions
Final Thoughts

The Problem

Let's start with a realistic scenario.

Imagine an LLM gateway with 100 API keys.

Each key may have different:

rate-limit availability
priority
health
cooldown state
provider
capacity
temporary availability

Thousands of requests arrive concurrently.

A simplified system looks like:

                   ┌──────────────┐
Request 1 ────────►              │
Request 2 ────────►              │
Request 3 ────────►   Gateway    │
Request 4 ────────►              │
Request 5 ────────►              │
   ...             │              │
Request N ────────►              │
                   └──────┬───────┘
                          │
                          ▼
                  ┌───────────────┐
                  │ Resource Pool │
                  └───────┬───────┘
                          │
           ┌──────────────┼──────────────┐
           ▼              ▼              ▼
        API Key 1      API Key 2      API Key N

The scheduler now has to answer:

Which resource should this request use?
Which resource has the best priority?
Is the resource currently active?
Can multiple requests use it simultaneously?
Should this resource temporarily leave the pool?
Which shard should we search?
Should requests stick to the same shard?
What happens when the resource is released?
How do we update its priority?
How do we observe all of this without slowing down the hot path?

That is the problem CRS tries to solve.

The Naive Approach

The easiest implementation looks something like:

type Scheduler struct {
    mu        sync.Mutex
    resources []*Resource
}

Then:

func (s *Scheduler) Acquire() *Resource {
    s.mu.Lock()
    defer s.mu.Unlock()

    // Scan resources.
    // Find the best one.
    // Return it.

    return best
}

At first glance, this looks perfectly reasonable.

For 10 resources and 2 goroutines, it probably is.

But imagine:

Resources:           10,000
Concurrent requests: 5,000

Now every operation fights over one lock.

                GLOBAL MUTEX
                     │
       ┌─────────────┼─────────────┐
       │             │             │
       ▼             ▼             ▼
    Worker 1      Worker 2      Worker 3
       │             │             │
       └─────────────┼─────────────┘
                     │
                  WAITING

The scheduler becomes serialized around the lock.

Why a Global Mutex Becomes a Problem

There are several problems.

1. Lock contention

Only one goroutine can manipulate the pool at a time.

2. Linear scanning

If resources are stored in an array, finding the best resource can become:

O(N)

per acquisition.

3. Priority maintenance

If resources have priorities that change, the scheduler has to continuously maintain ordering.

4. State transitions

Resources can move between:

ACTIVE
INACTIVE
REMOVED

and those transitions must be synchronized.

5. Observability

Metrics and event callbacks should not block the scheduler.

The challenge is therefore not just:

"How do I build a priority queue?"

It is:

"How do I build a concurrent priority resource manager where priority, acquire, lifecycle, and observability coexist?"

The Core Idea Behind CRS

The central architectural decision was:

Don't put one global lock around the entire priority structure.

Instead, CRS partitions resources into independently locked shards.

Conceptually:

                  CRS
                   │
        ┌──────────┼──────────┐
        │          │          │
        ▼          ▼          ▼
     Shard 1    Shard 2    Shard N
        │          │          │
      Heap       Heap       Heap
        │          │          │
     Mutex      Mutex      Mutex

Each shard owns its own heap and its own lock.

This is the heart of CRS.

Architecture at a Glance

                        APPLICATION
                             │
               Add / Acquire / Release / Update
                             │
                             ▼
                  ┌────────────────────┐
                  │   CRS Scheduler    │
                  └─────────┬──────────┘
                            │
            ┌───────────────┼────────────────┐
            │               │                │
            ▼               ▼                ▼
      Acquire          Lookup          Inactive
       Strategy            Map             Store
            │               │                │
            ▼               ▼                │
     Candidate Shard     O(1) Node          │
            │                                │
            ▼                                │
    ┌─────────────────────────────────┐      │
    │         ACTIVE HEAP SHARDS       │      │
    │                                 │      │
    │  Heap 1     Heap 2     Heap N   │      │
    │  +Mutex     +Mutex     +Mutex   │      │
    └─────────────────────────────────┘      │
            │                                │
            └──────────────┬─────────────────┘
                           │
                           ▼
                    EVENT DISPATCHER
                           │
                 ┌─────────┴──────────┐
                 ▼                    ▼
             Telemetry            Cooldown
                 │                    │
                 ▼                    ▼
             Prometheus          Resource State

This separation is intentional.

Sharded Priority Heaps

Each shard maintains a priority heap.

For example:

Shard 1

        [Priority 10]
        /           \
 [Priority 20]    [Priority 30]
    /     \
[40]      [50]

Another shard:

Shard 2

        [Priority 5]
        /          \
 [Priority 15]   [Priority 25]

Every shard has its own synchronization boundary.

Shard 1                 Shard 2                 Shard 3

┌─────────────┐        ┌─────────────┐        ┌─────────────┐
│   Mutex     │        │   Mutex     │        │   Mutex     │
├─────────────┤        ├─────────────┤        ├─────────────┤
│ Priority    │        │ Priority    │        │ Priority    │
│ Heap        │        │ Heap        │        │ Heap        │
└─────────────┘        └─────────────┘        └─────────────┘

There is no global heap mutex.

Why Sharding Helps

Suppose we have 32 shards.

Instead of:

1 global lock

we have:

32 independently locked heaps

Different goroutines can operate on different shards simultaneously.

Goroutine A ─────► Shard 1 ─────► lock
Goroutine B ─────► Shard 7 ─────► lock
Goroutine C ─────► Shard 19 ────► lock
Goroutine D ─────► Shard 27 ────► lock

The locks are independent.

This doesn't magically eliminate contention.

If every request targets the same shard, that shard can still become contended.

That's why CRS also separates:

acquire strategy

from

priority ordering

This distinction is extremely important.

The O(1) Lookup Map

A heap is excellent at answering:

"What is the best resource?"

But a heap is not ideal for answering:

"Where is resource X?"

Searching a heap can require scanning.

CRS therefore maintains an additional lookup structure.

Conceptually:

ID
 │
 ▼
┌──────────────────────┐
│ Lookup Map            │
│                      │
│ "backend-01" ───────► Node
│ "backend-02" ───────► Node
│ "backend-03" ───────► Node
└──────────────────────┘

The lookup map is protected independently with a read/write mutex.

This gives the scheduler an O(1)-style membership/location lookup by application-defined key.

That is particularly useful for:

Get
Update
Remove
Release
Exclude
Include

without scanning every heap.

Priority and Acquire Are Different Problems

This is one of the most important design ideas in CRS.

A resource can have:

Priority = 10

but that doesn't necessarily tell us:

Which shard should we inspect first?

These are separate decisions.

CRS therefore separates:

                REQUEST
                   │
                   ▼
            ACQUIRE STRATEGY
                   │
                   ▼
              SHARD SELECTION
                   │
                   ▼
             PRIORITY HEAP
                   │
                   ▼
             BEST RESOURCE

This allows different routing strategies to be plugged into the scheduler without changing the underlying heap implementation.

Acquire Strategies

CRS provides several acquire approaches:

Round Robin
Weighted
Adaptive
Consistent Hashing for affinity routing

Each solves a different problem.

Round Robin

Round Robin is the simplest.

Request 1 → Shard 1
Request 2 → Shard 2
Request 3 → Shard 3
Request 4 → Shard 4
Request 5 → Shard 1
...

It is simple and predictable.

Use it when:

shards are roughly equivalent
you want even distribution
resource capacity is similar

Weighted Acquire

Not every shard is necessarily equal.

Imagine:

GPU 1 → 24 GB VRAM
GPU 2 → 24 GB VRAM
GPU 3 → 80 GB VRAM
GPU 4 → 80 GB VRAM

You may want larger resources to receive more work.

Weighted acquire lets you express relative capacity.

Conceptually:

Shard 1: weight 1
Shard 2: weight 1
Shard 3: weight 4
Shard 4: weight 4

Traffic can then be distributed proportionally.

This is useful for:

heterogeneous GPUs
backend instances with different capacity
API providers with different quotas
worker pools with different performance characteristics

Adaptive Acquire

Round Robin doesn't know anything about current load.

Adaptive acquire attempts to account for shard activity.

Conceptually:

             REQUEST
                │
                ▼
       ┌──────────────────┐
       │ Inspect shard    │
       │ load information  │
       └────────┬─────────┘
                │
       ┌────────┼────────┐
       ▼        ▼        ▼
    Shard A   Shard B   Shard C
      busy      low       busy
                │
                ▼
            choose B

The scheduler uses lightweight shard-level state to favor less-contended shards without introducing another global lock.

This is useful when the resource pool is dynamic and simple round-robin distribution isn't enough.

Affinity Routing

Sometimes you don't want random distribution.

You want:

user-123 → same shard
user-456 → same shard

CRS supports affinity routing through consistent hashing.

Conceptually:

                   HASH RING

             ┌───────────────────┐
             │                   │
        S1   │        S2         │
             │                   │
             │                   │
        S4   │        S3         │
             │                   │
             └───────────────────┘

              ▲
              │
        hash("user-123")

The same affinity identifier deterministically maps to the same shard.

This is useful for:

sticky sessions
tenant affinity
cache locality
connection locality
stateful workers

Shared vs Exclusive Acquisition

CRS supports two major acquisition semantics.

Shared

The resource remains active.

ACTIVE
  │
  │ Acquire
  ▼
ACTIVE

Multiple callers can acquire the same resource.

This is useful when resources represent things like:

API keys
read replicas
stateless endpoints
shared provider capacity

Exclusive

The resource temporarily leaves the active pool.

ACTIVE
   │
   │ Acquire
   ▼
INACTIVE
   │
   │ Release
   ▼
ACTIVE

This is useful when a resource represents something that cannot safely be used by multiple concurrent operations.

Examples:

GPU worker
exclusive connection
physical device
single-use worker
exclusive job executor

Resource Lifecycle

A CRS resource essentially moves through states.

                  ┌───────────┐
                  │   ADD     │
                  └─────┬─────┘
                        ▼
                 ┌─────────────┐
                 │   ACTIVE    │
                 └──────┬──────┘
                        │
              ┌─────────┼─────────┐
              │         │         │
           Acquire    Exclude   Remove
              │         │         │
              ▼         ▼         ▼
          INACTIVE   INACTIVE   DELETED
              │
            Release
              │
              ▼
           ACTIVE

The important invariant is:

A resource should exist in exactly one state/location at a time.

Atomic State Transitions

Imagine:

Goroutine A: Acquire(resource)
Goroutine B: Remove(resource)
Goroutine C: Update(resource)

all happening at almost the same time.

Without careful synchronization, you can get:

Resource exists in heap
AND
Resource exists in inactive store

or:

Lookup map says ACTIVE
but heap doesn't contain it

Those are catastrophic consistency bugs.

CRS therefore treats state transitions as carefully synchronized operations involving:

lookup state
heap state
inactive state
shard locks

The Inactive Store

The inactive store is particularly important for Exclusive acquisition.

Suppose:

backend-01

is acquired exclusively.

It is removed from the active heap and stored as inactive.

ACTIVE HEAP

backend-01
backend-02
backend-03

After acquisition:

ACTIVE HEAP

backend-02
backend-03


INACTIVE STORE

backend-01

When released:

INACTIVE STORE
      │
      │ Release
      ▼
ACTIVE HEAP

The lookup map allows CRS to locate the resource without scanning every heap.

Batch Operations

Adding resources one at a time is easy.

But imagine importing 10,000 resources.

You don't want partial state like:

Batch:
1 ✓
2 ✓
3 ✓
4 ✓
5 ✗
6 ?
7 ?
...

CRS provides BatchAdd with atomic insertion behavior.

Conceptually:

                BatchAdd
                   │
                   ▼
          ┌─────────────────┐
          │ Validate batch  │
          └────────┬────────┘
                   │
                   ▼
          ┌─────────────────┐
          │ Prepare changes │
          └────────┬────────┘
                   │
                   ▼
          ┌─────────────────┐
          │ Insert shards   │
          └────────┬────────┘
                   │
                   ▼
              COMPLETE

The goal is to avoid exposing a partially inserted batch.

Updates Without Destroying Heap Ordering

Suppose:

A = priority 10
B = priority 20
C = priority 30

Now:

C → priority 5

C should move toward the top.

Simply changing the value isn't enough.

The heap must be repaired.

Conceptually:

Before:

        A(10)
       /     \
    B(20)   C(30)


Update:

C(30) → C(5)


After:

        C(5)
       /    \
    A(10)  B(20)

This preserves the priority-queue invariant.

Cooldowns

Real resources sometimes need a cooldown period.

For example:

API key hits rate limit
        │
        ▼
   cooldown 5s
        │
        ▼
available again

CRS includes a cooldown extension.

The architecture is event-driven:

Acquire
   │
   ▼
Release
   │
   ▼
Event Dispatcher
   │
   ▼
Cooldown Manager
   │
   ▼
Exclude resource
   │
   ▼
wait
   │
   ▼
Include resource

The cooldown extension is asynchronous. That means there can be a small eventual-consistency window between a release event and the cooldown observer processing it.

That is intentional.

Asynchronous Events

Observability and extensions should not unnecessarily slow the scheduler's hot path.

CRS therefore uses an event dispatcher.

                 Scheduler
                     │
                     │ emit event
                     ▼
             ┌───────────────┐
             │ Buffered      │
             │ Event Stream  │
             └───────┬───────┘
                     │
              background worker
                     │
          ┌──────────┼──────────┐
          ▼          ▼          ▼
      Observer 1  Observer 2  Observer 3
          │          │          │
          ▼          ▼          ▼
       Metrics   Cooldown   Prometheus

The scheduler doesn't need to execute arbitrary observer logic while holding heap locks.

Observability

A production scheduler should answer:

How many acquisitions happened?
How many releases?
Which resources are being used?
How many failures?
What is the current resource count?

CRS includes telemetry support using atomic counters and asynchronous events.

The goal is:

Scheduler hot path
      │
      ▼
lightweight event
      │
      ▼
asynchronous telemetry

rather than doing expensive monitoring work while holding scheduler locks.

Prometheus Integration

CRS provides a Prometheus exporter extension.

Conceptually:

               CRS
                │
                ▼
          Telemetry
                │
                ▼
      ┌─────────────────┐
      │ Atomic Counters │
      └────────┬────────┘
               │
               ▼
       Prometheus Collector
               │
               ▼
       /metrics endpoint

This makes it possible to expose scheduler activity to an existing monitoring stack.

Concurrency Model

The concurrency model is based on multiple independent synchronization boundaries.

                   Scheduler
                       │
          ┌────────────┼────────────┐
          │            │            │
          ▼            ▼            ▼
      Shard 1      Shard 2       Shard N
       Mutex         Mutex         Mutex
          │            │            │
          ▼            ▼            ▼
        Heap          Heap          Heap

Separately:

Lookup Map
    │
sync.RWMutex

And:

Telemetry
    │
atomic counters / event channel

This is much more granular than one mutex around everything.

Complexity

The scheduler is designed around heap and lookup properties.

Operation	Complexity	Synchronization
Add	O(log N)	Single shard lock
BatchAdd	O(log N) per insertion	Shard locks
Acquire	Acquire + heap operation	Shard lock
AcquireByAffinity	Hash lookup + heap operation	Shard lock
Release	O(log N)	Single shard lock
Update	O(log N) active / O(1) inactive	Shard lock
Remove	O(log N) active / O(1) inactive	Shard lock
Get	O(1)-style lookup	Lookup synchronization
Stats	O(Shards)	Short shard reads

Where:

N = resources in a shard

Actual performance depends on workload, shard count, resource distribution, acquire policy, and contention.

Testing the Library

A concurrency library cannot be validated with only:

go test ./...

Unit tests answer:

"Does this operation behave correctly?"

Load tests answer:

"What happens when thousands of goroutines continuously hammer it?"

The CRS test strategy includes:

                  TESTING
                     │
       ┌─────────────┼─────────────┐
       │             │             │
       ▼             ▼             ▼
   Unit Tests    Race Tests    Load Tests
       │             │             │
       ▼             ▼             ▼
  correctness    data races    behavior

Tests cover:

scheduler construction
configuration validation
Add
BatchAdd
Get
Len
Stats
Acquire
Release
Update
Include
Exclude
Remove
Shutdown
acquire strategies
event observers
cooldown
telemetry
Prometheus collection
concurrent stress behavior

Race Detector Validation

One of the most important commands for a concurrent Go library is:

go test -count=1 -race ./...

The race detector doesn't prove that a library is bug-free.

But it can catch a class of extremely dangerous concurrency problems that ordinary tests may miss.

The final release was validated with the race detector without reported data races.

Real-World Load Testing

For CRS, I created a dedicated load-test harness.

The harness models:

concurrent workers
backend processing delay
random request durations
backend failures
request cancellation
different scenarios
resource utilization
acquisition latency
backend latency
total latency
throughput
cooldown behavior
burst traffic

The important distinction is that these are load-test results, not universal benchmarks.

10,000 Concurrent Workers

One of the larger tests used:

Concurrency: 10,000 workers
Resources:   4 backends
Duration:    60 seconds
Scenario:    normal
Acquire:     adaptive
Policy:      shared
Race detector: enabled

The test produced approximately:

Total requests:        2,248,610
Successful:            2,226,165
Failed:                22,445

Throughput:            ~37,137.28 req/s

Peak simultaneous:     10,000

Resource utilization was extremely well-balanced:

backend-01    25.01%
backend-02    24.96%
backend-03    25.03%
backend-04    25.00%

The scheduler acquisition latency was remarkably low, and should be clearly distinguished from the simulated backend latency:

Acquire Latency:
mean: 7.59µs
max:  51.893ms

Backend Latency:
p50: 260.334ms
p95: 477.661ms
p99: 496.863ms

Total Request Latency:
p50: 262.430ms
p95: 479.425ms
p99: 498.559ms

Accounting check verified complete consistency:

success + backend failures = 2,248,610
backend requests = 2,248,610
total request attempts = 2,248,610
accounting: OK

The test environment was:

OS:           Windows
Architecture: amd64
CPU cores:    12
GOMAXPROCS:   12
Go:           1.25.5

The backend latency was simulated and should not be confused with CRS scheduler latency. These are workload-specific test results and should NOT be presented as universal benchmark claims.

Burst Testing

Real systems often receive bursts.

Traffic
  │
  │       ███████████
  │       ███████████
  │       ███████████
  │
  │  ███
  │  ███
  │  ███
  └────────────────────────► time

The burst scenario generated approximately:

Requests:        468,791
Successful:      464,109
Failed:          4,682

Throughput:      ~7,746 req/s

Peak simultaneous: 5,000

The success rate was approximately:

99%

Backend latency remained around:

p50 ≈ 260 ms
p95 ≈ 476 ms
p99 ≈ 495 ms

These backend numbers came from the simulated workload.

Failure Testing

A scheduler should also behave correctly when resources fail.

I tested a failure scenario with:

Concurrency: 1000
Resources:   4
Duration:    60 seconds
Failure rate: 10%

The workload produced approximately:

Requests:       231,443
Successful:     208,293
Failed:          23,150
Success rate:    90%

The scheduler reported:

Acquire failures: 0
Release failures: 0
Backend failures: 23,150

That distinction matters.

The scheduler successfully acquired resources while simulated backend operations failed at the expected rate.

Cooldown Stress Testing

Cooldown was one of the more interesting tests.

With exclusive acquisition, resources temporarily leave the active pool.

                4 resources
                     │
                     ▼
              ┌─────────────┐
              │   ACTIVE    │
              └──────┬──────┘
                     │
                Acquire
                     │
                     ▼
              ┌─────────────┐
              │  INACTIVE   │
              └──────┬──────┘
                     │
                  cooldown
                     │
                     ▼
              ┌─────────────┐
              │   ACTIVE    │
              └─────────────┘

At 1,000 concurrent workers, the scheduler generated a very large number of acquisition attempts while only a limited number of resources were available.

This demonstrated an important property:

High concurrency doesn't mean unlimited successful backend concurrency.

If only four resources exist and the policy is exclusive, four resources are still four resources.

What the Load Tests Actually Tell Us

The load tests gave several useful observations.

1. Sharding worked under heavy concurrency

The scheduler continued operating with thousands of concurrent workers without race-detector failures.

2. Acquire remained balanced

The adaptive workload distributed acquisitions across four backends at approximately 25% each.

3. Resource state remained consistent

The accounting checks showed that acquired resources were not silently lost.

4. Failures remained distinguishable

The harness separated:

Acquire failure
Backend failure
Release failure
Timeout

A scheduler failure and a backend failure are very different operational problems.

5. Cooldown changes the workload completely

With exclusive resources, the bottleneck becomes resource availability rather than CPU.

A Minimal Example

The basic usage pattern is:

package main

import (
    "fmt"
    "log"

    "github.com/phero20/concurrent-resource-scheduler/config"
    "github.com/phero20/concurrent-resource-scheduler/scheduler"
)

type Worker struct {
    ID       string
    Priority int
}

func main() {
    compare := func(a, b *Worker) int {
        if a.Priority < b.Priority {
            return -1
        }

        if a.Priority > b.Priority {
            return 1
        }

        return 0
    }

    keyFunc := func(w *Worker) string {
        return w.ID
    }

    cfg := config.Config[*Worker, string]{
        HeapCount: 1,
        Comparator: compare,
        KeyFunc:    keyFunc,
    }

    sched, err := scheduler.New(cfg)
    if err != nil {
        log.Fatal(err)
    }

    defer sched.Shutdown()

    sched.Add(&Worker{
        ID:       "worker-1",
        Priority: 50,
    })

    sched.Add(&Worker{
        ID:       "worker-2",
        Priority: 10,
    })

    resource, err := sched.Acquire()

    if err != nil {
        log.Fatal(err)
    }

    fmt.Println("Acquired:", resource.ID)
}

The application controls the resource type, key, and comparison logic.

The scheduler handles the concurrent resource management.

LLM Gateway Example

Imagine:

type APIKey struct {
    ID        string
    Provider  string
    Remaining int
}

Your application could define priority based on remaining quota.

Conceptually:

API Key
   │
   ├── Provider
   ├── Remaining quota
   └── Health

The comparator could make a key with more remaining capacity more desirable.

The scheduler doesn't need to know what those fields mean.

The application owns that logic.

CRS maintains the ordering and concurrent lifecycle.

This is the core idea behind being domain-agnostic.

Why CRS Is Domain-Agnostic

The scheduler doesn't contain logic like:

if resource.IsGPU() {
    ...
}

or:

if provider == "openai" {
    ...
}

Instead, the application provides:

Resource type
     +
Key function
     +
Comparator
     +
Acquire strategy

This makes the same scheduler applicable to many domains.

Example: GPU Workers

                Job
                 │
                 ▼
          ┌─────────────┐
          │ CRS         │
          └──────┬──────┘
                 │
       ┌─────────┼─────────┐
       ▼         ▼         ▼
      GPU 1     GPU 2     GPU 3
      24 GB     24 GB     80 GB

Weighted acquire could favor larger GPUs.

Exclusive acquisition could prevent two jobs from taking the same GPU.

Example: Database Replicas

                 Query
                   │
                   ▼
                 CRS
                   │
        ┌──────────┼──────────┐
        ▼          ▼          ▼
      DB-1       DB-2       DB-3
       20%        70%        35%

The application could define priority around current load.

The scheduler remains unaware that the resources happen to be databases.

Example: Proxy Pool

                  HTTP Request
                       │
                       ▼
                     CRS
                       │
        ┌──────────────┼──────────────┐
        ▼              ▼              ▼
    Proxy A         Proxy B        Proxy C
    healthy         cooldown       healthy

Cooldown can temporarily remove a failing proxy.

Affinity can keep a particular tenant or session mapped consistently.

Project Structure

concurrent-resource-scheduler/
│
├── config/
├── scheduler/
├── acquire/
├── internal/
│   ├── heap/
│   ├── lookup/
│   └── node/
├── extensions/
│   ├── cooldown/
│   ├── metrics/
│   └── prometheus/
├── events/
├── errors/
├── stats/
├── examples/
│   ├── basic/
│   ├── batch/
│   ├── affinity/
│   ├── cooldown/
│   ├── exclusive/
│   ├── shared/
│   └── prometheus/
├── docs/
│   ├── API.md
│   ├── ARCHITECTURE.md
│   ├── OVERVIEW.md
├── README.md
├── CHANGELOG.md
├── CONTRIBUTING.md
├── LICENSE
└── go.mod

The main architectural boundary is:

Public API
    │
    ▼
Scheduler
    │
    ├── Acquire
    ├── Heap
    ├── Lookup
    ├── Events
    └── Extensions

Design Principles

Several principles shaped CRS.

1. No global heap lock

The heap state is partitioned.

2. Separate routing from priority

Acquire chooses where to look.

The heap chooses what is best.

3. Keep the scheduler domain-agnostic

The application owns business logic.

4. Keep observability outside the hot path

Telemetry should not become the scheduler bottleneck.

5. Make state transitions explicit

Resources are either active, inactive, or removed.

6. Optimize for concurrent workloads

Concurrency is not an afterthought.

It is part of the architecture.

Lessons Learned

Building a concurrent library taught me something important:

Concurrency problems are usually not caused by one complicated function.

They are caused by interactions between simple functions.

For example:

Acquire
   +
Release
   +
Update
   +
Remove
   +
Cooldown
   +
Events
   +
Concurrent callers

Each function can be correct individually while the combination is broken.

That's why concurrency testing needs to go beyond unit tests.

Another Lesson: Observability Is Part of the Design

When you have:

10,000 requests
4 resources
multiple shards
multiple goroutines
failures
cooldowns

you need to know what actually happened.

That's why CRS exposes events and telemetry around scheduler activity.

Without observability, debugging concurrent systems becomes guesswork.

Another Lesson: Load Tests Need Accounting

One of the most useful parts of the load harness was accounting.

For example:

total requests
      │
      ├── acquire failures
      │
      └── reached backend
              │
              ├── successful
              └── backend failure

If:

success + backend_failure

doesn't match:

backend_requests

something is wrong.

Accounting checks are powerful for detecting concurrency bugs.

When You Should NOT Use CRS

CRS isn't intended to replace every queue or pool implementation.

You probably don't need it if you simply have:

10 objects
+
one goroutine

A simple slice may be better.

You also probably don't need CRS if your problem is purely:

"I need a FIFO queue."

Use a simpler primitive.

CRS becomes interesting when you need combinations of:

concurrency
+
priority
+
acquire
+
resource lifecycle
+
affinity
+
cooldowns
+
observability

Future Directions

Potential future areas include:

deeper benchmarking
additional acquire strategies
richer scheduling policies
more advanced resource health models
better operational tooling
additional observability integrations
workload-specific tuning
broader real-world validation

The goal is not to optimize for feature count.

The goal is to keep the core scheduler:

small
predictable
composable
concurrent
domain-agnostic

Installation

Install CRS with:

go get github.com/phero20/concurrent-resource-scheduler

Then explore the examples and documentation in the repository.

Final Thoughts

The interesting part of building a scheduler isn't writing:

Acquire()

The interesting part is everything around it.

You need to think about:

                    RESOURCE SCHEDULING

                         ┌─────────┐
                         │ Priority│
                         └────┬────┘
                              │
        ┌─────────────────────┼─────────────────────┐
        │                     │                     │
        ▼                     ▼                     ▼
   Concurrency           Acquire              Lifecycle
        │                     │                     │
        ▼                     ▼                     ▼
   Sharded heaps       Adaptive/Weighted       Active/Inactive
        │               Round Robin             Release
        │                     │                     │
        └─────────────────────┼─────────────────────┘
                              │
                              ▼
                         Observability
                              │
                   ┌──────────┴──────────┐
                   ▼                     ▼
               Telemetry            Prometheus

CRS was built around that complete picture.

It is not just a priority queue.

It is a concurrent resource-management layer that combines:

sharded priority heaps
concurrent-safe lookup
configurable acquire
priority ordering
shared and exclusive acquisition
affinity routing
resource lifecycle management
cooldown extensions
asynchronous events
telemetry
Prometheus integration

And the project has been tested beyond basic unit tests, including:

Race detector
       +
Concurrent stress tests
       +
1,000 workers
       +
2,000 workers
       +
5,000 workers
       +
10,000 workers
       +
Burst workloads
       +
Failure workloads
       +
Cooldown workloads

The most important result isn't a single throughput number.

It's that the architecture gives me a foundation where concurrency, scheduling policy, resource state, and observability can evolve independently.

That's what I wanted to build.

If You're Building Something Similar

I'd love to hear how you approach resource scheduling.

Especially if you're working on:

LLM gateways
GPU schedulers
proxy pools
database routing
distributed workers
API key rotation
connection pools
high-concurrency Go services

What would you change about this architecture?

What workloads should I test next?

What concurrency problems have you encountered in production?

I'd genuinely like to hear from people who have operated systems like these in the real world.

Project

Support on GitHub: github.com/phero20/concurrent-resource-scheduler (Give it a star if you find it useful!)

View Docs: pkg.go.dev/github.com/phero20/concurrent-resource-scheduler

Concurrent Resource Scheduler (CRS)

If you find the project useful, consider giving it a star.

Found a bug? Open an issue.

Have an idea? Start a discussion.

The best validation for a concurrency library isn't another local test.

It's seeing it survive workloads you didn't design yourself.

SlaveCode Featured in the Official Judge0 Repository

Feroz — Thu, 16 Jul 2026 13:07:19 +0000

Explore the Platform: slavecode.codes
Support on GitHub: github.com/phero20/slavecode (Give it a star)

Recently, I woke up to an unexpected GitHub notification. Herman Zvonimir Došilović, the creator of Judge0, opened an issue on the SlaveCode repository to say he enjoyed the project.

Even better, he added the SlaveCode repository to the official Judge0 README.md showcase.

Judge0 is one of the most widely used open-source code execution engines available, and it serves as a critical component in SlaveCode's architecture. Having the project recognized by its creator is a great milestone. I wanted to share a brief overview of why we chose Judge0 and how it fits into our system design.

Why Judge0?

When building SlaveCode, a platform that compiles and executes user-submitted code in real-time, security and isolation were the highest priorities. Running arbitrary code safely requires a robust sandbox environment.

As detailed in my Full Architecture Walkthrough, SlaveCode relies on a strict multi-cloud topology. While the primary REST API and databases are hosted on Google Cloud Platform (GCP), the code execution engine is completely isolated on Microsoft Azure VMs.

Judge0 was the perfect fit for this isolated environment. By leveraging its heavily containerized execution environment on Azure, we ensure that the execution sandbox is physically separated from our core databases and API infrastructure. This separation of concerns means that the blast radius of any potential sandbox escape is strictly contained, keeping the main platform and user data completely secure.

By offloading the complexities of secure compilation and execution to Judge0, the SlaveCode backend is free to focus entirely on its core platform logic and features.

The Value of Building in Public

This recognition reinforces an important lesson: documenting your engineering decisions is just as important as the code itself.

Herman likely discovered SlaveCode because I took the time to write a detailed architectural breakdown of the platform on Dev.to. Sharing the reasoning behind our technology stack, the trade-offs we accepted, and the problems we solved helped the project reach the right people in the open-source community.

If you are building a platform that requires secure, scalable code execution, I highly recommend checking out Judge0 and giving the repository a star.

Thank you to Herman and the open-source community for providing the tools that make projects like SlaveCode possible.

(If you like the project, I would love a star on the SlaveCode GitHub repository!)

I Built the Ultimate All-in-One Platform for Software Engineers — Here's the Full Architecture

Feroz — Wed, 08 Jul 2026 14:36:16 +0000

Explore the Platform: slavecode.codes

Support on GitHub: github.com/phero20 (Give it a star)

"Your code is your master. Serve it well."

That's the philosophy I built this around. Not just a tagline — a design constraint. Every system decision in SlaveCode comes back to that principle: the code should be judged fairly, the infrastructure should serve users well, and nothing should break under pressure.

SlaveCode is a competitive programming platform I built from the ground up. It is not a weekend project. It is a full production system with:

11,000+ coding problems across multiple difficulty levels
82+ language learning tracks in the Academy (from Go to Haskell to Gleam)
Real-time multiplayer Arena for up to 50 players simultaneously
460+ company interview profiles (Google, Meta, Amazon, Apple, and more)
56-topic System Design curriculum with an AI-powered diagram workspace
A global contest aggregator pulling from LeetCode, Codeforces, AtCoder, CodeChef
A compiler playground available to guests with zero login required

This post is a full architecture walkthrough. Not "here's what I used" — but why every decision was made, what breaks if you get it wrong, and how it all fits together.

The Big Picture: A Multi-Cloud Architecture

Before diving into any single system, it's worth understanding the physical topology first. Where things run matters as much as how they run.

┌──────────────────────────────────────────────────────────────────────┐
│                        VERCEL EDGE NETWORK                           │
│  Next.js 15 (App Router) — SSR + Static + CDN-distributed assets    │
└───────────────────┬───────────────────────────────┬─────────────────┘
                    │ HTTPS REST                     │ WSS WebSockets
                    ▼                                ▼
┌────────────────────────────────────────────────────────────────────┐
│                   GOOGLE CLOUD PLATFORM (Compute Engine VMs)        │
│                                                                      │
│   ┌──────────────────┐   ┌───────────────────┐   ┌─────────────┐   │
│   │  Hono API Server │   │  Golang Arena Hub │   │ BullMQ      │   │
│   │  (Bun runtime)   │   │  (WebSocket mgr)  │   │ Workers     │   │
│   └──────────────────┘   └───────────────────┘   └─────────────┘   │
└──────────┬─────────────────────────────────────────────┬────────────┘
           │ Redis Protocol                              │ HTTP POST
           ▼                                             ▼
┌──────────────────────────┐          ┌─────────────────────────────┐
│  MANAGED DBaaS           │          │  MICROSOFT AZURE (VMs)      │
│  ┌──────────────────────┐│          │  ┌─────────────────────────┐│
│  │ Neon (Postgres)      ││          │  │ Judge0 Code Sandbox     ││
│  │ MongoDB Atlas        ││          │  │ (Dockerized, isolated)  ││
│  │ Aiven Valkey (Redis) ││          │  └─────────────────────────┘│
│  └──────────────────────┘│          └─────────────────────────────┘
└──────────────────────────┘

There are three intentional multi-cloud decisions here:

Why GCP for the API and Arena? Raw Compute Engine VMs give persistent connections, no cold starts, and no 10-second serverless timeout that would kill a long-running BullMQ worker mid-job. Vercel serverless functions cannot hold a BullMQ consumer — period.

Why Azure for Judge0? This is the most important security decision in the entire platform. The code execution sandbox runs on an entirely separate cloud provider (Azure), isolated from the primary databases (GCP). If a sophisticated attacker crafts a zero-day Docker escape exploit and compromises the Judge0 VM, they land on Azure — not inside the same VPC as the API, Postgres, or MongoDB. The "blast radius" is physically separated across two clouds.

Why Vercel for the frontend? Edge CDN distribution, automatic CI/CD on push, SSR support, and zero infrastructure management for static assets. It offloads a dimension of complexity I do not need to operate.

The Complete Tech Stack

Layer	Technology	Why
Frontend	Next.js 15 (App Router), React, TailwindCSS, Zustand	App Router enables co-located server components for SEO + client islands for interactivity
Code Editor	Monaco Editor	Same engine as VS Code — mature, syntax-aware, extensible
Whiteboard	Tldraw (infinite canvas)	Best-in-class React-native infinite canvas for System Design diagrams
REST API	Hono on Bun	Hono is 2-3× faster than Express on Bun. Bun's native speed matters for hot paths
Dependency Injection	Awilix	Enables clean service-layer architecture at scale without tight coupling
Multiplayer Engine	Golang (Fiber, WebSockets)	Go goroutines handle tens of thousands of persistent WebSocket connections with far less overhead than Node
Background Jobs	BullMQ (Bun Workers)	Reliable Redis-backed queue with retries, delays, and priority scheduling
Relational DB	PostgreSQL (Neon serverless)	Serverless Postgres — scales to zero, no idle VM cost
Document DB	MongoDB Atlas (Mongoose)	Flexible schema for problems, test cases, submissions — highly nested JSON
In-Memory Store	Aiven Valkey (Redis fork)	Open-source Redis drop-in, handles cache + queue + pub/sub + arena state
Auth	Clerk	Cryptographic JWT edge verification, OAuth, webhook user sync
Code Execution	Judge0 (primary) + Wandbox (fallback)	Dockerized sandbox that compiles and runs untrusted code safely
AI Integration	Unified AI proxy (multiple providers)	Key rotation, load balancing, and automatic provider failover
ORM	Drizzle ORM (Postgres) + Mongoose (MongoDB)	Type-safe SQL generation for Postgres; Mongoose for Mongo document modeling
Image Storage	Cloudinary	Bug report screenshots, user avatars

Database Strategy: Why Three Databases?

This is the question I get most often. The answer is that the three databases own fundamentally different shapes of data, and forcing one data model onto all three use cases would be an architectural mistake.

PostgreSQL (Neon) — The Social Graph & Analytics Engine

Postgres owns everything that requires joins, aggregations, and referential integrity:

-- A simplified view of what Postgres owns
users           → user_stats         (1:1)
users           → user_activity      (1:many — daily heatmap)
users           → user_solved_problems (many:many with Mongo IDs as text FK)
users           → follows            (self-referential many:many)
users           → solutions          (community editorial posts)
categories      → category_problems  (hierarchical DSA roadmap nodes)
contests        (aggregated from external APIs by a cron worker)

Notice user_solved_problems.problemId is a text column that stores a MongoDB ObjectId string. This is a cross-database foreign key — a deliberate trade-off. I get relational semantics for user stats without forcing the problem definitions into Postgres where they don't fit.

Drizzle ORM handles this with full type safety. The column is declared as text() but semantically treated as a foreign reference in the service layer.

MongoDB Atlas — The Content Engine

MongoDB owns everything that is large, nested, or schema-flexible:

Problem definitions — title, description, constraints, hints, editorial, companies tagged, difficulty, language support flags. A single problem document can be 20KB+ of nested JSON.
Hidden test cases — hundreds of {input, expectedOutput} pairs per problem that the judge evaluates against.
Submissions — the full execution log: source code, stdout per test, stderr, time and memory per test case, final verdict. This can easily reach 100KB per submission.
Academy tracks — deeply nested exercise trees with descriptions, hints, and starter code per language.
System Design workspaces — Tldraw serialized JSON documents. These are opaque blobs that Postgres has no business indexing.

Forcing this into Postgres would mean either (a) JSONB columns everywhere and losing schema enforcement, or (b) a schema migration nightmare every time a problem field changes.

Aiven Valkey (Redis) — Four Jobs, One Store

Redis is the most overloaded layer in the system. It does four distinct things:

Role	Key Pattern	TTL	Owner
Job Queue	`bull:submission-evaluation:*`	Until consumed	BullMQ Workers
Job Queue	`bull:arena-cleanup:*`	Until consumed	BullMQ Workers
Job Queue	`bull:contest-sync:*`	Until consumed	BullMQ Workers
Arena State	`arena:room:{roomId}`	24 hours	Go Arena Hub
Arena State	`arena:user_room:{userId}`	24 hours	Go Arena Hub
API Cache	`company:`, `llm:`, `diagram:*`	24h–7 days	Hono API
Pub/Sub	`arena:events:{roomId}`	N/A (channel)	API → Go
Pub/Sub	`arena:match:started:{roomId}`	N/A (channel)	API → Go
Pub/Sub	`arena:submission:{roomId}`	N/A (channel)	Worker → Go

The most interesting part is that Redis decouples services that would otherwise need to know each other's IP addresses. The Bun API and the Golang Arena Hub never talk directly. They shout into Redis, and the other side listens.

The Submission Pipeline: Async Code Evaluation

This is the most complex workflow in the platform. When a user clicks "Submit", this is what actually happens:

Phase 1: Synchronous Fast Path (the API's job)

Client → POST /submissions/submit { sourceCode, languageId, problemId }
       → SubmissionService: Insert { status: "PENDING" } into MongoDB
       → BullMQ: Enqueue job { submissionId }
       → Response: 202 Accepted { submissionId }

The API responds in ~50ms. It does not wait for compilation. It cannot — compilation can take 1–30 seconds depending on the language and problem complexity.

The client immediately starts polling:

// Client-side polling — every 2 seconds
const poll = async (submissionId: string) => {
  const result = await fetch(`/api/submissions/${submissionId}/status`);
  if (result.status === "PENDING" || result.status === "RUNNING") {
    setTimeout(() => poll(submissionId), 2000);
  } else {
    displayVerdict(result);
  }
};

Phase 2: Asynchronous Execution (the worker's job)

BullMQ dequeues job
  → MongoDB: Fetch public + hidden test cases for problemId
  → ExecutionService.runFullSubmission(sourceCode, tests)
      → For each test case batch:
          → Judge0: POST { source_code, language_id, stdin }
          → Wait for Judge0 response: { stdout, stderr, time, memory, status }
  → Calculate overall verdict

This is where the Driver layer matters. Before sending code to Judge0, a language-specific wrapper is applied. For a Python problem, the driver injects the boilerplate that calls the user's function with the test input and captures stdout. This means users only write the algorithm — not the I/O scaffolding. The /driver directory contains these wrappers for Java, C++, C, Go, Rust, Python, and more.

Judge0 is the primary and definitive judge. It compares the program's stdout against the expected output for each hidden test case. For 95%+ of problems, this is all that runs.

For a small subset of problems where multiple valid outputs exist — such as questions asking for "any valid topological order" or "all valid parentheses combinations" — a pure string comparison would incorrectly reject correct answers. In those cases, an AI-based semantic check runs as a fallback, evaluating whether the user's output is logically equivalent even if the string differs. This is the exception, not the rule.

Phase 3: Finalization

Worker → MongoDB: Update submission { status, executionResults, finalVerdict }
Worker → Postgres (parallel): Update user_stats { totalSolved++, streak, points }
Worker → Redis Pub/Sub: Publish arena:submission:{roomId} (if arena match)

The stats update runs in parallel with the MongoDB save using Promise.all. It handles streak calculation (checking lastSolveDate against today's date), difficulty-specific counters (easySolved, mediumSolved, hardSolved), language distribution tracking via a jsonb languageCounts column, and ELO-style arena point deltas.

Arena: Real-Time Multiplayer at Scale

The Arena is the feature that required the most architectural thought. Coordinating 50 simultaneous players — each submitting code, getting verdicts, and watching a live leaderboard update — requires a completely different stack than the REST API.

Why Golang?

Node.js uses an event loop — it is single-threaded I/O multiplexed. For WebSockets, this means each message goes through the same event queue. Under high concurrency, this creates latency jitter. A single slow operation (like a heavy JSON parse) can cause perceptible delays for unrelated clients.

Go's concurrency model is fundamentally different: each WebSocket client gets its own goroutine (a lightweight green thread, ~2KB stack vs ~8MB for OS threads). The Go runtime scheduler multiplexes thousands of goroutines across available CPU cores. Communication between goroutines uses channels — typed, blocking data pipes that are inherently safe without mutex locks.

// Arena Hub (simplified) — each room is a Go goroutine cluster
type Hub struct {
    rooms  map[string]*Room
    mu     sync.RWMutex
    redis  *redis.Client
}

type Room struct {
    id      string
    clients map[string]*Client  // userId → websocket connection
    pub     chan []byte          // outbound broadcast channel
}

// Each client has a dedicated read goroutine
func (c *Client) readPump() {
    for {
        _, message, err := c.conn.ReadMessage()
        if err != nil { break }
        c.hub.handleMessage(c, message)
    }
}

Redis as the State Store (Lua Atomic Scripts)

The arena room state lives in Redis, not in Go memory. This is a critical design decision: if the Go server restarts (deploy, crash, OOM), the match state is not lost. Redis is the source of truth.

But Redis is also shared between the Bun API (which handles join requests) and the Go Hub (which manages live connections). Two processes writing to the same key creates race conditions. The solution: embedded Lua scripts that execute atomically on the Redis server itself.

-- arena_join.lua (Lua runs atomically in Redis — no concurrent modification)
local roomKey = KEYS[1]
local userRoomKey = KEYS[2]
local userId = ARGV[1]
local maxPlayers = tonumber(ARGV[2])

local roomJson = redis.call("GET", roomKey)
if not roomJson then
    return redis.error_reply("ROOM_NOT_FOUND")
end

local room = cjson.decode(roomJson)

-- Atomic check: is room full?
if #room.players >= maxPlayers then
    return redis.error_reply("ROOM_FULL")
end

-- Atomic check: is user already in a room?
local existingRoom = redis.call("GET", userRoomKey)
if existingRoom then
    return redis.error_reply("ALREADY_IN_ROOM")
end

-- Both checks passed — atomically update
table.insert(room.players, userId)
redis.call("SET", roomKey, cjson.encode(room), "EX", 86400)
redis.call("SET", userRoomKey, KEYS[1], "EX", 86400)

return redis.status_reply("OK")

Without this Lua script, two simultaneous join requests could both pass the "room full" check and push the room over capacity. The Lua script ensures this is compare-and-swap in a single atomic operation — guaranteed by Redis's single-threaded command processing.

The Pub/Sub Bridge

The Bun workers (which handle code evaluation) and the Go Hub (which manages WebSocket clients) need to communicate. But they run on different processes. The bridge is Redis Pub/Sub:

BullMQ Worker evaluates submission
    → Publishes to Redis channel: "arena:submission:{roomId}"
    → Payload: { userId, verdict, testsPassed, totalTests, score }

Go Hub subscribes to "arena:submission:*"
    → Receives message
    → Finds the correct Room by roomId
    → Broadcasts scoreboard update to all connected clients in that room

This design means the Bun API server and the Go Arena Hub are completely decoupled. Neither knows the other's hostname or IP. They only share knowledge of the Redis channel name format. This makes independent scaling and deployment of each service trivial.

The Arena State Machine

An Arena match transitions through these states, managed via Redis + Go:

[*] → WAITING  (Host creates room, others join via PIN)
WAITING → LOBBY  (Host toggles lobby mode for readiness checks)
LOBBY → WAITING  (Host toggles back)
WAITING/LOBBY → PLAYING  (Host starts match — requires ≥ 2 players)
PLAYING → FINISHED  (20-minute timer expires OR host aborts)

The match timer is managed via a BullMQ delayed job: when the host starts a match, a job is enqueued with a delay of matchDurationMs. When that job fires, the Arena Worker publishes a "match:ended" event to Redis Pub/Sub, which the Go Hub receives and broadcasts to all clients.

System Design: AI-Powered Learning Canvas

The System Design section is more than static documentation. It is a 56-topic curriculum backed by an interactive canvas and AI assistant.

Each topic (e.g., "Load Balancer", "Message Queue", "CDN") has:

Structured content — a deep-dive explanation with diagrams and real-world examples
A Tldraw workspace — an infinite canvas where users sketch their architecture diagrams
An AI chat assistant — context-aware of the current topic being studied
AI diagram generation — describe a system in prose, get a Tldraw diagram back

The AI tutor prompt is aware of the current system design topic. If the user is on the "Caching" topic and asks "where does Redis fit?", the AI knows the context and responds with topic-specific advice, not generic information.

The diagram generation pipeline:

User: "Draw a system design for a URL shortener"
   → API: POST /systemdesign/workspaces/{id}/generate
   → AI: Generate Tldraw JSON schema from description
   → Parse + validate the returned JSON
   → Merge into the existing workspace document in MongoDB
   → Return updated Tldraw state to the client

AI responses and diagram outputs are cached in Redis (7-day TTL) so identical prompts never cost a second API call — useful when many users study the same topics concurrently.

The Codebase: 70+ Interconnected Classes

The backend (/api) uses Awilix for dependency injection — a container-based DI system similar to what you'd see in NestJS, but lighter and built for Hono. Every service is registered in the container and injected into controllers via the constructor.

// container.ts
container.register({
  // Repositories
  problemRepository: asClass(ProblemRepository).singleton(),
  submissionRepository: asClass(SubmissionRepository).singleton(),

  // Services
  executionService: asClass(ExecutionService).singleton(),
  submissionService: asClass(SubmissionService).singleton(),

  // Controllers
  submissionController: asClass(SubmissionController).singleton(),
});

The service layer boundary is strict:

Controllers — handle HTTP routing, input validation, auth middleware
Services — contain all business logic, orchestrate between repositories and external APIs
Repositories — contain all database queries, return typed model objects

The codebase has 4,765 indexed nodes and 9,732 dependency edges across all packages. The submission domain alone chains: SubmissionController → SubmissionService → ExecutionService → DriverJudgeExecutionService → LLMService.

Feature Scope at a Glance

Feature	What It Does
Problems	11,000+ coding problems with hidden test cases, Judge0 execution, solution voting
Academy	82+ language tracks (Go, Rust, Haskell, Gleam...) with progressive exercise-based learning
Roadmap	Interactive DSA roadmap — from Arrays to Tries — with problem links at each node
Arena	Real-time multiplayer coding battles, up to 50 players, live leaderboard via Go + Redis
System Design	56-topic curriculum, Tldraw canvas, AI chat tutor, AI diagram generation
Companies	460+ company profiles with tagged interview questions
Contests	Unified calendar: LeetCode, Codeforces, AtCoder, CodeChef — auto-synced via cron
Compilers	Guest-accessible Monaco playground, full Judge0 execution, built-in Tldraw scratchpad
Leaderboards	ELO-style arena rating, problem solve streaks, language-specific stats
Social	Follow users, post solutions with voting, GitHub/LinkedIn/LeetCode profile links
Admin	Internal dashboard for problem creation, category management, and content moderation

What I Would Do Differently

1. WebSocket for submission status instead of polling. The client polling every 2 seconds works, but it creates unnecessary API load at scale. A Redis Pub/Sub → Server-Sent Events (SSE) channel would push the verdict to the client the moment it's ready, with zero polling overhead.

2. Read replicas for MongoDB. All submission reads go to the primary. Under high read load, a read replica for the GET /submissions/{id}/status endpoint would reduce primary pressure significantly.

3. Separate Redis instances by concern. All four Redis roles (cache, queue, state, pub/sub) share one Valkey instance. Under heavy arena load, the pub/sub throughput could starve the BullMQ queue processing. Separating pub/sub into a dedicated instance is the right move at scale.

4. gRPC instead of Redis Pub/Sub for Go↔Node communication. Redis Pub/Sub is "fire and forget" — there's no acknowledgment. If the Go Hub restarts and misses a message, that leaderboard update is lost. A gRPC streaming connection with delivery guarantees would be more robust.

The Repository Structure

slavecode/
├── /api        → Hono API (Bun, Awilix, Drizzle, Mongoose, BullMQ)
├── /arena      → Go WebSocket Hub (Fiber, goroutines, Redis Pub/Sub)
├── /web        → Next.js 15 frontend (Monaco, Tldraw, Zustand)
├── /admin      → Internal admin portal (Next.js)
├── /driver     → Language-specific code wrappers (Java, C++, Go, Rust...)
├── /cloud      → Azure VM provisioner + health checks
├── /infra      → Docker Compose + Dockerfiles
├── /scripts    → DB seeding, Drizzle migrations, Docker utilities
├── /testings   → Integration tests (network, DB pool, endpoints)
└── /docs       → Full UML diagrams, ERDs, architecture docs

Conclusion

Building SlaveCode taught me that distributed systems are not about choosing the right technology — they're about understanding the data flow and failure modes first, and then choosing the technology that maps cleanly to those constraints.

The Judge0 sandbox is on Azure because blast radius matters. The Arena is in Go because goroutines genuinely outperform an event loop for persistent WebSocket connections at scale. Redis does four jobs because it is the right tool for all four of those jobs at this scale.

Every piece is there because the alternative broke something.

SlaveCode is live at slavecode.codes. Find me on GitHub at @phero20. Your code is your master. Serve it well.

DEV Community: Feroz

Concurrent Resource Scheduler v1.2.3: Sharded Priority Heaps Under Concurrent Load

Concurrent Resource Scheduler v1.2.3: Sharded Priority Heaps Under Concurrent Load

First: What Problem Are We Actually Solving?

The Obvious Design: One Slice, One Mutex

The Main Architectural Decision

Why a Priority Heap?

The Scheduler Is More Than a Heap

Resource Lookup Is a Separate Problem

Acquisition Strategies Are Pluggable

Adaptive

Weighted

Round Robin

Affinity

Shared vs Exclusive Acquisition

Resource Lifecycle

Cooldown Is an Extension, Not Core Scheduling Logic

Why v1.2.3 Exists

Cooldown initialization documentation

Cooldown example timing

Benchmark package boundary

Documentation consistency

Prometheus extension

The Numbers: Actual v1.2.3 Microbenchmarks

Acquire Strategy Benchmarks

Scheduler Hot-Path Benchmarks

Why We Benchmark Multiple Heap Counts

Batch Operations Have a Different Cost Profile

Validation Is More Important Than a Single Benchmark

Normal tests

Race detector

Static analysis

Formatting

Why the Race Detector Matters

The Load-Test Question

10,000 Concurrent Workers

A Failure Is Not Necessarily a Scheduler Failure

Why This Matters for Resource Pools

Domain-Agnostic by Design

A Minimal v1.2.3 Example

What v1.2.3 Changed Technically

1. Cooldown initialization correction

2. Cooldown example stabilization

3. Benchmark boundary cleanup

4. Documentation corrections

5. Prometheus module alignment

What I Don't Want These Numbers to Mean

When Sharding Helps

Why I Chose This Architecture

v1.2.3 Release Checklist

Where This Project Goes Next

Final Thoughts

Project

I Built a Concurrent Resource Scheduler in Go with Sharded Priority Heaps

Table of Contents

The Problem

The Naive Approach

Why a Global Mutex Becomes a Problem

1. Lock contention

2. Linear scanning

3. Priority maintenance

4. State transitions

5. Observability

The Core Idea Behind CRS

Architecture at a Glance

Sharded Priority Heaps

Why Sharding Helps

The O(1) Lookup Map

Priority and Acquire Are Different Problems

Acquire Strategies

Round Robin

Weighted Acquire

Adaptive Acquire

Affinity Routing

Shared vs Exclusive Acquisition

Shared

Exclusive

Resource Lifecycle

Atomic State Transitions

The Inactive Store