Throttling rejects requests randomly. Scheduling rejects them deliberately. Deliberate rejection is more useful.
Free model endpoints share one quota across every caller. A batch job can starve interactive users. First-come-first-served is not fairness. It is luck.
This tutorial builds a priority queue. It schedules model requests by importance. Every step is verifiable.
The failure mode
Two services share one endpoint. A nightly batch job sends thousands of prompts. Users wait. Their requests time out.
Rate limiting makes it worse. It drops requests at random. A user request has the same chance as a batch request. That is wrong.
A priority queue fixes the ordering. It does not add capacity. It makes the scarce capacity go to the right caller.
What you will build
A scheduler with three parts.
- A priority queue
- A concurrency limiter
- An aging mechanism
The queue holds pending requests. The limiter controls how many reach the endpoint. Aging prevents starvation.
MonkeyCode's free model access is the upstream in this walkthrough. Its free server option can host the scheduler. The code itself is generic Node.js.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Step 0: Prerequisites
Node.js 22 or newer. One model endpoint. One process.
npm init -y
npm pkg set type=module
Verify Node.
node --version
Expect v22 or higher.
Step 1: The priority queue
Create queue.js.
export class PriorityQueue {
#items = [];
push(item, priority) {
this.#items.push({ item, priority, at: Date.now() });
this.#items.sort((a, b) => a.priority - b.priority || a.at - b.at);
}
pop() {
return this.#items.shift()?.item ?? null;
}
get size() {
return this.#items.length;
}
}
Sorting on every push is O(n log n). For a few hundred pending requests, that is fine. For thousands, swap in a binary heap.
The at timestamp breaks ties. Older requests win. That is the first fairness rule.
Verify:
node -e "
import('./queue.js').then(({ PriorityQueue }) => {
const q = new PriorityQueue();
q.push('low', 10);
q.push('high', 1);
q.push('mid', 5);
console.log(q.pop(), q.pop(), q.pop());
})"
Expect high mid low.
Step 2: The scheduler
Create scheduler.js.
import { PriorityQueue } from "./queue.js";
export class Scheduler {
#queue = new PriorityQueue();
#active = 0;
#maxConcurrent;
#endpoint;
constructor(endpoint, maxConcurrent = 2) {
this.#endpoint = endpoint;
this.#maxConcurrent = maxConcurrent;
}
submit(prompt, priority) {
return new Promise((resolve, reject) => {
this.#queue.push({ prompt, resolve, reject }, priority);
this.#drain();
});
}
async #drain() {
while (this.#active < this.#maxConcurrent && this.#queue.size > 0) {
const job = this.#queue.pop();
this.#active++;
this.#run(job).finally(() => {
this.#active--;
this.#drain();
});
}
}
async #run({ prompt, resolve, reject }) {
try {
const res = await fetch(this.#endpoint, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ prompt }),
});
const text = await res.text();
resolve({ status: res.status, text });
} catch (err) {
reject(err);
}
}
}
The scheduler holds promises. Callers await their turn. The concurrency limiter caps active requests.
The #drain loop is the engine. It runs after every push. It runs after every completion. The queue never sits idle while capacity exists.
Verify with a mock.
import { Scheduler } from "./scheduler.js";
let calls = [];
global.fetch = async (url, opts) => {
const { prompt } = JSON.parse(opts.body);
calls.push(prompt);
await new Promise(r => setTimeout(r, 10));
return { status: 200, text: async () => "ok" };
};
const s = new Scheduler("http://fake", 2);
await Promise.all([
s.submit("user-request", 1),
s.submit("batch-1", 10),
s.submit("batch-2", 10),
s.submit("user-request-2", 1),
]);
console.log(calls);
Expect user-request and user-request-2 first. The batch jobs wait.
Step 3: Aging
Starvation is real. A constant stream of high-priority requests can block low-priority ones forever.
Aging raises the priority of waiting jobs. Every second, the queue boosts them.
Update queue.js.
export class PriorityQueue {
#items = [];
push(item, priority) {
this.#items.push({ item, priority, at: Date.now() });
this.#sort();
}
age(boostMs = 1000) {
const now = Date.now();
for (const entry of this.#items) {
const waited = now - entry.at;
entry.priority -= Math.floor(waited / boostMs);
}
this.#sort();
}
#sort() {
this.#items.sort((a, b) => a.priority - b.priority || a.at - b.at);
}
pop() {
return this.#items.shift()?.item ?? null;
}
get size() {
return this.#items.length;
}
}
Expose age on the scheduler.
age() {
this.#queue.age();
}
Call it on an interval.
setInterval(() => scheduler.age(), 1000);
Now low-priority jobs climb. They cannot starve forever. The aging interval is a tuning knob.
Verify: submit one high-priority job per second. Submit one low-priority job. Wait five seconds. The low-priority job should run.
Step 4: The full test
Build a deterministic test. No real network. No real clock.
import assert from "node:assert/strict";
import { Scheduler } from "./scheduler.js";
const order = [];
let active = 0;
let maxActive = 0;
global.fetch = async (url, opts) => {
const { prompt } = JSON.parse(opts.body);
active++;
maxActive = Math.max(maxActive, active);
await new Promise(r => setTimeout(r, 5));
order.push(prompt);
active--;
return { status: 200, text: async () => "ok" };
};
const s = new Scheduler("http://fake", 2);
await Promise.all([
s.submit("high-1", 1),
s.submit("high-2", 1),
s.submit("low-1", 100),
s.submit("low-2", 100),
]);
assert.deepEqual(order.slice(0, 2), ["high-1", "high-2"]);
assert.ok(maxActive <= 2, "concurrency exceeded");
console.log("all assertions passed");
Run it.
node test.js
Expect all assertions passed.
Step 5: Deploy
The scheduler is a module. Wrap it in an HTTP server. Expose a /submit route.
import { createServer } from "node:http";
import { Scheduler } from "./scheduler.js";
const scheduler = new Scheduler(process.env.MODEL_URL, 2);
createServer(async (req, res) => {
if (req.method !== "POST") {
res.statusCode = 405;
return res.end("POST only");
}
let body = "";
for await (const chunk of req) body += chunk;
const { prompt, priority = 5 } = JSON.parse(body);
try {
const result = await scheduler.submit(prompt, priority);
res.setHeader("Content-Type", "application/json");
res.end(JSON.stringify(result));
} catch (err) {
res.statusCode = 502;
res.end(JSON.stringify({ error: err.message }));
}
}).listen(3000, () => console.log("scheduler on 3000"));
Test the deployment.
curl -s -X POST localhost:3000 -d '{"prompt":"hello","priority":1}'
curl -s -X POST localhost:3000 -d '{"prompt":"background","priority":10}'
The high-priority request returns first. The background request waits.
Priority table
| Caller | Priority | Rationale |
|---|---|---|
| Interactive user | 1 | Waiting human |
| API request | 3 | SLO-bound |
| Background job | 10 | No one waits |
| Pre-generation | 20 | Nice to have |
Aging still protects the bottom rows. The table is a starting point. Tune it with real latency data.
Limitations
The queue lives in memory. A restart drops pending requests. The queue is per-process. Multi-instance deployments need Redis or similar.
The scheduler does not retry. Failed requests reject their callers. Combine this with a retry layer if needed.
Priority is a number. It does not capture deadlines. A request with priority 1 and a request with priority 2 both expire. Add a deadline check if your callers need it.
Who should not use this
Single-user tools do not need a queue. One caller cannot starve itself.
Strict FIFO workloads should not use this. If order matters more than importance, a plain queue is better.
Systems without shared quota gain nothing. The scheduler shines when one endpoint serves many callers.
The takeaway
Throttling is random. Scheduling is deliberate. A priority queue puts the scarce quota where it matters.
The code is small. The tests are deterministic. The behavior is visible.
MonkeyCode's free server option can host this scheduler. The same code runs on any Node host. Start with the queue. Add aging. Measure the difference.
Top comments (0)