Jordan Huang

Posted on Aug 16

A Rebase Wave of Parallel CI Jobs Drowned My Free Model. Singleflight Saved It.

#gitlab #cicd #ai #go

A rebase wave hit my GitLab repo last week. Twelve jobs? Forty? I stopped counting. Every parallel job called the same free model with the same prompt. The free server held. The model rate limit did not.

Why didn't my cache save me? Because a content cache helps on the second call. A wave of first calls hits all at once.

Same input, same moment, wasted calls

This is not a quota problem. It is a concurrency problem.

Here is what happened:

GitLab started many jobs for one commit.
Each job loaded the same diff or prompt.
Each job sent a model request at almost the same moment.
The cache was empty, so the upstream model received many identical calls.

A cache fixes repeat traffic. It does not fix a stampede.

Put a singleflight gate in front of the model

The fix is a small gate. When several requests share the same key, only one request goes upstream. The others wait and reuse the result.

I run this gate on a small free server. I use MonkeyCode's free model access and the free server option to host it. Disclosure: This article was prepared as part of MonkeyCode's product outreach.

What singleflight does

The first request for a key does the real work.
Other requests for that key wait.
All requests receive the same result.
One upstream call replaces N identical calls.

The key should be the request fingerprint: model name, prompt, and temperature.

The code

Here is a small Go sidecar that combines a short TTL cache with singleflight.

package main

import (
    "crypto/sha256"
    "encoding/hex"
    "encoding/json"
    "fmt"
    "log"
    "net/http"
    "sync"
    "time"

    "golang.org/x/sync/singleflight"
)

type modelRequest struct {
    Model  string  `json:"model"`
    Prompt string  `json:"prompt"`
    Temp   float64 `json:"temp"`
}

type cacheEntry struct {
    value  any
    expiry time.Time
}

var (
    group singleflight.Group
    mu    sync.Mutex
    cache = map[string]cacheEntry{}
)

func key(r modelRequest) string {
    h := sha256.Sum256([]byte(fmt.Sprintf("%s|%s|%.2f", r.Model, r.Prompt, r.Temp)))
    return hex.EncodeToString(h[:])
}

func callModel(r modelRequest) (any, error) {
    // Replace with a call to the free model endpoint.
    // Keep an explicit timeout, for example 5 seconds.
    return nil, nil
}

func writeJSON(w http.ResponseWriter, v any) {
    w.Header().Set("Content-Type", "application/json")
    _ = json.NewEncoder(w).Encode(v)
}

func handler(w http.ResponseWriter, r *http.Request) {
    var req modelRequest
    if err := json.NewDecoder(r.Body).Decode(&req); err != nil {
        http.Error(w, err.Error(), http.StatusBadRequest)
        return
    }

    k := key(req)

    mu.Lock()
    if e, ok := cache[k]; ok && time.Now().Before(e.expiry) {
        mu.Unlock()
        writeJSON(w, e.value)
        return
    }
    mu.Unlock()

    v, err, _ := group.Do(k, func() (any, error) {
        return callModel(req)
    })
    if err != nil {
        http.Error(w, err.Error(), http.StatusInternalServerError)
        return
    }

    mu.Lock()
    cache[k] = cacheEntry{value: v, expiry: time.Now().Add(2 * time.Minute)}
    mu.Unlock()

    writeJSON(w, v)
}

func main() {
    http.HandleFunc("/model", handler)
    log.Fatal(http.ListenAndServe(":8080", nil))
}

Your GitLab CI job can call the sidecar instead of the model directly. Keep the key stable across jobs.

model-check:
  script:
    - KEY=$(printf '%s|%s|%.2f' "$MODEL" "$PROMPT" "$TEMP" | sha256sum | cut -d' ' -f1)
    - curl -s -X POST http://sidecar:8080/model
      -d "{\"model\":\"$MODEL\",\"prompt\":\"$PROMPT\",\"temp\":$TEMP}"

Rules for the gate

Keep the settings boring.

Set a short upstream timeout, such as 5 seconds.
Use a short TTL for cached results.
Cache only deterministic requests, usually temp: 0.
Never cache errors.
Include model, prompt, and temperature in the key.

A missing timeout turns a stuck model call into a stuck CI job.

When singleflight helps

Situation	Use singleflight?	Why
Parallel jobs send identical deterministic input	Yes	One upstream call is enough
Parallel jobs send different inputs	No	There is no shared result to reuse
Prompt includes job ID or timestamp	No	The key changes for every job
Creative output with temperature above 0	No	Shared output may harm variety
Very rare requests with no concurrency	Maybe not	The gate adds moving parts

Limitations

Singleflight is not magic.

It is process-local. If the free server restarts with multiple replicas, each replica can still call the model once.
The TTL cache is in memory. A cold start loses it.
A short TTL can serve slightly stale results.
It does not replace a real queue for long-running work.
It only helps when requests arrive close together.

If a model call takes more than a few seconds, move it behind a durable queue instead.

Who should skip this

Do not add this gate if:

Every CI job needs a unique model output.
Your prompts include branch, commit, or job-specific context.
You run a single job at a time and never see bursts.
You do not want to operate another sidecar process.

The gate is boring. That is the point. It turns a stampede into one request and a short-lived answer.

If your CI already uses free models, a boring gate beats a surprise rate limit.

DEV Community