DEV Community

Jordan Huang
Jordan Huang

Posted on

CI Is a Bad Place to Repeat Model Calls. I Moved That Verdict to a Free Server Sidecar.

#ai #cicd #devops #go

CI is a bad place to repeat model calls. A pipeline can re-run the same job on every push, every retry, and every scheduled trigger. If the job includes a direct model request, an unchanged prompt can hit the same free route again and again. The pipeline still looks green, but the free quota is gone.

I moved that call out of CI. The verdict now lives behind a small sidecar hosted on MonkeyCode's free server option. CI calls the sidecar; the sidecar calls the free model routes only when it has to.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The real failure was placement, not the model

The model route was not broken. The problem was the caller.

A CI job is a bad network client. It repeats, retries, and runs under time pressure. When the job calls a free route directly, every rerun is a fresh upstream request even when the prompt did not change.

That creates three costs:

  • quota burn without new information
  • flaky latency inside an otherwise deterministic job
  • a wider attack and failure surface in the pipeline

The sidecar fixes the placement. It takes the repeated part away from CI and keeps the model route behind one local contract.

The sidecar has three jobs

  1. Normalize: turn the provider response into a single ok / text shape.
  2. Cache: reuse the last good verdict for the same prompt within a short window.
  3. Fail closed: return 502 when the upstream response is empty, malformed, or missing.

CI does not need to understand provider JSON, retry logic, or model names. It only needs one local endpoint.

The Go code

This uses only the Go standard library. That matters on a free server slot because there are no extra build dependencies to install.

package main

import (
    "bytes"
    "crypto/sha256"
    "encoding/hex"
    "encoding/json"
    "io"
    "log"
    "net/http"
    "os"
    "sync"
    "time"
)

type UpstreamRequest struct {
    Prompt string `json:"prompt"`
}

type UpstreamResponse struct {
    Choices []struct {
        Message struct {
            Content string `json:"content"`
        } `json:"message"`
    } `json:"choices"`
}

type Verdict struct {
    OK        bool   `json:"ok"`
    Text      string `json:"text"`
    Source    string `json:"source"`
    Cached    bool   `json:"cached"`
    ElapsedMS int64  `json:"elapsed_ms"`
}

type cacheEntry struct {
    Verdict   Verdict
    ExpiresAt time.Time
}

var (
    upstreamURL = os.Getenv("UPSTREAM_URL")
    apiKey      = os.Getenv("UPSTREAM_API_KEY")
    modelName   = os.Getenv("UPSTREAM_MODEL")
    cacheTTL    = 10 * time.Minute

    cacheMu sync.Mutex
    cache   = make(map[string]cacheEntry)
)

func keyFor(prompt string) string {
    sum := sha256.Sum256([]byte(prompt))
    return hex.EncodeToString(sum[:])
}

func cachedVerdict(key string) (Verdict, bool) {
    cacheMu.Lock()
    defer cacheMu.Unlock()
    entry, ok := cache[key]
    if !ok || time.Now().After(entry.ExpiresAt) {
        return Verdict{}, false
    }
    return entry.Verdict, true
}

func storeVerdict(key string, verdict Verdict) {
    cacheMu.Lock()
    defer cacheMu.Unlock()
    cache[key] = cacheEntry{Verdict: verdict, ExpiresAt: time.Now().Add(cacheTTL)}
}

func callUpstream(prompt string) (Verdict, error) {
    payload := map[string]any{
        "messages": []map[string]string{
            {"role": "user", "content": prompt},
        },
    }
    if modelName != "" {
        payload["model"] = modelName
    }
    body, err := json.Marshal(payload)
    if err != nil {
        return Verdict{}, err
    }

    client := &http.Client{Timeout: 5 * time.Second}
    req, err := http.NewRequest(http.MethodPost, upstreamURL, bytes.NewReader(body))
    if err != nil {
        return Verdict{}, err
    }
    req.Header.Set("Content-Type", "application/json")
    if apiKey != "" {
        req.Header.Set("Authorization", "Bearer "+apiKey)
    }

    started := time.Now()
    resp, err := client.Do(req)
    if err != nil {
        return Verdict{}, err
    }
    defer resp.Body.Close()
    elapsed := time.Since(started).Milliseconds()

    if resp.StatusCode != http.StatusOK {
        return Verdict{OK: false, ElapsedMS: elapsed}, nil
    }

    raw, err := io.ReadAll(resp.Body)
    if err != nil {
        return Verdict{}, err
    }

    var parsed UpstreamResponse
    if err := json.Unmarshal(raw, &parsed); err != nil {
        return Verdict{OK: false, ElapsedMS: elapsed}, nil
    }

    text := ""
    if len(parsed.Choices) > 0 {
        text = parsed.Choices[0].Message.Content
    }
    if text == "" {
        return Verdict{OK: false, ElapsedMS: elapsed}, nil
    }

    return Verdict{OK: true, Text: text, Source: "upstream", ElapsedMS: elapsed}, nil
}

func handleVerdict(w http.ResponseWriter, r *http.Request) {
    if r.Method != http.MethodPost {
        http.Error(w, "method not allowed", http.StatusMethodNotAllowed)
        return
    }

    var request UpstreamRequest
    if err := json.NewDecoder(r.Body).Decode(&request); err != nil || request.Prompt == "" {
        http.Error(w, "bad request", http.StatusBadRequest)
        return
    }

    key := keyFor(request.Prompt)
    if verdict, ok := cachedVerdict(key); ok {
        verdict.Cached = true
        writeJSON(w, http.StatusOK, verdict)
        return
    }

    verdict, err := callUpstream(request.Prompt)
    if err != nil {
        http.Error(w, "upstream unavailable", http.StatusBadGateway)
        return
    }

    if verdict.OK {
        storeVerdict(key, verdict)
    }

    if !verdict.OK {
        writeJSON(w, http.StatusBadGateway, verdict)
        return
    }

    writeJSON(w, http.StatusOK, verdict)
}

func writeJSON(w http.ResponseWriter, status int, value any) {
    w.Header().Set("Content-Type", "application/json")
    w.WriteHeader(status)
    _ = json.NewEncoder(w).Encode(value)
}

func main() {
    if upstreamURL == "" {
        log.Fatal("UPSTREAM_URL is required")
    }
    http.HandleFunc("/v1/verdict", handleVerdict)
    http.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
        writeJSON(w, http.StatusOK, map[string]bool{"ok": true})
    })
    addr := os.Getenv("ADDR")
    if addr == "" {
        addr = ":8080"
    }
    log.Printf("verdict sidecar listening on %s", addr)
    log.Fatal(http.ListenAndServe(addr, nil))
}
Enter fullscreen mode Exit fullscreen mode

Run it with:

go run verdict_sidecar.go
Enter fullscreen mode Exit fullscreen mode

Then call it:

curl -X POST http://localhost:8080/v1/verdict -H 'Content-Type: application/json' -d '{"prompt":"Is this test failure likely environmental?"}'
Enter fullscreen mode Exit fullscreen mode

Expected first response:

{"ok":true,"text":"...","source":"upstream","cached":false,"elapsed_ms":812}
Enter fullscreen mode Exit fullscreen mode

The second identical call should return the same text with cached:true and a much lower elapsed_ms.

What the GitLab job sees 

model-verdict:
  image: curlimages/curl:latest
  variables:
    SIDECAR_URL: "http://your-sidecar-host:8080/v1/verdict"
  script:
    - curl --fail-with-body -X POST "$SIDECAR_URL" -H 'Content-Type: application/json' -d '{"prompt":"Does this error look like a flake?"}' -o model_verdict.json
    - cat model_verdict.json
  artifacts:
    paths:
      - model_verdict.json
    when: always
Enter fullscreen mode Exit fullscreen mode

The CI job never sees the provider. It only sees the sidecar's normalized verdict.

The cache is a safety valve, not a truth source

I cache by SHA-256 of the prompt. Same prompt, same key.

The cache prevents identical reruns from burning quota. But it also means an outage upstream can look healthy if the sidecar returns the last good answer.

For my use, that is acceptable. For a fail-closed freshness requirement, set cacheTTL to a very low value or skip the cache. The trade-off is explicit.

Failure rules

Sidecar outcome HTTP status What it means
Correct JSON, non-empty text, upstream 200 Fresh verdict
Same prompt within TTL 200 with cached:true Reused verdict
Non-200 from provider, empty text, or bad JSON 502 Upstream unusable
Upstream timeout or unreachable 502 No verdict
Empty prompt or non-POST 400 Caller error

This table is the whole contract. CI only needs to check the curl exit code.

What this sidecar does not do

  • It does not judge answer quality.
  • It does not pick the best model.
  • It does not protect sensitive input.
  • It does not authenticate the caller in the sample above.
  • It does not make a slow free route fast.

It only stops CI from repeating raw model calls through a poorly shaped client.

Who should skip this pattern

  • Teams with strict freshness: a short cache can hide provider drift.
  • Sensitive prompts: do not send those through an unauthenticated local endpoint.
  • Production decision gates: this is a deterministic smoke check, not a model evaluation.
  • Multi-region setup: a single sidecar adds a single point of failure.

For those cases, keep the model call in a proper eval harness with auth, privacy controls, and explicit freshness.

The local contract is the point

The sidecar is not sophisticated. It is a narrow contract boundary between a repeatable CI job and a non-deterministic free route.

If you already have a free server slot, deploy the sidecar next to your route and let CI call /v1/verdict instead of the provider. The model stays behind one door. CI stays small.

Top comments (0)