When Your Free Model Endpoint Drifts, Fail the Build

The prompt did not change.
The output did.

That is the failure mode a normal CI job never catches. You pin a free model endpoint, it returns 200, and the build goes green while the model behind the URL quietly changes.

The problem: a moving target dressed as a fixed dependency

Readiness checks answer one question: can the endpoint answer at all? Token watchdog jobs answer another: did the run stay cheap? Both are useful. Neither detects a model that changed routing, decoding parameters, quantization, or an injected system prompt while still answering quickly.

A hard outage is loud. Silent drift is not. The endpoint can keep returning valid JSON and valid text while the behavior you depended on has moved.

The failure you need to catch is not that the model is down. It is that the model is not the model you reviewed last week.

A fingerprint, not another monitor

Instead of asking is it up?, this job asks did it produce the same deterministic output as the checked-in baseline?

You send a small set of boring, single-token prompts. You hash the normalized responses. You check those hashes into the repo. On every CI run, the job compares the hashes and fails closed when one changes.

The fingerprint is not a benchmark. It is a tripwire.

The prompt set

Keep the prompts boring on purpose.

• Short and factual
• Single-token or near single-token answers
• No dates, times, web lookups, or private data
• Temperature 0 where the endpoint supports it
• Enough prompts to catch decode or system-prompt changes, not enough to become an eval harness

Here is a small Python script that emits a JSON fingerprint.

#!/usr/bin/env python3
'''Compute a deterministic fingerprint for a model endpoint.'''
import hashlib
import json
import os
import urllib.request

PROMPTS = [
    'Reply with only the word: blue',
    'Reply with only the next prime after 7:',
    'Reply with only the French word for yes:',
    'Reply with only the opposite of north:',
]

def call(endpoint, prompt, temperature=0):
    payload = json.dumps({
        'prompt': prompt,
        'temperature': temperature,
        'max_tokens': 8,
    }).encode()
    req = urllib.request.Request(
        endpoint,
        data=payload,
        headers={'Content-Type': 'application/json'},
        method='POST',
    )
    with urllib.request.urlopen(req, timeout=30) as resp:
        return json.load(resp)

def normalize(text):
    return ' '.join(text.strip().lower().split())

def main():
    endpoint = os.environ['MODEL_ENDPOINT']
    out = {}
    for prompt in PROMPTS:
        raw = call(endpoint, prompt)
        # Change this path to wherever your endpoint puts generated text.
        text = raw.get('choices', [{}])[0].get('text', '')
        norm = normalize(text)
        out[prompt] = {
            'normalized': norm,
            'sha256': hashlib.sha256(norm.encode()).hexdigest(),
        }
    print(json.dumps(out, indent=2, sort_keys=True))

if __name__ == '__main__':
    main()

The request shape is deliberately small. Some endpoints expect a chat-messages array, others expect max_new_tokens instead of max_tokens, and some ignore temperature. Adjust the path in raw.get() before running. This is not tied to a specific model or vendor.

Compare live output with the checked-in baseline

Generate the baseline once after a human review:

python fingerprint.py > baseline.json

Then run the comparison in CI.

#!/usr/bin/env python3
'''Fail closed when a live fingerprint no longer matches baseline.'''
import json
import sys

baseline = json.load(open('baseline.json'))
actual = json.load(sys.stdin)

failed = False
for prompt, record in baseline.items():
    expected = record['sha256']
    found = actual.get(prompt, {}).get('sha256')
    if found != expected:
        failed = True
        print(f'DRIFT: {prompt}')
        print(f'  expected {expected}')
        print(f'  found    {found}')

if failed:
    sys.exit(1)

print('fingerprint stable')

GitLab CI job

The job is tiny. It sends a few tokens and exits badly when the fingerprint moves.

stages:
  - check

model-fingerprint:
  stage: check
  image: python:3-slim
  variables:
    MODEL_ENDPOINT: $MODEL_ENDPOINT
  script:
    - python fingerprint.py > actual.json
    - python compare.py baseline.json < actual.json
  artifacts:
    when: always
    paths:
      - actual.json
    expire_in: 1 day

Make the failure useful

A failing fingerprint is worse than no fingerprint if it is noisy.

• Save actual.json as an artifact so the on-call person can inspect the raw output.
• Print the specific prompt that drifted first.
• Change the baseline only through an explicit commit, never silently inside CI.
• Compare normalized text before hashing. A single punctuation difference should be legible in the log.

Where MonkeyCode fits

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

MonkeyCode's free model access matters because this workflow needs an endpoint to watch without adding a separate bill. The free server option matters because the job is small enough to run as a scheduled check rather than as a paid runner task. I keep the script generic because a tripwire should not be tied to one request format. I am not assuming a specific quota, model name, or hardware guarantee here; those would need to be verified before relying on them.

Limitations

Exact hashes are deliberately narrow.

• Exact matching is brittle. A punctuation fix can trip the job even when the new behavior is better.
• A fingerprint detects change, not quality. It cannot say whether the new output is worse, only that it is different.
• If sampling is enabled, the hashes vary for no reason. Lock temperature or use a deterministic mode.
• The baseline must be reviewed. Otherwise drift becomes noise and the job gets disabled.
• This is not a security or privacy boundary. Prompts must be safe for third-party processing.

Who should skip this

Do not use an exact fingerprint when the workload is long-form, creative, or paraphrasing-style output. The false-alarm rate will be too high.

Skip it if no one will maintain the baseline. A tripwire nobody resets is just a red light people learn to ignore.

Also skip it if you need semantic equivalence rather than exact wording. This method cares about the literal response, not the meaning.

Start with four boring prompts

Run the fingerprint once. Inspect the normalized fields. Commit the baseline only when the answers look right. Then let the next run fail closed on any change.

The point is not to stop model updates. It is to make them visible before they reach production.