Make Free Model CI Jobs Replayable Before You Retry Them

The retry trap

A free model CI job fails on a timeout.

You click retry.

The whole pipeline starts over: checkout, build, dependencies, model call.

That is the trap.

Why re-run the world for one timeout?

Retrying the pipeline does not isolate the flaky step.

It makes a small problem expensive.

I wanted a workflow that replays just the model call, not the whole pipeline.

So I made every free model call leave behind a tiny reproducible record.

A record has two halves: the input envelope and the output hash.

If the job fails, I can replay the input against the same model and compare the output hash.

No full pipeline re-run.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

I use MonkeyCode's free model access for the model step and its free server option as a small replay store.

I do not assume exact quotas, model names, or availability windows here.

The pattern works with any free HTTP model endpoint and any tiny key-value store or CI artifact.

Why a hash and not the full prompt

Full prompt logs are useful until they are not.

A free model job may receive a snippet of a merge request, an error message, or an environment variable.

Store the raw text in CI logs and you can accidentally leak source or secrets.

Store a hash and the replay input in a locked artifact, and the risk drops.

A hash also gives me one cheap comparison target.

I do not need to reason about the entire response to see that an endpoint changed.

I only need byte-level equality.

The record shape

For every model call, I save the fields below.

• request_id: a hash derived from model, prompt hash, and a timestamp.
• prompt_hash: the hash of the normalized prompt.
• response_hash: the hash of the raw response.
• status: the HTTP status of the original call.
• bytes: the length of the response.

The exact hash algorithm matters less than using the same one on both sides.

I use SHA-256 because it is available everywhere.

GitLab CI wiring

I run two jobs.

The first job calls the model and posts the record to the free server.

The second job is manual and only replays the recorded input.

stages:
  - model_call
  - replay_check

model_call:
  stage: model_call
  image: python:3.11-slim
  variables:
    MODEL_ENDPOINT: ${FREE_MODEL_URL}
    REPLAY_STORE: ${FREE_REPLAY_STORE_URL}
  script:
    - python model_call.py
  artifacts:
    paths:
      - replay_record.json

replay_check:
  stage: replay_check
  image: python:3.11-slim
  needs:
    - model_call
  when: manual
  script:
    - python replay_check.py replay_record.json
  allow_failure: false

The manual job is the interesting part.

It lets me ask one question: can this exact input produce the same output again?

If the answer is no, I stop and inspect.

The model call script

This snippet is a compact reference, not a production client.

Error handling is intentionally minimal so the flow is visible.

import hashlib
import json
import os
import time
import urllib.request

def sha256(text):
    return hashlib.sha256(text.encode('utf-8')).hexdigest()

prompt = os.environ.get('PROMPT', 'Summarize the diff in three bullets')
model = os.environ.get('MODEL_NAME', 'free-model')
endpoint = os.environ['MODEL_ENDPOINT']
store = os.environ['REPLAY_STORE']

request_id = sha256(f'{model}:{sha256(prompt)}:{time.time_ns()}')
prompt_hash = sha256(prompt)

payload = json.dumps({'prompt': prompt}).encode('utf-8')
req = urllib.request.Request(
    endpoint,
    data=payload,
    headers={'Content-Type': 'application/json'},
)
with urllib.request.urlopen(req, timeout=30) as resp:
    output = resp.read().decode('utf-8')

record = {
    'request_id': request_id,
    'prompt_hash': prompt_hash,
    'response_hash': sha256(output),
    'status': 'ok',
    'bytes': len(output),
}

post = urllib.request.Request(
    store + '/records',
    data=json.dumps(record).encode('utf-8'),
    headers={'Content-Type': 'application/json'},
    method='POST',
)
urllib.request.urlopen(post, timeout=30)

with open('replay_record.json', 'w') as f:
    json.dump({'record': record, 'prompt': prompt}, f)

The replay input is stored in the CI artifact.

The response hash is stored in the free server.

That split keeps the raw prompt out of the server and keeps the comparison cheap.

The replay check

The replay job reads the saved artifact and calls the model again.

import hashlib
import json
import os
import sys
import urllib.request

def sha256(text):
    return hashlib.sha256(text.encode('utf-8')).hexdigest()

with open(sys.argv[1]) as f:
    saved = json.load(f)

record = saved['record']

req = urllib.request.Request(
    os.environ['MODEL_ENDPOINT'],
    data=json.dumps({'prompt': saved['prompt']}).encode('utf-8'),
    headers={'Content-Type': 'application/json'},
)
with urllib.request.urlopen(req, timeout=30) as resp:
    fresh = resp.read().decode('utf-8')

fresh_hash = sha256(fresh)

if fresh_hash != record['response_hash']:
    print('DRIFT', record['response_hash'][:10], '->', fresh_hash[:10])
    sys.exit(1)

print('REPLAY MATCH', fresh_hash[:10])

A matching hash means the endpoint returned byte-identical output.

A mismatch means the model, the endpoint, or the input normalization changed.

That is a signal worth chasing.

What this does and does not catch

Here is a quick decision table.

Situation	What I do
HTTP 429 or timeout	back off and retry; do not compare hashes
hash mismatch after replay	inspect model or endpoint drift
hash match but downstream test still fails	debug the code and configuration, not the model
replay times out	treat it as a capacity issue, not a prompt issue

Hashing does not forgive non-deterministic output.

If the model returns a timestamp, an ID, or random formatting, the same input produces a different hash.

Normalize before hashing: strip timestamps, IDs, and trailing whitespace where possible.

Rate limits still exist.

If both the original call and the replay hit the same free-tier cap, you see a timeout rather than drift.

Record the HTTP status separately from the response hash.

Otherwise a 429 looks like model drift.

Also, byte equality is not usefulness.

A free endpoint can change its response format without changing meaning.

A hash only proves exact reproducibility, not that the answer is correct.

Who should not use this

Skip this if every model call is already cheap, idempotent, and failure-tolerant.

Skip it if your team can safely read full prompts and responses in CI logs and already has a rollout gate.

Skip it if your free tier lets you re-run the whole pipeline and you do not need reproducibility.

This is for low-trust, low-blast-radius free model jobs.

It makes them behave more like a deterministic function.

The point

The goal is not another monitor.

It is to make a free model call replayable.

Same input, same output, or an explicit drift signal.

That turns retries from a guessing game into a comparison.

If you already use MonkeyCode's free model access, try the two-job envelope once.

It costs little and changes how you debug flaky model jobs.