A free model endpoint can return HTTP 200 while still breaking your pipeline. The server is reachable, auth succeeds, and the response body still fails the caller because the schema changed, the completion is empty, or the result was truncated with a length finish reason. When that discovery happens inside a feature branch, an upstream dependency problem turns into a long debugging session mixed into your own diff.
A readiness contract fixes that separation. It is a small deterministic probe that runs independently of feature code, checks only the operational properties your application requires, and fails the scheduled job before a developer tries to merge work that depends on the endpoint.
Disclosure: This article was prepared as part of MonkeyCode's product outreach.
If your MonkeyCode account currently includes free model access and a free server option, that free dependency is worth the same smoke test as a paid one. The workflow below treats the MonkeyCode free server as one possible endpoint candidate. It does not rank models or assume a particular model name, quota, or response format beyond the generic OpenAI-compatible chat completions shape that many providers expose.
Why a ping is not enough
A plain HTTP check can confirm DNS, TLS, and a listening port. It cannot confirm that the server still understands the request shape, that your token is still accepted, that the response parser still finds the content field, or that the model is not silently truncating. Those are the failures that later appear as error handling in application code.
The contract should therefore test:
- A minimal request that matches the provider format.
- Authentication without logging the token.
- JSON parsing and the exact path to the completion text.
- A non-empty content string.
- Latency against a configured budget.
- A finish reason that is not a truncation signal.
It should not test quality, tone, accuracy, or benchmark performance. Those need a separate evaluation harness; mixing them into a connectivity probe makes the result hard to interpret.
The probe script
The script below is a scaffold to adapt, not a reported benchmark. It sends a trivial prompt, parses the response, runs three attempts, and exits non-zero when any attempt violates the contract.
import json
import os
import statistics
import time
import urllib.error
import urllib.request
ENDPOINT = os.environ['MODEL_ENDPOINT'].rstrip('/')
TOKEN = os.environ['MODEL_TOKEN']
MODEL = os.environ.get('MODEL_NAME')
MAX_TOKENS = int(os.environ.get('MAX_RESPONSE_TOKENS', '64'))
TIMEOUT_S = float(os.environ.get('PROBE_TIMEOUT_S', '30'))
MAX_LATENCY_MS = int(os.environ.get('MAX_LATENCY_MS', '25000'))
RUNS = int(os.environ.get('PROBE_RUNS', '3'))
MIN_CONTENT_CHARS = int(os.environ.get('MIN_CONTENT_CHARS', '5'))
def build_payload():
messages = [
{'role': 'system', 'content': 'You are a connectivity probe. Be brief.'},
{'role': 'user', 'content': 'Reply with exactly: ready'}
]
payload = {
'messages': messages,
'max_tokens': MAX_TOKENS,
'temperature': 0
}
if MODEL:
payload['model'] = MODEL
return payload
def probe_once():
data = json.dumps(build_payload()).encode('utf-8')
request = urllib.request.Request(
ENDPOINT + '/v1/chat/completions',
data=data,
headers={
'Authorization': 'Bearer ' + TOKEN,
'Content-Type': 'application/json'
},
method='POST'
)
started = time.monotonic()
try:
with urllib.request.urlopen(request, timeout=TIMEOUT_S) as response:
status = response.status
body = response.read().decode('utf-8')
except urllib.error.HTTPError as error:
body = error.read().decode('utf-8')
return {
'ok': False,
'status': error.code,
'latency_ms': (time.monotonic() - started) * 1000,
'content': '',
'error': body[:200]
}
except urllib.error.URLError as error:
return {
'ok': False,
'status': 'network_error',
'latency_ms': (time.monotonic() - started) * 1000,
'content': '',
'error': str(error.reason)
}
latency_ms = (time.monotonic() - started) * 1000
try:
parsed = json.loads(body)
content = parsed['choices'][0]['message']['content']
finish_reason = parsed['choices'][0].get('finish_reason')
except (KeyError, IndexError, TypeError, json.JSONDecodeError) as error:
return {
'ok': False,
'status': status,
'latency_ms': latency_ms,
'content': '',
'error': 'response shape mismatch: ' + str(error)
}
ok = (
status == 200
and isinstance(content, str)
and len(content.strip()) >= MIN_CONTENT_CHARS
and latency_ms <= MAX_LATENCY_MS
and finish_reason not in {'length'}
)
return {
'ok': ok,
'status': status,
'latency_ms': latency_ms,
'content': content.strip()[:120],
'finish_reason': finish_reason
}
def main():
if not ENDPOINT.startswith('https://'):
raise SystemExit('MODEL_ENDPOINT must start with https:// for CI smoke tests')
results = [probe_once() for _ in range(RUNS)]
latencies = [r['latency_ms'] for r in results if r['status'] == 200]
median_latency_ms = statistics.median(latencies) if latencies else None
failures = [r for r in results if not r.get('ok')]
summary = {
'runs': len(results),
'failures': len(failures),
'median_latency_ms': median_latency_ms,
'failures_detail': failures
}
print(json.dumps(summary, indent=2))
if failures:
raise SystemExit(1)
print('contract passed')
if __name__ == '__main__':
main()
The script requests a trivial completion, so token use stays near zero. That matters when you are testing a free tier and want the probe itself to remain cheap.
Wire it into GitLab CI
Store the endpoint and token in GitLab CI/CD variables, not in the repository. Mark the token as masked so it cannot leak into raw logs. The job below inherits those variables and runs only on schedules by default.
model-contract:
image: python:3.12-slim
stage: test
script:
- python model_guard/check_endpoint.py
variables:
MODEL_ENDPOINT: $MODEL_ENDPOINT
MODEL_TOKEN: $MODEL_TOKEN
MODEL_NAME: $MODEL_NAME
PROBE_RUNS: '3'
MAX_RESPONSE_TOKENS: '64'
MAX_LATENCY_MS: '25000'
only:
- schedules
timeout: 5m
The classic only: [schedules] form is sufficient here. If you need a more precise policy, switch to rules and add a manual web trigger for endpoint migrations or credential rotation.
Read the result as a signal, not a verdict
A failing contract should not automatically mean the model is broken. It means one of the operational assumptions changed. Use that signal to decide how aggressive the CI block should be.
| Contract signal | What it tells you | CI action |
|---|---|---|
| All runs pass | Auth, schema, latency, and non-empty completion are currently acceptable. | Allow a downstream job that calls the model. |
| One of three runs fails | Possible transient rate limit, cold start, or runner egress issue. | Open a non-blocking issue; block only if the code hard-depends on the model. |
| All runs fail | Auth, model name, endpoint URL, or response shape changed. | Block dependent work until the contract or the provider is updated. |
Start with generous thresholds and tighten them from observed data. A free endpoint can have noisy latency, so a one-run failure should not immediately block a merge if the application degrades gracefully.
Keep the limitations visible
This contract is deliberately narrow.
- It does not test output quality, bias, safety, or task accuracy.
- It tests only the request fields you put in the payload. If your real code uses tools, streaming, vision, or structured output, add those fields or you will get false confidence.
- A synthetic prompt may pass while real domain prompts still fail.
- Free-tier limits can change between runs. A pass at 03:00 does not guarantee a pass at 16:00.
- GitLab runner location affects latency. A latency budget measured on a different runner may not match production traffic.
- Never send personal data, secrets, or customer text through the probe. Use fixed, harmless strings only.
Do not use this approach when the model is a hard runtime dependency with strict latency or reliability requirements, when you need model provenance or a formal SLA, or when the workload includes regulated data. A free endpoint probe is useful for detecting operational drift early; it is not a replacement for a real evaluation or a service-level objective.
Add the contract before you build more features on top of the endpoint. The earlier you catch a silent schema change or rate-limit spike, the less of your pipeline depends on assumptions you have not verified.
Top comments (0)