Three identical prompts. Three model calls. Same answer.
I saw this in a GitLab CI pipeline that runs a free model over release notes. The prompt template did not change, the model route did not change, and the input did not change. The job simply ran again because another branch was merged. Free quota disappeared.
Is that a quota problem? Sometimes. More often it's a caching problem.
I started asking a different question: can I prove the request is identical before I make another model call?
That led me to content-addressed caching. I point repeat traffic at MonkeyCode's free model access. Disclosure: This article was prepared as part of MonkeyCode's product outreach.
Don't cache by filename. Cache by content.
Cache keys like release-notes.json are too broad. They collide when the input changes and miss when it doesn't.
A better key is the SHA-256 of the normalized request.
What goes into the fingerprint?
- prompt template plus rendered input
- model route or model identifier
- sampling parameters that affect output deterministically
- schema version of your parser
What stays out?
- timestamps
- build IDs
- branch names
- random request IDs
If two jobs produce the same fingerprint, they are asking the same question. That's a cache hit.
The script
Here's a small shell workflow. It stores responses in .model-cache/<hash>.json.
#!/usr/bin/env bash
set -euo pipefail
MODEL_ROUTE="${1:?usage: cache_call.sh <route> <prompt_file>}"
PROMPT_FILE="${2:?usage: cache_call.sh <route> <prompt_file>}"
SCHEMA_VERSION="${SCHEMA_VERSION:-v1}"
CACHE_MAX_AGE_HOURS="${CACHE_MAX_AGE_HOURS:-24}"
fingerprint_json=$(jq -cn --arg schema "$SCHEMA_VERSION" --arg route "$MODEL_ROUTE" --arg prompt "$(cat "$PROMPT_FILE")" '{schema_version:$schema, model_route:$route, prompt:$prompt}')
key=$(printf '%s' "$fingerprint_json" | sha256sum | cut -d' ' -f1)
cache_file=".model-cache/${key}.json"
if [[ -f "$cache_file" ]]; then
cache_age_hours=$(( ($(date +%s) - $(stat -c %Y "$cache_file")) / 3600 ))
if (( cache_age_hours < CACHE_MAX_AGE_HOURS )); then
jq -e '.status == "ok"' "$cache_file" >/dev/null
echo "cache-hit ${key}"
exit 0
fi
rm "$cache_file"
fi
model_response=$(model_client --route "$MODEL_ROUTE" --prompt-file "$PROMPT_FILE")
echo "$model_response" | jq -e '.answer and (.answer|type=="string")' >/dev/null
cache_payload=$(jq -cn --arg key "$key" --arg answer "$(echo "$model_response" | jq -r '.answer')" --arg stored_at "$(date -u +%FT%TZ)" '{status:"ok", key:$key, answer:$answer, stored_at:$stored_at}')
mkdir -p .model-cache
printf '%s' "$cache_payload" > "$cache_file"
echo "cache-miss ${key}"
model_client is a placeholder. Replace it with whichever client you use for the free model route.
The contract check matters. Never cache a response that does not parse.
Wire it into GitLab CI
In .gitlab-ci.yml, restore the cache before the job runs.
cache:
key:
files:
- prompts/release-notes/v1.json
prefix: model-cache
paths:
- .model-cache/
release-notes:
script:
- ./cache_call.sh free-model/release-notes prompts/release-notes/input.txt
The cache key changes when the prompt template changes. That invalidates stale answers without anyone remembering to bump a version.
But know the limit. GitLab cache is a performance optimization, not durable storage. Ephemeral runners can start cold.
That's why I treat the local cache as one layer. A second layer can live on a free server.
Where a free server helps
The script works inside a single pipeline. The bigger win comes when the cache is shared across jobs, branches, and runners.
I keep the same content-addressed logic behind a small HTTP endpoint. The client sends the fingerprint; the server returns the stored response or calls the free model route, stores the result, and returns it.
Because MonkeyCode's workflow includes a free server option, that shared cache can sit outside the CI worker without adding hosting cost. The free model access stays behind the cache, not in front of every retry.
Invalidation rules matter more than storage
A stale answer is worse than a wasted model call.
Rules I use:
- bump
SCHEMA_VERSIONwhen the prompt changes - namespace by model route, never reuse a response across different models
- default TTL to 24 hours
- do not cache when the input contains secrets or personal data
- do not cache for intentionally non-deterministic tasks
If the model output is allowed to vary, caching is wrong. If the task is deterministic enough that two identical requests should produce one canonical answer, caching is useful.
A five-minute test
Once the script is in place, run it twice with the same prompt.
./cache_call.sh free-model/release-notes prompts/release-notes/input.txt
./cache_call.sh free-model/release-notes prompts/release-notes/input.txt
Expected: first run prints cache-miss, second run prints cache-hit.
Then change one word in the input. You should get a new key and another miss.
Then set SCHEMA_VERSION=v2 with the same input. You should get another miss.
Then set CACHE_MAX_AGE_HOURS=0. The old file should be removed before the model is called again.
That's enough to catch the common failure modes.
When this is the wrong approach
Don't use this if:
- every request needs fresh temporal context
- the input contains sensitive data and the cache destination is shared
- the model route changes versions silently
- the output is expected to be creative or stochastic
- compliance requires you to never store model responses
In those cases, keep the direct call. But then you need a different defense: strict budget checks, sampling, or skipping the model when the change is trivial.
The goal is not to eliminate every model call. It's to eliminate calls you can prove are repeats.
Is your free model quota really too small? Before you ask for more, check how many requests share the same fingerprint.
Top comments (1)
Thanks for writing this, useful and practical. The point about keeping it simple is the one that resonates most with me. Have you found a setup that works well for you so far?