A Two-Prompt Watchdog for the Free Model Endpoint You Already Rely On

Here is why this is worth reading: the current AI news cycle keeps circling around detecting generated text and making model outputs identifiable, but a solo builder with a free endpoint has a smaller version of that same problem. You do not need to watermark text. You need to know when the model behind your endpoint changed the shape of its answer and broke your parser. This guide builds a two-prompt watchdog that detects that drift, using MonkeyCode's free model access and the free server option you can leave running without adding a paid host.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

The problem is not the model, it is the silent change

Free model endpoints frequently change routing, quantization, context handling, and instruction-following behavior without a changelog. You notice it when your JSON parsing fails, or your shell helper starts treating a text response as a command. The failure is often not the model being "wrong." It is the output contract drifting.

For a solo project, you rarely need a heavy observability stack. You need a cheap check that runs on a schedule, sends the same two inputs every time, and tells you when the normalized output no longer matches the previous run.

What the watchdog does

The watchdog is deliberately small:

1. It sends two fixed prompts to the configured model endpoint.
2. It normalizes each response with trim().toLowerCase().
3. It computes a short SHA-256 digest, not the raw output, so history files stay small.
4. It records whether a required JSON key survived in the response.
5. If the digest or key status changes from the previous run, the shell wrapper prints a diff and can send a notification.

This is not a benchmark, so it will not tell you whether one model is better than another. It only tells you whether the endpoint you depend on changed in a way that your application should care about.

Build the probe script

Create a small Node.js project with TypeScript:

mkdir drift-watchdog && cd drift-watchdog
npm init -y
npm install -D typescript @types/node tsx

Save this as probe.ts:

import { createHash } from "node:crypto";

type Probe = {
  id: string;
  prompt: string;
  expectKey?: string;
};

const probes: Probe[] = [
  {
    id: "strict-json",
    prompt:
      'Return only JSON with keys "city" and "temp_c". No prose.',
    expectKey: "temp_c",
  },
  {
    id: "http-method",
    prompt:
      "Answer with exactly one word: the HTTP method used to update a resource.",
  },
];

const endpoint = process.env.MONKEYCODE_API_URL;
const token = process.env.MONKEYCODE_TOKEN;

async function callModel(prompt: string): Promise<string> {
  if (!endpoint || !token) {
    throw new Error("Set MONKEYCODE_API_URL and MONKEYCODE_TOKEN");
  }

  const response = await fetch(endpoint, {
    method: "POST",
    headers: {
      "content-type": "application/json",
      authorization: `Bearer ${token}`,
    },
    body: JSON.stringify({
      prompt,
      max_tokens: 24,
      temperature: 0,
    }),
  });

  if (!response.ok) {
    throw new Error(`HTTP ${response.status}`);
  }

  return (await response.text()).trim();
}

function digest(value: string): string {
  return createHash("sha256")
    .update(value.toLowerCase())
    .digest("hex")
    .slice(0, 12);
}

async function main() {
  const results: string[] = [];

  for (const probe of probes) {
    const output = await callModel(probe.prompt);
    const shortDigest = digest(output);
    const keyStatus = probe.expectKey
      ? output.includes(probe.expectKey)
        ? "key-present"
        : "key-missing"
      : "-";

    results.push(`${probe.id}\t${shortDigest}\t${keyStatus}`);
  }

  console.log(results.join("\n"));
}

main().catch((error) => {
  console.error(
    `probe failed: ${error instanceof Error ? error.message : error}`,
  );
  process.exit(1);
});

Set your endpoint and token as environment variables before running:

export MONKEYCODE_API_URL="your-monkeycode-endpoint"
export MONKEYCODE_TOKEN="your-token"
npx tsx probe.ts

You should see two rows similar to:

strict-json  9f3b2c1a4e6d  key-present
http-method  7d1a9e0b3c5f  -

This is a ready-to-adapt scaffold, not a guarantee of a specific output from any live endpoint. Adjust the prompts to match your own account before trusting the result.

Add a failure fixture you can verify

To prove the watchdog catches the failure mode you care about, change the expected key to one the model is unlikely to return. For example, set expectKey to "humidity_pct" on the strict JSON probe:

expectKey: "humidity_pct",

Run the probe again. If the model returns the original keys, the status becomes key-missing, the digest row differs, and the shell wrapper flags a drift. That is the same signal you want when a model silently stops respecting "Return only JSON" or renames a field you parse downstream.

Change expectKey back to "temp_c" before deploying.

Run it on a free server

Put the project on a small free server that can stay online. A cron job every six hours is enough for a change detector; you are not monitoring availability, you are monitoring behavior.

Create drift-watchdog.sh:

#!/usr/bin/env bash
set -euo pipefail

LOG_FILE="${LOG_FILE:-./probe-results.tsv}"
STATE_FILE="${STATE_FILE:-./probe-state.tsv}"

npx tsx probe.ts > "$LOG_FILE"

if [[ -f "$STATE_FILE" ]]; then
  if ! cmp -s "$STATE_FILE" "$LOG_FILE"; then
    echo "drift detected" >&2
    diff "$STATE_FILE" "$LOG_FILE" || true

    if [[ -n "${NTFY_TOPIC:-}" ]]; then
      curl -s -d "Model drift detected on $(hostname)" \
        "https://ntfy.sh/$NTFY_TOPIC" || true
    fi
  fi
fi

cp "$LOG_FILE" "$STATE_FILE"

Make it executable and add a cron entry:

chmod +x drift-watchdog.sh
crontab -e

Add:

0 */6 * * * cd $HOME/drift-watchdog && LOG_FILE=$HOME/watchdog-runs/last.tsv STATE_FILE=$HOME/watchdog-runs/state.tsv ./drift-watchdog.sh >> $HOME/watchdog-runs/watchdog.log 2>&1

Create the log directory first:

mkdir -p $HOME/watchdog-runs

The state file grows to only a few hundred bytes, no matter how many runs you perform.

Cost boundary and abandonment criteria

With two prompts capped at 24 tokens each, a run uses at most 48 output tokens plus the fixed prompt tokens. Six runs per day gives roughly 360 output tokens daily, or about 10,800 tokens in a 30-day month. If MonkeyCode's advertised allowance at the time of writing is 30 million tokens, this watchdog consumes less than 0.04% of the trial capacity each month, leaving the rest for actual work.

Set two exit conditions before you rely on this:

• If a probe ever uses more than 5% of your monthly allowance, stop and reduce the frequency to twice a day.
• If the endpoint returns a non-JSON error for three runs in a row, pause the watchdog and test the endpoint manually. The watchdog's job is behavior change, not endpoint health; a broken endpoint should fail your application checks anyway.

Who should not use this

Do not use this if you need to compare model quality, measure latency under load, or detect content policy changes. A two-prompt digest is too thin for those questions. Do not use it as an availability monitor; a cron job every six hours will miss the small blips you can tolerate and catch the long behavior changes that matter.

The one thing to try today

Take the two prompts you already depend on in your own side project, wrap them in a digest comparison, and run the first state snapshot. If the next scheduled run is identical, you have a working baseline. If it is not, you found a drift before your user did.

The free endpoint and free server from MonkeyCode are useful here because the experiment costs days, not dollars. Run the watchdog with a small token budget, keep the state file under version control, and treat every diff as a release note the endpoint did not publish.

What is the smallest output contract in your current project that would hurt if it silently changed?