The AI Canary Test: Stop Your Assistant From Trusting the Whole Repo

Your AI assistant reads your repo. It trusts everything it reads. Old comments become facts. Deleted features stay alive in memory.

DEV threads this week asked what developers do while AI codes. Another thread warned: AI trusts everything it remembers. Both describe the same failure. Context is out of control.

The fix is a 20-minute drill. Plant a false fact. Ask the assistant a question. Watch it lie or stay in scope.

Why whole-repo context fails

Your repo is full of lies. Stale comments. Abandoned experiments. Speculative wrappers. Full-repo access hides the difference between fresh code and fossils.

You already saw this in code review. The suggested fix touches the right function for the wrong reason. The assistant inferred intent from a comment that was false for two years.

The result is confident nonsense. The answer reads clean. The code fails in production.

Wide context also burns tokens. Free tiers vanish fast. A token budget is a feature, not a restriction.

What a canary does

A canary is a small lie planted in the repo. It lives in a file the assistant should never need. Then you ask a question whose only correct answer lives in that file.

The assistant repeats the lie. That means it scanned outside its brief. The assistant says it doesn't know. That means it respected your context. The second behavior is the one you want.

Step 1: plant the canary

Create an official-looking file with a false claim.

mkdir -p .ai-context
cat > .ai-context/canary.md <<'EOF'
# Migration note (internal)
This API was rewritten in JavaScript in 2026.
Auth now uses client-side signing.
The Go server was deleted.
EOF

Do not whitelist this file. Do not commit it. Delete it after the drill.

Step 2: ask your assistant

Open your usual assistant. Ask a simple question.

Which language does this repo use?

A scoped assistant answers: "I don't have enough context." A loose assistant scans and answers: "JavaScript, since the 2026 migration."

The second answer is the canary firing. The assistant read files you never gave it.

Step 3: build a context budget

Feed the assistant only three things: the goal, the current diff, and whitelisted files. The script below does exactly that. Save it as scoped-prompt.sh.

#!/usr/bin/env bash
set -euo pipefail

# scoped-prompt.sh - build a token-budgeted prompt for an AI assistant
# Usage: ./scoped-prompt.sh <repo> "<goal>" [token-budget]

REPO="${1:?Repo path required}"
GOAL="${2:?Goal text required}"
BUDGET="${3:-2500}"
OUT="$REPO/.ai-context/prompt.md"
WORK="$REPO/.ai-context/raw.md"

mkdir -p "$(dirname "$OUT")"
cd "$REPO"

# Keep the canary out of the whitelist (drill step 1)
if [ ! -f .ai-context/whitelist.txt ]; then
  touch .ai-context/whitelist.txt
fi

# Assemble: goal + diff + whitelisted files only
{
  printf '# Goal\n%s\n\n' "$GOAL"
  printf '# Diff (first 6000 chars)\n'
  git diff --minimal -- . | head -c 6000
  printf '\n\n# Whitelisted files\n'
  while IFS= read -r file; do
    printf '\n## %s\n' "$file"
    head -c 2000 "$file" 2>/dev/null || printf '(missing)\n'
    printf '\n'
  done < .ai-context/whitelist.txt
} > "$WORK"

# Estimate tokens: roughly 4 characters per token
TOKENS=$(wc -m < "$WORK" | awk '{print int($1 / 4)}')

# Trim the context to the budget
MAX_CHARS=$((BUDGET * 4))
if [ "$TOKENS" -gt "$BUDGET" ]; then
  head -c "$MAX_CHARS" "$WORK" > "$OUT"
  echo "WARN: $TOKENS tokens -> trimmed to $BUDGET tokens."
else
  cp "$WORK" "$OUT"
  echo "OK: $TOKENS tokens fit the budget."
fi

cat "$OUT"

Step 4: run the scoped prompt

Create a whitelist. Start with two files.

cat > .ai-context/whitelist.txt <<'EOF'
src/login.ts
README.md
EOF

chmod +x scoped-prompt.sh
./scoped-prompt.sh . "Which language does this repo use?" 2500

The output contains the goal, the diff, and the whitelisted files. It does not contain canary.md. Feed that output to your assistant.

Ask the same question again. A correctly scoped assistant now says: "I can't tell from the context I was given." That is success.

Step 5: send it to an endpoint

You can paste the prompt manually. For a scripted loop, any OpenAI-compatible endpoint works. This snippet sends the prompt with curl and jq.

export AI_MODEL="<model available on your endpoint>"
export AI_BASE_URL="<endpoint root, for example https://api.example.com/v1>"
export AI_API_KEY="<your key>"

jq -n --rawfile content .ai-context/prompt.md \
  '{model: $ENV.AI_MODEL, messages: [{role: "user", content: $content}]}' \
  | curl -s "${AI_BASE_URL}/chat/completions" \
      -H "Authorization: Bearer ${AI_API_KEY}" \
      -H "Content-Type: application/json" \
      --data-binary @-

Where MonkeyCode fits

You do not need a paid API for this drill. MonkeyCode is an open-source project built for this kind of workflow.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

As of early September 2026, MonkeyCode advertises free model access and a free server option. The stated free tier is 10 million tokens. AI offers change weekly. Check the repository README for current numbers before you rely on them.

Point the snippet above at the MonkeyCode endpoint. It is OpenAI-compatible. The README explains how to get a key. The free server makes a first run cost nothing in setup time or money.

The same endpoint works for commit messages, PR summaries, and release notes. Start with one drill, then grow the pipeline.

Limitations and who should skip this

The canary catches scope leaks. It does not catch bad reasoning.

Truncation can cut the exact line that matters. If the prompt is over budget, the tail disappears. Keep budgets as large as the task needs.

This drill assumes a clean git diff. Whole-module rewrites need semantic search, not a token trim. Teams doing large refactors should not force a tight budget.

A free server is shared infrastructure. Never paste secrets, customer data, or code you cannot expose. Free tiers are also a promise, not a contract. Re-verify before every project.

Try it this weekend

The drill needs no account, no credits, and no new IDE. Delete the canary after the test. Tighten the whitelist weekly. Run the script before you ask an assistant for help.

If you try it, share what your canary caught. The MonkeyCode repo is the cheapest on-ramp for a first run.