DEV Community

Casey Sun
Casey Sun

Posted on

The Do-Not-Call List for Free AI Tiers: A Request-Level Policy You Can Version

A weekly dependency bot opens a pull request against a release branch. A bot posts an AI-written summary of the diff into the PR thread. Nobody reviews that summary before merge. That is the whole incident, and it needed no attacker.

This is a composite of ordinary failure modes, not a customer report. It happens because teams gate at the pipeline level. The risky call is usually a single request inside a green pipeline.

This article is an anti-use field guide. It teaches where a free AI lane must not be the caller, and what to call instead.

Free changes the risk, not the task

A paid tier and a free tier can run the same prompt. The difference is who owns the failure. Free access usually means fewer contractual guarantees, shared capacity, and terms you did not negotiate.

So the question is never "is the model good enough". The question is "what is the blast radius when this specific call is wrong". Classify the request, then pick the lane.

Four questions before any call

Ask these for every request, not every service.

  1. Data class — public, internal, PII, or secret?
  2. Sink — draft, comment, signed artifact, production config, or money?
  3. Reversibility — can you undo the effect in one step?
  4. Tool surface — does the caller hold shell, network, or write credentials?

A request that is public, reversible, and tool-free is a fine candidate for a free lane. A request that is internal, irreversible, and shell-capable is not.

Put the policy in data, not in a wiki

Policies in prose drift. Policies in a file can be hashed, diffed, and pinned per release.

Start with a small JSON policy. Keep it boring on purpose.

{
  "version": 3,
  "hard_deny": {
    "data_class": ["secret", "pii"],
    "sink": ["signed_artifact", "prod_config", "money"]
  },
  "sandbox_only": {
    "data_class": ["internal"],
    "agent_tools": ["shell", "network"]
  },
  "allow": {
    "task": ["summarize", "rename_suggestion", "changelog_draft"]
  }
}
Enter fullscreen mode Exit fullscreen mode

Note the shape. Deny rules are absolute. Sandbox rules are advisory and require human review. The allow list is explicit, so unknown tasks fall to review instead of shipping.

The classifier

This runs before the model call. It is stdlib-only and short enough to read in a review.

# classify.py — Python 3.11+, no third-party deps
from __future__ import annotations

import hashlib
import json
import sys
from dataclasses import dataclass
from enum import Enum
from pathlib import Path


class Decision(str, Enum):
    ALLOW = "ALLOW"
    SANDBOX = "SANDBOX"
    DENY = "DENY"


@dataclass(frozen=True)
class Request:
    task: str
    data_class: str          # public | internal | pii | secret
    sink: str                # draft | comment | signed_artifact | prod_config | money
    reversible: bool
    human_in_loop: bool
    agent_tools: tuple[str, ...] = ()


def load_policy(path: Path) -> tuple[dict, str]:
    raw = path.read_bytes()
    return json.loads(raw), hashlib.sha256(raw).hexdigest()[:12]


def classify(req: Request, policy: dict) -> tuple[Decision, list[str]]:
    reasons: list[str] = []

    for name, blocked in policy["hard_deny"].items():
        if getattr(req, name) in blocked:
            reasons.append(f"hard_deny:{name}={getattr(req, name)}")
    if reasons:
        return Decision.DENY, reasons

    for name, gated in policy["sandbox_only"].items():
        value = getattr(req, name)
        hit = any(v in value for v in gated) if isinstance(value, tuple) else value in gated
        if hit:
            reasons.append(f"sandbox_only:{name}={value}")

    if req.task not in policy["allow"]["task"]:
        reasons.append(f"task_not_allowlisted:{req.task}")

    if not req.reversible and not req.human_in_loop:
        reasons.append("irreversible_without_review")

    return (Decision.SANDBOX if reasons else Decision.ALLOW), reasons or ["clean"]
Enter fullscreen mode Exit fullscreen mode

The policy hash is the important line. Every decision record carries the exact policy version that produced it.

Log every decision, then query it

The log is JSONL. One line per request. It is the artifact you will actually use during an incident.

def emit(req: Request, decision: Decision, reasons: list[str], policy_hash: str, out=sys.stdout) -> None:
    record = {
        "policy": policy_hash,
        "decision": decision.value,
        "reasons": reasons,
        **{k: list(v) if isinstance(v, tuple) else v for k, v in req.__dict__.items()},
    }
    out.write(json.dumps(record) + "\n")
Enter fullscreen mode Exit fullscreen mode

Run it against a batch and read the distribution before you trust it.

python classify.py policy.json < requests.jsonl > decisions.jsonl
jq -r 'select(.decision != "ALLOW") | .reasons[]' decisions.jsonl \
  | sort | uniq -c | sort -rn | head
Enter fullscreen mode Exit fullscreen mode

That second command is the debugging workflow. It tells you which rule fires most. If task_not_allowlisted dominates, your policy is too narrow. If hard_deny:sink=signed_artifact fires often, you have a design problem upstream.

Pin the decisions with tests

A classifier without tests is a comment. Parametrize the matrix and run it in CI.

# test_classify.py
import pytest
from classify import Decision, Request, classify

POLICY = {
    "version": 3,
    "hard_deny": {"data_class": ["secret", "pii"],
                  "sink": ["signed_artifact", "prod_config", "money"]},
    "sandbox_only": {"data_class": ["internal"], "agent_tools": ["shell", "network"]},
    "allow": {"task": ["summarize", "rename_suggestion", "changelog_draft"]},
}

CASES = [
    (Request("summarize", "public", "draft", True, False), Decision.ALLOW),
    (Request("summarize", "pii", "draft", True, True), Decision.DENY),
    (Request("summarize", "public", "prod_config", True, True), Decision.DENY),
    (Request("summarize", "internal", "comment", True, True), Decision.SANDBOX),
    (Request("deploy", "public", "draft", False, False, ("shell",)), Decision.SANDBOX),
]

@pytest.mark.parametrize("req,expected", CASES)
def test_matrix(req, expected):
    assert classify(req, POLICY)[0] is expected
Enter fullscreen mode Exit fullscreen mode

The fifth case is the one that saves you. A capable agent on an unlisted task lands in review, not in production.

Red flags and what to call instead

Most free-lane failures have a cheaper replacement. Reach for that first.

Red flag Why the free lane is wrong Better alternative
Output signs or approves anything No accountability trail on a shared tier Deterministic signer, human approver
Input carries PII or secrets Terms and retention are not yours Redaction step, then classify again
Caller holds shell or network tools Irreversible side effects, no rollback Sandbox with no credentials
Same prompt runs on every commit Cost and drift scale silently Cached fixture, golden-file test
Output lands in a config or migration One bad token is an outage Schema validation, then review
Task is formatting or renaming Deterministic problem, probabilistic tool Formatter, AST rewrite, linter
Latency sits on a user request path Shared capacity, unknown queueing Precompute, or a committed tier

Print that table. Tape it to the review checklist. The first three rows are hard stops.

Exit criteria for the free lane

Write these before adoption, not after the first incident. Trigger any one and move the workload.

  1. P95 latency exceeds your user-facing budget for two consecutive weeks.
  2. Error or refusal rate crosses the threshold you set at adoption.
  3. Any request gains write credentials or touches a signed path.
  4. The data classification changes, even if the task does not.
  5. A review finds that nobody read the sandbox output for a full sprint.
  6. Output quality drifts on a golden set you can no longer explain.

Criterion five is the quiet one. A sandbox nobody reads is a production lane with extra steps.

Where a free lane legitimately fits

MonkeyCode ships as an open-source project with a free model allowance and a free server option. Those are operator-stated availability claims.

Disclosure: This article was prepared as part of MonkeyCode's product outreach.

Treat both as shared capacity, not as entitlements. Quotas, limits, and terms change, so read the project's current documentation instead of trusting any blog post, including this one.

In this policy, that lane maps to task in allow, data_class == public, sink == draft, and agent_tools == (). Summarize a public doc. Draft a changelog entry a human will edit. Suggest a test name. Keep the classifier in front of it, and the free server stays a staging surface rather than a signing surface.

If you want to test the workflow, start with the summarizer task. Leave the deploy bot out of it.

Limitations and who should not use this

This classifier is a policy check, not an enforcement boundary. A caller can pass wrong labels. Pair it with egress rules, scoped credentials, and separate identities per lane.

Skip this approach in three cases:

  • You have no data classification scheme. Build that first, or the labels are guesses.
  • You cannot review sandbox output within a stated SLA. Then every sandbox is really an allow.
  • You operate regulated workloads with obligations you have not read against the vendor terms.

The pattern is small on purpose. Copy the two files, replace the allow list with your real tasks, and run the test matrix once. If more than half the cases land in DENY, your pipeline shape is the problem, and no tier will fix it.

Top comments (0)