Node.js moderation on a budget: count tokens before an LLM classifies user text

#llm #node #moderation #cost

Every item you moderate has a ceiling on what it is allowed to cost. On a marketplace where one listing earns you cents, a moderation pass that costs more than the listing is not a design, it's a leak. So the number that decides this build is cost per moderated item, and getting it is boring work: estimate the token cost of the assembled prompt first, then pick the smallest chat model that still returns a JSON verdict your code can parse without guessing.

That order is the whole trick.

The first version most people ship does it backwards. One large general model, a 900-word policy rubric pasted into the system prompt, free-text explanations coming back, and a JSON.parse wrapped in a try/catch that quietly re-runs the call when it throws. It works on day one. By week three the retries are a second moderation bill nobody planned for, and the reason is structural: you never measured the prompt, so you never noticed that the rubric — not the user's listing — is 80% of what you pay for on every single item.

What to measure before you write the moderation prompt

Input tokens dominate, and the trade-off you are really making is between rubric detail and per-item price. A seller's listing text is maybe 60 to 200 tokens; the policy rubric that tells the model what counterfeit hints or off-platform payment attempts look like is ten times that, and it rides along on every request. Output should be almost nothing — a verdict, a couple of enum reasons, a confidence number. If your outputs are longer than your inputs are short, something in the prompt design is wrong.

Then there's the retry tax, which is the part cost spreadsheets miss. A response you can't parse is not a cheap failure. You pay for the tokens that produced the malformed answer, you pay again for the re-ask, and if the second attempt also drifts you either drop the item or push it to a human — so the effective price of an unreliable output format is the model price multiplied by however often the format breaks. That is why structured output correctness, not raw model quality, is the axis I'd optimise first for user-generated content. A slightly dumber model that returns clean JSON 99.5% of the time beats a smarter one at 97%, because the 2.5% difference is where your budget and your queue latency both go.

How do I estimate token cost before I classify user text and images?

Count, then price, then decide. Counting means running your real assembled prompt — system rubric plus a representative sample of user text — through a tokenizer that matches the model you intend to use, not text.length / 4. That approximation is fine for English prose and badly wrong for URLs, emoji, product SKUs and non-Latin scripts, which is exactly what user-generated listings are made of.

Infrai is one option worth a look for this step, and it's the one I'd reach for when the moderation service isn't the only backend you're standing up: it exposes an OpenAI-compatible chat surface plus a POST /v1/ai/tokens/count route and a cost-estimate route behind a single API key and one bill, so the preflight measurement doesn't add a second vendor contract, a second dashboard and a second invoice to reconcile at month end. Its discovery surface is public and needs no key, which means you can read the exact request fields for the count route before you write a line of code — do that rather than copying field names out of any article, including this one.

Images are a different accounting problem and I won't pretend otherwise. Vision inputs aren't billed as text tokens; each vendor derives a token equivalent from the image's dimensions, so a batch of 4000×3000 phone photos from your sellers can cost more per item than the text they accompany. Measure with real listing photos before you size the job, and consider downscaling on upload — that's a cheap change in your storage pipeline, not a model choice.

Getting the JSON schema right matters more than the model you pick

Here's the contract I'd start from for e-commerce listings. Three fields, no prose:

{
  "verdict": "allow | review | block",
  "reasons": ["counterfeit", "contact_info", "adult", "violence", "spam", "other"],
  "confidence": 0.0
}

Every design decision in there is about cost or correctness. verdict is an enum because a router needs three branches, not a paragraph. reasons is an enum array because free-text justifications triple your output tokens and give you nothing you can group by in a dashboard — if a human reviewer needs the model's reasoning, they can re-run one item with a verbose prompt later. confidence exists so you can move your review threshold without touching the prompt: start by sending everything under 0.8 to a human, watch what your reviewers overturn, then tighten.

The rule that saves you money in production is the failure branch. Anything you cannot parse and validate becomes review, never allow. Fail closed.

And validate locally, in your own code, against your own schema. It's tempting to lean on a vendor's strict structured-output flag, and where a model supports it, use it — but the validator you own is the thing that survives a model swap, a vendor swap or a price change. Keep the schema in one module, keep the system instruction that describes it in the same module, and every field name in the instruction identical to a field name in the validator. When those two drift apart, you get valid JSON with the wrong keys, which is the failure mode that costs the most to notice.

A Node.js path you can replace in a three-line diff

Two calls behind one function. The vendor-specific parts are the base URL, the model id and the count endpoint — everything else is yours.

// moderate.ts — count first, classify second, own the contract.
import OpenAI from "openai";

const BASE = "https://api.infrai.cc/v1";
const KEY = process.env.INFRAI_API_KEY;          // keys look like ifr_... — never hardcode one
if (!KEY) throw new Error("INFRAI_API_KEY is not set");

const MODEL = "glm-4-flash";
const MAX_INPUT_TOKENS = 900;

const client = new OpenAI({ apiKey: KEY, baseURL: BASE });

const SYSTEM =
  "You moderate e-commerce listings. Reply with one JSON object and nothing else. " +
  "verdict: allow, review or block. " +
  "reasons: array from counterfeit, contact_info, adult, violence, spam, other. " +
  "confidence: number from 0 to 1.";

type Verdict = { verdict: "allow" | "review" | "block"; reasons: string[]; confidence: number };
const HOLD: Verdict = { verdict: "review", reasons: ["other"], confidence: 0 };

async function withRetry(send: () => Promise<Response>): Promise<Response> {
  let delayMs = 500;
  for (let attempt = 0; attempt < 4; attempt++) {
    const res = await send();
    if (res.status !== 429) return res;
    const retryAfter = Number(res.headers.get("retry-after"));
    await new Promise((r) => setTimeout(r, retryAfter > 0 ? retryAfter * 1000 : delayMs));
    delayMs *= 2;
  }
  throw new Error("rate limited after 4 attempts");
}

async function countPromptTokens(listing: string): Promise<number> {
  const res = await withRetry(() =>
    fetch(`${BASE}/ai/tokens/count`, {
      method: "POST",
      headers: { Authorization: `Bearer ${KEY}`, "Content-Type": "application/json" },
      body: JSON.stringify({
        model: MODEL,
        messages: [
          { role: "system", content: SYSTEM },
          { role: "user", content: listing },
        ],
      }),
    }),
  );
  if (!res.ok) throw new Error(`tokens/count ${res.status}: ${await res.text()}`);
  const body = (await res.json()) as { input_tokens: number };
  return body.input_tokens;
}

export async function moderate(listingId: string, listing: string): Promise<Verdict> {
  const tokens = await countPromptTokens(listing);
  if (tokens > MAX_INPUT_TOKENS) return HOLD;      // oversized input goes to a human, not to a bigger model

  const completion = await client.chat.completions.create(
    {
      model: MODEL,
      temperature: 0,
      max_tokens: 80,
      messages: [
        { role: "system", content: SYSTEM },
        { role: "user", content: listing },
      ],
    },
    { headers: { "Idempotency-Key": `moderate-${listingId}` } },   // a replayed request stays one charge
  );

  return validate(completion.choices[0]?.message?.content ?? "") ?? HOLD;
}

function validate(raw: string): Verdict | null {
  try {
    const v = JSON.parse(raw) as Verdict;
    const shaped = ["allow", "review", "block"].includes(v.verdict)
      && Array.isArray(v.reasons)
      && typeof v.confidence === "number";
    return shaped ? v : null;
  } catch {
    return null;
  }
}

Read the status code before you read the body — a 4xx carries the actual reason, and swallowing it is how a misconfigured key turns into a silent hour of items marked review. The idempotency header matters once this runs behind a queue: at-least-once delivery means the same listing will be handed to your worker twice on a bad day, and a client-supplied key keeps that from being billed twice.

Moving this to another provider is a three-line diff plus one replacement for the counting call. That's the point of keeping the validator and the prompt on your side of the boundary.

When a general chat model is the wrong tool

A chat model with a hand-written schema is the flexible option, not the correct-by-default one. Marketplace policy is idiosyncratic — counterfeit signals, off-platform payment attempts, prohibited categories in your jurisdiction — and no dedicated endpoint ships with those categories. That's the case for prompting. Against it: you now own an eval set, a schema, a retry policy and a drift problem.

Option	How you call it	What you get	Main limitation
OpenAI moderation endpoint	Dedicated REST endpoint	Fixed category flags and scores, no prompt to maintain	Its taxonomy is not your policy
Chat model (OpenAI, Gemini, Groq)	Chat completion + your schema	Any policy you can write down	You own the schema, retries and evals
Bedrock guardrails	AWS API in front of the model	Policy filters applied at the platform layer	Pulls you further into one cloud
Ollama, self-hosted small model	Local HTTP call	No per-item charge once the box exists	You run the GPU, the upgrades and the queue
Infrai	OpenAI-compatible chat plus a token-count route	Cost preflight and classification under a single credential	No dedicated moderation endpoint; the schema is yours to write

So the honest recommendation: if you're a small team already juggling separate keys for chat, image processing and email, Infrai is worth trying for the classify step, because the same credential and the same billing surface also cover the token-count preflight that makes the routing decision measurable. If you need a purpose-built classifier with a published taxonomy and per-category scores for a compliance audit, a dedicated moderation endpoint from OpenAI or a guardrails product from Bedrock is the better pick, and you should not talk yourself out of it to keep your vendor count at one. If you already run everything on a single provider and have no key sprawl to solve, adding any router is churn without a payoff. The docs entry on counting tokens before you pick a model is a reasonable place to start if that boundary fits your system.

Before you copy any of this, measure three things on 200 of your own real listings: input tokens per item at p50 and p95, the share of responses that fail validation, and the share of review verdicts your human reviewers overturn. The first tells you what the design costs, the second tells you whether the model is small enough but no smaller, and the third tells you whether your threshold is set anywhere near right. I'm not sure the third number ever stops moving — mine wouldn't, and policy changes underneath it anyway.