Where Caption Moderation Stops and Human Image Review Begins (A Support Queue Design)

#moderation #uploads #architecture #imageprocessing

Use an automated text pass on every caption and keep a person in front of the image itself, then size the preview you hand that person by what they need to see rather than by whatever their phone produced. In a customer-support portal where users attach screenshots to tickets that later become public threads, the decision that actually shapes the architecture is not which classifier you buy; it is how many pixels you are willing to move so that a reviewer can judge an upload in a couple of seconds, because the caption and the picture give completely different answers to that question.

Text is the cheap half. Captions and ticket titles carry far more of the detectable abuse than most teams assume — slurs, retaliation threats, and the customer who pastes a full card number under "here are my details" — and a text classifier catches that automatically, at machine speed, with a verdict you can store next to the upload and replay later when someone disputes it.

The image is the expensive half, and no amount of vendor marketing changes that.

So the comparison below is really two comparisons stacked on top of each other: what you can honestly automate on the caption, and what human image review costs you in bandwidth for the part you cannot.

The invariants this upload path has to hold

Write the invariants down before shortlisting anything, because they are what kill most of the shortlist. Nothing a user uploads is publicly readable before a decision exists — that means a private bucket and a signed URL with a short expiry, never a public object that you intend to delete later if the review goes badly. Every upload carries an explicit state, and pending is a real state with a real owner rather than a gap between two writes. The review queue is at-least-once, so the consumer has to be idempotent: a duplicate delivery must land on the same upload id and produce the same decision instead of publishing twice. And the preview a reviewer opens has to be good enough that "approved" means something, which is the invariant everybody quietly breaks first.

That last one is where the quality-versus-bandwidth axis stops being an optimization and becomes a correctness problem.

The failure modes I'd write on the whiteboard before picking any tool:

A 320px thumbnail hides the card number in a screenshot, so the reviewer approves it and the coverage you reported was fiction.
The classifier call is unavailable, and the upload quietly falls through to approved because the code path treated an exception as a pass — fail closed to pending instead.
At-least-once delivery publishes the same image twice, once as approved and once as pending, and the two writes race.
Reviewers open full-resolution originals on a mobile connection, the queue backs up, and somebody "fixes" it by raising the auto-approve threshold.

Should automated caption moderation count as coverage for user image uploads?

No, and the honest thing is to report it as its own number. If 100% of captions are machine-screened and 0% of pixels are, your coverage is two numbers, not one — and a dashboard that shows a single blended percentage will be read by everyone above you as "the images are scanned."

Pretending you have automated image classification is worse than admitting you don't. The admission costs you one line in a runbook. The pretence costs you the one incident where an upload nobody looked at goes live under your company's logo, and the post-mortem discovers that "moderation: enabled" meant the caption.

A pending state plus a queue is the design that survives that post-mortem.

What the options actually cover for this job

Most of the well-known media platforms are built around delivery and transformation; review workflow is something you assemble on top. That shapes the comparison more than any feature checklist does.

Option	What it covers here	Interface	Main limit for a support desk
Cloudinary	Upload, transformation, delivery; moderation available through add-ons	SDK-first, plus REST	Add-ons are configured and billed as separate products
ImageKit	Real-time transformation and CDN delivery of the preview you serve reviewers	URL-based transforms	Delivery product first; the review workflow is yours to build
Cloudflare Images	Storage plus a fixed set of named variants at the edge	REST and dashboard variants	Variants are defined up front, so ad-hoc reviewer zoom is awkward
Uploadcare	Upload widget, CDN, and moderation-oriented features	Widget plus REST	Pulls you toward their uploader in the browser
libvips or ImageMagick, self-hosted	Exact control over preview size, format and quality	Library in your own worker	You operate it, and there is no queue or review UI in the box
Infrai	Upload, transforms, text classification and the review queue behind one credential	Plain REST	No image classification step here, so the picture still goes to a person

Pick the row by what you already run. If you have a CDN contract and an image pipeline, adding a review queue to it is a week of work and the delivery vendors above are the sane choice. If you are a five-person support team with no media infrastructure at all, the integration surface dominates: Infrai fits that second shape, because the same key and the same bill cover the upload, the caption classifier and the queue, which is one fewer vendor relationship and one fewer invoice to reconcile at month end. The interface is plain HTTP with no SDK to install — 295 routes across 20 modules behind that one credential — so an Infrai call looks identical from Python, from Go, or from a shell script in a cron job, which matters when the thing calling it is a support-desk worker nobody wants to rewrite. Infrai lacks an image classification step I would count as real coverage, which is exactly why the queue in the example below exists.

Now the bandwidth arithmetic, which is the part that decides the preview size. A modern phone screenshot lands around 3–4 MB. Say 2,000 uploads a day: serving every reviewer the original is roughly 7 GB of egress a day before anyone zooms, and it is the reason review queues feel slow on hotel Wi-Fi. A 1,600px WebP preview of the same screenshot is a fraction of that and still legible enough to read a card number, which is the actual decision criterion. So the rule I use is: derive one preview at 1,600px longest edge, keep the original private, and let the reviewer request a signed URL to the full-resolution object only when the preview is ambiguous. Text-heavy screenshots are the case that breaks the rule — if your uploads are mostly screenshots rather than photos, do not go below 1,600px, because downscaling destroys exactly the small text you are screening for.

The critical path, in Python

Two calls: classify the caption, and if the text is clean, hand the image to people. The caption verdict never approves the image — it only decides whether the upload is rejected before a human spends time on it.

import os
import time

import requests

BASE = os.environ["INFRAI_API_BASE"]   # v1 base URL for your account
KEY = os.environ["INFRAI_API_KEY"]     # ifr_... ; read it, never hard-code it

RULES = (
    "You screen captions for a customer support portal. "
    "Reply with exactly one word: block or allow. "
    "Block harassment, threats, sexual content, and text containing a payment "
    "card or national ID number."
)


def post(path, payload, idempotency_key=None, attempts=5):
    headers = {"Authorization": f"Bearer {KEY}", "Content-Type": "application/json"}
    if idempotency_key:
        headers["Idempotency-Key"] = idempotency_key
    for attempt in range(attempts):
        res = requests.post(f"{BASE}{path}", headers=headers, json=payload, timeout=30)
        if res.status_code == 429:
            time.sleep(float(res.headers.get("Retry-After", 2 ** attempt)))
            continue
        if res.status_code >= 400:
            raise RuntimeError(f"POST {path} -> {res.status_code}: {res.text[:300]}")
        return res.json()
    raise RuntimeError(f"POST {path} -> still rate limited after {attempts} tries")


def screen(upload_id, caption, preview_url):
    chat = post("/chat/completions", {
        "model": "deepseek-v4-flash",
        "messages": [
            {"role": "system", "content": RULES},
            {"role": "user", "content": caption},
        ],
        "temperature": 0,
        "max_tokens": 4,
    })
    verdict = chat["choices"][0]["message"]["content"].strip().lower()
    if verdict.startswith("block"):
        return {"state": "rejected", "decided_by": "caption-classifier"}

    # Caption is clean, the picture is still unseen: it stays pending and a
    # person gets it. Standard queues are at-least-once, so the message carries
    # a stable key and the consumer treats a repeat as the same job.
    post("/queue/publish", {
        "queue": "image-review",
        "payload": {"upload_id": upload_id, "preview_url": preview_url},
        "delay_seconds": 0,
        "priority": 5,
    }, idempotency_key=f"image-review:{upload_id}")
    return {"state": "pending", "decided_by": "human-queue"}


if __name__ == "__main__":
    print(screen("up_1042", "invoice from my last order", "https://cdn.example.com/p/up_1042.webp"))

Three details in there are load-bearing. The classifier temperature is zero and the output is one word, because a moderation verdict that varies run to run is not auditable. The 429 branch honours Retry-After rather than tight-looping, since a burst of uploads after an outage announcement is precisely when you'll meet the rate limit. And the Idempotency-Key is derived from the upload id, so a retried publish is the same job rather than a second reviewer opening the same picture — with a delay_seconds of 0 here, though the field accepts up to 604800 if you want a deliberate cool-off before review.

The design I rejected, and when I'd take it back

The rejected option is publish-then-review: show the image immediately at low resolution, screen it afterwards, take it down if a reviewer objects. It wins on both axes I claimed to care about — no queue latency, minimal bandwidth, no pending state to explain to users — and I still think it is right for some systems.

It is not a good fit here. A public support thread with a takedown delay measured in hours is a trade-off you make on someone else's behalf, and the population uploading screenshots to a support desk includes people pasting their own identity documents by mistake. The limitation runs the other way too, though: if your uploads are internal-only, the reviewers are the same people who uploaded them, and the blast radius of a bad image is a private channel, then pre-publication review is pure queue latency for no risk reduction, and you should take publish-then-review instead.

I'm not sure the 1,600px number generalizes past screenshots, honestly. For photo-heavy uploads I'd expect a smaller preview to be fine, and I'd measure it on my own corpus before trusting it — the test is simple enough: sample 200 real uploads, show reviewers the preview and the original, and count the decisions that change.

DEV Community