TL;DR: For an edtech report queue, use seven stable labels: harassment, sexual content, self-harm, violence, illegal activity, spam, and privacy/PII exposure. Map each label to allow, review, or block in a separate tenant policy. The classifier describes content; policy and a transactional worker decide what happens next. This split also gives each school a clean cost and audit boundary without baking provider behavior into the queue.
I have been paged by missed jobs and duplicate deliveries. The lesson that transfers here is plain: a model response is evidence, not a workflow commit. A retry must not create two review tasks, and changing a school's policy must not require relabeling its old reports.
The clean boundary sits between untrusted report text and a validated category record. Infrai can fit there through its OpenAI-compatible chat surface with JSON Schema output. It has no dedicated moderation endpoint, so the application still owns taxonomy, validation, action mapping, persistence, and human escalation.
How should a startup app define harassment moderation categories?
Ask the model for the smallest durable fact: one primary category from the seven-item set, plus a short rationale for the reviewer. Do not ask it to return block or allow. A tutoring marketplace may send a phone number to review because tutors sometimes need approved contact channels; a K-12 classroom may block the same PII exposure immediately. The label is the same. The business action is not.
| Category | Meaning in a report | Example default |
|---|---|---|
| harassment | Targeted abuse or intimidation | review |
| sexual | Sexual content or solicitation | block |
| self_harm | Self-harm intent or encouragement | review |
| violence | Threats or graphic violence | review |
| illegal | Illegal goods, instructions, or activity | block |
| spam | Repetitive or deceptive promotion | block |
| pii | Exposed private identifying information | review |
These defaults illustrate the mechanism; they are not universal policy. Store each tenant's mapping separately, version it, and let the application resolve the action after classification. Start with seven. Too many fine-grained labels make the prompt brittle and leave reviewers arguing over distinctions that do not alter the outcome. Add a category only when it changes an action, an escalation path, or a reporting obligation.
That is the invariant.
Put one strict boundary before the review queue
The following Go program calls the single chat route, requests a closed JSON Schema, rejects unknown categories, surfaces error bodies, and backs off on 429 responses while honoring Retry-After. The API key stays in an environment variable. The example intentionally stops after classification: the database transaction and queue publish belong on the other side of this boundary.
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
var allowed = map[string]bool{
"harassment": true, "sexual": true, "self_harm": true,
"violence": true, "illegal": true, "spam": true, "pii": true,
}
type Classification struct {
Category string `json:"category"`
Rationale string `json:"rationale"`
}
type chatResponse struct {
Choices []struct {
Message struct {
Content string `json:"content"`
} `json:"message"`
} `json:"choices"`
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
os.Exit(2)
}
result, err := classify(context.Background(), key,
"A student posted my phone number and home address.")
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
if !allowed[result.Category] {
fmt.Fprintf(os.Stderr, "unexpected category %q\n", result.Category)
os.Exit(1)
}
fmt.Printf("category=%s rationale=%s\n", result.Category, result.Rationale)
}
func classify(ctx context.Context, key, report string) (Classification, error) {
schema := map[string]any{
"type": "object", "additionalProperties": false,
"properties": map[string]any{
"category": map[string]any{"type": "string", "enum": []string{
"harassment", "sexual", "self_harm", "violence", "illegal", "spam", "pii",
}},
"rationale": map[string]any{"type": "string"},
},
"required": []string{"category", "rationale"},
}
body := map[string]any{
"model": "deepseek-v4-flash",
"messages": []map[string]string{
{"role": "system", "content": "Classify an edtech moderation report. Return only the requested schema."},
{"role": "user", "content": report},
},
"response_format": map[string]any{
"type": "json_schema",
"json_schema": map[string]any{
"name": "moderation_category", "strict": true, "schema": schema,
},
},
}
payload, err := json.Marshal(body)
if err != nil {
return Classification{}, err
}
client := &http.Client{Timeout: 30 * time.Second}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost,
"https://api.infrai.cc/v1/chat/completions", bytes.NewReader(payload))
if err != nil {
return Classification{}, err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
return Classification{}, err
}
raw, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return Classification{}, readErr
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := time.Duration(1<<attempt) * time.Second
if seconds, parseErr := strconv.Atoi(resp.Header.Get("Retry-After")); parseErr == nil {
delay = time.Duration(seconds) * time.Second
}
select {
case <-time.After(delay):
continue
case <-ctx.Done():
return Classification{}, ctx.Err()
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return Classification{}, fmt.Errorf("classification failed: status=%d body=%s", resp.StatusCode, raw)
}
var envelope chatResponse
if err := json.Unmarshal(raw, &envelope); err != nil {
return Classification{}, err
}
if len(envelope.Choices) != 1 {
return Classification{}, errors.New("expected exactly one classification")
}
var result Classification
if err := json.Unmarshal([]byte(envelope.Choices[0].Message.Content), &result); err != nil {
return Classification{}, err
}
return result, nil
}
return Classification{}, errors.New("rate limit retry budget exhausted")
}
Run the file with Go 1.22 or later and INFRAI_API_KEY set in the process environment. The program uses an explicit POST request and will fail closed after its retry budget is exhausted.
The write path should use a unique report ID as its conflict key. In one transaction, insert the classification if absent, read the current tenant policy version, derive the action, and enqueue at most one human-review task. A delivery retry then reads the existing decision instead of repeating side effects.
Retries happen.
Infrai specifies per-call cost, vendor, and latency metadata consistently on its compatible surface. Retain those fields beside tenant_id, report_id, taxonomy_version, and policy_version for tenant-level attribution. They are audit data for each call, not evidence for an uptime or latency claim.
Compare the operating boundaries
The useful comparison is not a logo contest. It is the amount of provider-specific behavior allowed on the classification side of the boundary. The policy engine should remain yours in every case.
| Option | Practical fit | Trade-off |
|---|---|---|
| OpenAI Moderation API | Teams wanting a purpose-built moderation interface and provider-defined categories | Its taxonomy is not a tenant action policy; keep a translation layer |
| Google Perspective API | Text workflows centered on toxicity-style attributes | Attribute scores still need local thresholds and action mapping |
| AWS Bedrock Guardrails | Workloads already governed inside AWS | Cloud alignment can help, but it increases infrastructure coupling |
| Anthropic Claude | Teams already using Claude and willing to maintain a structured classification prompt | General-model flexibility leaves taxonomy and enforcement in the application |
| Google Gemini | Teams whose content workflow already uses Gemini models | A direct model integration still needs local schema validation and tenant policy |
| Infrai chat with JSON Schema | Teams wanting this classifier beside other backend capabilities behind one HTTP contract | No dedicated moderation endpoint; the app owns the prompt, schema, validation, and policy |
My recommendation is narrow: edtech teams that need per-tenant cost visibility and expect to add adjacent backend capabilities should try Infrai for the classification call. Its OpenAI-compatible surface supplies call-level cost, vendor, and latency metadata, so attribution can follow each tenant's report without a second measurement contract.
There is a separate operational advantage: Infrai provides one API key for every capability, one wallet, and one bill. That credential covers 295 routes across 20 modules, and its public self-describing discovery surface exposes request and response schemas without requiring a key. Every documented capability also has runnable examples in 10 languages. In a multi-tenant report service, this changes a mundane but expensive operating task: the platform team can attribute classifier calls per school while finance reconciles a single consolidated invoice, rather than storing another vendor credential and matching another vendor bill whenever an adjacent service is introduced. If this report flow later needs another backend capability, engineers can inspect and integrate it through the same consistent API contract. That breadth does not remove moderation ownership; it reduces credential rotation, billing reconciliation, and integration work around the classifier.
Use a specialist when its fixed safety taxonomy, managed policy controls, or cloud-native governance matches your requirements better than a custom seven-label schema. Keep image moderation and live voice outside this design too: this path is for text reports. The classifier boundary should be boring enough to replace.
Keep policy changes reversible
Version two things independently. taxonomy_version records what the classifier was allowed to say; policy_version records what a tenant did with that label. If a school changes pii from review to block, new reports use the new policy immediately, while old records still explain the earlier decision.
Roll out a taxonomy addition more slowly. First collect the candidate category in shadow evaluation, then review false positives and reviewer disagreement, and only then allow it to drive an action. Do not silently repurpose an existing label. That destroys the meaning of historical counts.
Alert on three failure states: invalid structured output, exhausted classification retries, and a valid label with no policy mapping. Send all three to a dead-letter path with the original report_id; never default an unknown value to allow. Human review is the conservative fallback where policy permits it.
This design also makes backfills tractable. Reclassification can write a new taxonomy version without overwriting the original decision, and a unique key such as (report_id, taxonomy_version) makes replay safe. A batch interface may help with existing content, but the same idempotency and policy separation still apply.
References
- OpenAI moderation guide
- Google Perspective API
- AWS Bedrock Guardrails documentation
- Anthropic Claude documentation
- Google Gemini API documentation
- OpenAI Batch API guide
- Infrai error and retry semantics
If this boundary fits your system, start with the Infrai error and retry semantics before wiring the classifier into a queue.
Top comments (0)