Use four durable states—pending, review, allow, and block—for a backend moderation pipeline, and publish no user-generated content until a typed decision has been validated. That is the operating rule for Node.js or Go. A chat-completions JSON response is evidence, not authorization; malformed output, timeouts, and ambiguous findings belong in the human-review queue rather than on the happy path.
Short answer: accept a code-review comment, assign an idempotency key, persist it as pending, ask a model for strict JSON, validate the JSON locally, and atomically move the item to a terminal state or review. Optimize the automatic path for a latency budget, but let quality win whenever confidence or parsing falls outside an explicit policy. The useful alert is not “the dashboard looks red.” It is “which page fired, for which queue state, and can an unsafe comment now be published?”
This guide builds that boundary in Go. The transport behind Classifier can be a chat-completions service, a self-hosted gateway, or an internal model runtime; the queue contract does not change.
How should a Node.js backend queue user-generated content for moderation?
Syntactic validity proves very little. A response can decode cleanly while carrying an unknown category, a confidence outside [0,1], an empty reason, or a decision your current policy does not recognize. Treating “JSON parsed” as “moderation succeeded” turns a schema detail into a publication permission.
There is another trap: retries. Picture the failure chain with one code-review comment. A worker sends the classification request, the classifier finishes, and the network drops the reply; the worker times out at its configured deadline while the broker redelivers the message. Two workers now hold the same logical job. The first response says review, the retry says allow, and both attempt to write. Without an idempotency key and a conditional state transition, one comment can produce contradictory audit records or be published twice. With compare-and-set, exactly one transition out of pending wins, while the other becomes a recorded duplicate. This is the sort of quiet defect that makes aggregate success-rate panels comforting and incident timelines useless.
Retries change outcomes.
The state machine should be deliberately small:
| State | Meaning | Allowed next states |
|---|---|---|
pending |
Accepted, not yet classified |
allow, block, review
|
review |
Human judgment required |
allow, block
|
allow |
Approved for publication | none |
block |
Rejected by policy | none |
Four states are enough to answer the pager question. If queue age rises, operators can distinguish unclassified work from deliberate human review instead of staring at one blended backlog.
Step 1: Make the decision contract narrower than the model
Start with types that represent only decisions the application can enforce. Keep the submitted text out of routine logs; the identifier and resulting policy fields are usually enough to debug workflow failures.
package moderation
import (
"context"
"time"
)
type State string
const (
Pending State = "pending"
Review State = "review"
Allow State = "allow"
Block State = "block"
)
type Finding struct {
Category string `json:"category"`
Severity string `json:"severity"`
Confidence float64 `json:"confidence"`
Reason string `json:"reason"`
}
type ModelDecision struct {
Decision string `json:"decision"`
Findings []Finding `json:"findings"`
}
type Item struct {
ID string
IdempotencyKey string
Comment string
State State
Findings []Finding
UpdatedAt time.Time
}
type Classifier interface {
Classify(ctx context.Context, comment string) ([]byte, error)
}
type Store interface {
InsertPending(ctx context.Context, item Item) (Item, error)
Transition(ctx context.Context, id string, from, to State, findings []Finding) error
}
The Classifier returns bytes on purpose. That keeps remote success separate from local acceptance and gives the application one place to reject trailing data, unexpected fields, or policy values. A chat-completions adapter should request JSON matching this contract, but the enforcement remains here. In a Node.js backend, the interfaces and validation library will look different; the queue states, conditional transition, and fail-to-review rule should remain the same.
Step 2: Fail to review, not to publish
Decode strictly, validate semantically, and convert every uncertain outcome into review. A transient model failure and an ambiguous high-impact finding are different operational events, yet both have the same safe publication result: no automatic release.
package moderation
import (
"bytes"
"encoding/json"
"fmt"
"io"
)
var categories = map[string]bool{
"harassment": true,
"secrets": true,
"spam": true,
}
var severities = map[string]bool{
"low": true, "medium": true, "high": true,
}
func decodeDecision(raw []byte) (ModelDecision, error) {
var d ModelDecision
dec := json.NewDecoder(bytes.NewReader(raw))
dec.DisallowUnknownFields()
if err := dec.Decode(&d); err != nil {
return d, fmt.Errorf("decode decision: %w", err)
}
if err := dec.Decode(&struct{}{}); err != io.EOF {
return d, fmt.Errorf("decision contains trailing data")
}
if d.Decision != "allow" && d.Decision != "block" && d.Decision != "review" {
return d, fmt.Errorf("unknown decision %q", d.Decision)
}
for i, f := range d.Findings {
if !categories[f.Category] || !severities[f.Severity] {
return d, fmt.Errorf("finding %d has unknown policy value", i)
}
if f.Confidence < 0 || f.Confidence > 1 || f.Reason == "" {
return d, fmt.Errorf("finding %d is incomplete", i)
}
}
return d, nil
}
Do not silently discard an unfamiliar field. It may indicate a rollout mismatch: the classifier is producing a newer contract than the worker understands. Sending that item to review preserves safety while the mismatch is investigated.
Now put the latency bound around classification and the quality threshold inside policy. The example below automatically blocks a high-severity finding only at or above 0.90; everything less certain waits for a person. That number is an example configuration, not a universal safety threshold. Set it from labeled evaluations for your own code-review traffic, then version it alongside the policy.
package moderation
import (
"context"
"time"
)
type Worker struct {
Classifier Classifier
Store Store
Timeout time.Duration
}
func (w Worker) Process(ctx context.Context, item Item) error {
callCtx, cancel := context.WithTimeout(ctx, w.Timeout)
defer cancel()
raw, err := w.Classifier.Classify(callCtx, item.Comment)
if err != nil {
return w.Store.Transition(ctx, item.ID, Pending, Review, nil)
}
decision, err := decodeDecision(raw)
if err != nil {
return w.Store.Transition(ctx, item.ID, Pending, Review, nil)
}
next := Review
if decision.Decision == "allow" && len(decision.Findings) == 0 {
next = Allow
}
for _, finding := range decision.Findings {
if finding.Severity == "high" && finding.Confidence >= 0.90 {
next = Block
break
}
}
return w.Store.Transition(ctx, item.ID, Pending, next, decision.Findings)
}
The store must implement Transition as a compare-and-set: update only where the current state is pending. A duplicate worker then loses a harmless race instead of rewriting a terminal decision. Keep the losing transition observable, but do not page on a single retry.
Step 3: Put quality and latency on separate gauges
One “moderation health” percentage hides the decision you actually need to make. Measure queue age and end-to-end decision latency for capacity; measure false allows, false blocks, review rate, and per-category disagreement against a labeled sample for quality. A faster response is not an improvement when it sends more harmful comments through, and a more accurate classifier is not operationally usable if its backlog grows without bound.
The limitation is deliberate: this pipeline is not suitable for workloads that require an immediate, fully automatic answer and cannot tolerate a review queue. It trades availability of instant publication for controlled failure. It also does not determine the right policy categories or thresholds for a team; those require representative labeled examples and accountable human owners. For low-risk content where delayed publication is worse than occasional classification error, a post-publication review flow may be the better architecture, provided the product can retract content and the risk assessment supports that choice.
Use two service objectives. The first covers how quickly an item reaches any safe decision, including review; the second covers decision quality on a delayed, human-labeled sample. Do not substitute model confidence for the second objective. Confidence is an input to policy, while agreement with an independently labeled outcome is evidence about that policy.
Page on symptoms that require immediate action: the oldest pending item breaching the publication latency budget, failed persistence of a terminal transition, or evidence that blocked content can bypass the gate. Ticket contract drift and gradual review-rate changes unless they threaten the queue budget. Alerts need a state, an impact, and a runbook link. Otherwise they are nighttime trivia.
This split also makes overload behavior explicit. When classification capacity is exhausted, new items remain unpublished in pending; after the attempt times out, they move to review. Do not lower the confidence threshold merely to drain the queue. Add worker capacity, shed nonessential rechecks, or temporarily pause submissions according to the product's stated behavior.
Step 4: Verify failure paths before rollout
Test the boundary, not just the prompt. Table-driven tests should cover unknown fields, trailing JSON, invalid confidence, an empty reason, cancellation, a duplicate delivery, and a store failure during transition. Then run a shadow phase in which decisions are recorded but cannot publish or block anything; compare them with the existing review outcome before enabling enforcement.
package moderation
import "testing"
func TestDecodeDecisionRejectsUnsafeShapes(t *testing.T) {
tests := []struct {
name string
raw string
}{
{"unknown field", `{"decision":"allow","findings":[],"publish":true}`},
{"trailing object", `{"decision":"allow","findings":[]} {}`},
{"invalid confidence", `{"decision":"block","findings":[{"category":"spam","severity":"high","confidence":1.4,"reason":"repetition"}]}`},
{"unknown category", `{"decision":"review","findings":[{"category":"other","severity":"low","confidence":0.4,"reason":"unclear"}]}`},
}
for _, tt := range tests {
t.Run(tt.name, func(t *testing.T) {
if _, err := decodeDecision([]byte(tt.raw)); err == nil {
t.Fatal("expected validation error")
}
})
}
}
Roll out automatic allow and automatic block independently. They have different failure costs, so coupling them behind one percentage flag makes rollback needlessly blunt. Start with shadow decisions, enable a small slice of automatic allows after the labeled comparison is acceptable, and enable blocks only after separately examining false positives in high-severity categories.
Rollback should be one durable policy change: force every nonterminal item to review and stop workers from creating new automatic terminal decisions. Already published comments require a separate, audited remediation path; rewriting history inside the queue worker makes incident reconstruction harder. Preserve the raw model response only under an appropriate retention and access policy, and preserve the validated decision, policy version, timestamps, and transition result in the audit record.
Ship the queue before the clever prompt. At 3 a.m., a small state machine, strict decoding, and an unambiguous fail-to-review path answer the questions that matter: what was exposed, which decision authorized it, and how to stop the next one.
Know the trade-off.
Top comments (0)