robertmiller4179

Posted on Oct 24

Text Content Moderation API vs JSON Schema Chat Completions: CRM Reliability (and Why)

#airuntime #contentmoderation #sre

For an OpenAI moderation API versus GPT-4o-mini chat completions comparison, the least complex reliable design is a four-stage contract: receive the sales-call transcript, classify its policy status, propose a CRM action, and commit that action with an idempotency key. Keep those stages observable and independently replayable. A moderation result should never be treated as permission to write a CRM record.

TL;DR: choose a narrow moderation response when the next decision is only policy-related. Use JSON Schema-constrained generation when the system needs a structured interpretation such as a follow-up task. The important boundary is ownership of the side effect. Persist the policy decision and the proposed action separately, attach a policy version to both, and make a replay return the previous CRM result instead of creating another task.

This is a marketplace workflow. A transcript comes in after a sales call; the system decides whether it can be processed and, if it can, turns the useful parts into CRM actions. The practical risk is not an abstract model score. It is a bad or repeated follow-up becoming operational data.

The page I want to see before a duplicate task exists

The page fires when a transcript has crossed the business freshness limit without a terminal policy decision, or when the number of action attempts exceeds the number of confirmed CRM writes. The on-call view should show the oldest job, its state, the policy version, the transcript revision, and the action key. A line that says “AI latency is elevated” gives me very little to do.

The diagnostic path matters. If queue wait is normal but model duration is high, inspect the model dependency and its timeout budget. If model duration is normal but CRM wait is high, stop blaming the classifier and inspect the adapter. If action attempts rise while confirmations stay flat, freeze new writes and reconcile the ledger. Those are different runbooks, even though all three can raise the same end-to-end latency chart. A useful page narrows the first action without asking the operator to read a transcript or guess which revision was evaluated.

The useful alert is specific: 184 transcripts are waiting for a policy decision, the oldest is 11 minutes old, and 31 action keys have no confirmed write. That is enough to inspect queue health, the moderation dependency, and the CRM adapter in that order. It also avoids putting transcript text into ordinary logs.

Work backwards from that page. The job needs durable state, a moderation record, an action record, and a write ledger. Each state transition should be safe to replay. A network timeout after a CRM request is the awkward case: the worker cannot know whether the downstream system committed the request, so it must retry with the same key and rely on idempotent behavior at the boundary.

I use a key made from the call identifier, transcript revision, and action type. A new revision is a new business input. A retry is not. That distinction prevents a harmless queue replay from looking like a second customer follow-up.

package workflow

type ModerationDecision struct {
    Allowed      bool   `json:"allowed"`
    Category     string `json:"category"`
    PolicyVersion string `json:"policy_version"`
}

type CRMAction struct {
    CallID       string `json:"call_id"`
    TranscriptID string `json:"transcript_id"`
    Action       string `json:"action"`
}

func actionKey(action CRMAction) string {
    return action.CallID + ":" + action.TranscriptID + ":" + action.Action
}

The shape is intentionally boring. Boring is good at an integration boundary.

Stop there.

Should I use a moderation API or structured chat completions?

The moderation call owns one question: can this content proceed under the current policy? Its output should be small enough that the worker can persist it with the transcript revision and policy version. It does not own CRM authorization, and it does not decide whether a seller deserves a follow-up.

The structured generation call owns interpretation. It may identify a proposed action, a small set of fields, and a reason for review. JSON Schema can make the response structurally checkable; it cannot establish that the proposed action is correct for the account or authorized for the worker. A valid object can still contain the wrong action.

That gives each result a different test. Policy classification needs labeled content and an explicit review path for uncertain cases. Action planning needs fixtures for missing fields, contradictory statements, and unsupported operations. The CRM adapter needs idempotency tests. Combining those tests behind one “successful response” counter makes an incident harder to diagnose.

The tempting implementation is one request that classifies the transcript, summarizes it, and emits a write-ready object. It has fewer calls in the diagram, but more meanings in one response. When it fails, the team cannot tell whether the policy decision, extraction, authorization, or schema interpretation caused the problem. I prefer two contracts because the failure can be quarantined at the contract that owns it.

There is a real limitation here.

Node.js can coordinate either API path quickly, but the language does not make an unsafe CRM write safe. The same boundary still needs durable state, schema validation, timeouts, and an idempotency ledger. A chat completion that satisfies JSON Schema is still only a structured proposal.

How should a team migrate the contract without replaying history?

Treat a moderation or action-policy change as a versioned migration. Store the policy version beside the decision, never infer it from the worker deployment. A later worker can then tell whether an old decision is still valid, needs review, or must be re-evaluated under the new policy.

Do not reprocess every historical transcript by default. Start with a fixture set that contains ordinary calls, ambiguous requests, policy-sensitive text, empty transcripts, and transcripts with more than one possible CRM action. Run the candidate contract in shadow mode and compare decisions without allowing shadow output to write anything. The point is to find disagreement before it becomes a side effect.

The rollout gate should be operational rather than ceremonial. Require stable counts for undecided jobs, schema-validation failures, retryable dependency errors, and action-key collisions during the observation window. The exact thresholds belong to the service objective and the labeled policy set; they should not be copied from an example in an article.

Once the new contract is active, keep the previous decision available for audit. A rollback should change which policy version can create new actions, not erase the record of what was already decided. If a transcript revision changes, start a new evaluation. If only the worker retries, preserve the old key.

This migration shape also works for changing the output schema. Add a field as optional first, teach validators and storage to preserve it, then make it required after old workers have drained. A schema migration that assumes every queue message was produced by the newest worker is a replay bug waiting for a deployment window.

Where does reliability live in the worker?

The worker should persist a decision before acknowledging the queue message. A crash between those operations produces a replay, which is acceptable when the state transition and downstream write are idempotent. An empty response must be an explicit uncertain or failed state, never an implicit approval.

At first glance, a single chat completions call looks easier to operate. Later, the missing policy version and the unclassified retry are what make the incident expensive.

func handle(ctx context.Context, job Job, moderator Moderator, planner Planner, crm CRM) error {
    decision, err := moderator.Check(ctx, job.Transcript)
    if err != nil {
        if retryable(err) {
            return RetryAfter(30 * time.Second)
        }
        return FailJob(err)
    }

    if err := PersistDecision(job.ID, job.TranscriptRevision, decision); err != nil {
        return RetryAfter(30 * time.Second)
    }
    if !decision.Allowed {
        return PersistTerminal(job.ID, "discarded")
    }

    action, err := planner.Plan(ctx, job.Transcript, decision)
    if err != nil {
        return FailJob(err)
    }
    if err := ValidateAction(action); err != nil {
        return FailJob(err)
    }

    return crm.UpsertByKey(ctx, actionKey(action), action)
}

The timeout in this example is a policy placeholder. Set it from the queue freshness requirement and the dependency behavior, then exercise it with failure injection. A retry count without a deadline can keep a stale sales call alive indefinitely. A deadline without a terminal state can make the same job look successful to one component and pending to another. Pin a policy version such as 2026-10 in the persisted decision so a later comparison can separate a contract change from a retry.

The CRM adapter is the final authority for the side effect. If it cannot return the existing result for a repeated idempotency key, keep a local ledger and reconcile ambiguous writes before issuing a new action. That is more work, but a duplicate customer task is a business event, not a harmless log line.

Replay safely.

Which signal tells the on-call what to fix?

Instrument the four clocks separately: queue wait, model request duration, CRM wait, and total job age. One end-to-end histogram hides the dependency responsible for the delay. Record transcript revision, policy version, job state, and action key as structured attributes. Keep sensitive transcript content out of logs and use a trace identifier to connect the stages.

The first dashboard should answer three questions. How many jobs are undecided? How many decisions are allowed but have no planned action? How many action keys have been attempted without a confirmed write? Those counts map directly to operator actions. A raw request counter does not.

Use a warning for a growing age distribution while the oldest job remains inside the freshness objective. Page when the oldest undecided job crosses the business limit, when retryable errors stay elevated, or when replayed action keys rise alongside missing confirmations. A threshold set on every latency fluctuation creates false positives during normal marketplace bursts. A threshold set only after the queue is full arrives too late.

Review the false-positive cost before finalizing the alert. A noisy page trains the team to ignore it; a late page allows stale or duplicated CRM work. The right threshold is the one that starts a known runbook action, not the one that produces the most statistically impressive chart.

The reliability rule I would keep

Use moderation for policy classification and structured generation for structured interpretation. Version both contracts. Persist their results separately. Validate the proposed action before it can cross the CRM boundary, and make a replay use the same key. The limitation is deliberate: neither a moderation API nor a GPT-4o-mini chat completion should own authorization and the CRM write in the same step.

That arrangement keeps quality, latency, and operational correctness visible as separate measurements. It also makes a migration reversible without pretending that historical decisions never happened. The extra record and the extra boundary are small compared with explaining a duplicate customer follow-up after a queue replay.

DEV Community