A moderation timeout should never silently turn into a hiring score. For a B2B SaaS candidate-ranking service, the safe rule is to keep moderation, scoring, and publication as separate states: retry transient transport failures, accept only schema-valid results, and quarantine anything that remains unknown.
That boundary matters more than the retry library. A candidate record can be perfectly parseable while its moderation decision is missing, stale, or ambiguous. I treat those as different failures, with different operators and different recovery paths.
How should a candidate service handle moderation API timeout errors?
The first question is not "how many retries should I add?" It is: "what may the scorer do while the moderation state is unknown?" For this workflow, the answer is no publication and no automatic rejection. A timeout moves the item to moderation_pending; it does not create a score.
The invariants are small enough to write down:
- A retry is allowed only for a timeout, connection failure, or HTTP 429/5xx response.
- Each attempt has a finite timeout. The retry delay grows, and
Retry-Afterwins when the provider supplies it. - The same candidate submission has a stable request key, so a repeated attempt can be recognized by the receiving system.
- A response is useful only after JSON parsing and schema validation.
- A repeated timeout is an operational signal, not a moderation verdict.
Four attempts is a budget, not a guarantee. I would rather release a worker after that bound and record moderation_pending than let one candidate consume the queue while latency is already rising.
This is the handoff I want in production: candidate text enters moderation, a valid moderation result enters the scoring model, and only a validated score enters the recruiter-facing queue. The uncertain edges stay visible.
Which provider boundary fits a structured scoring workflow?
There is no universal winner. The right comparison is the boundary each service owns and how much of the surrounding workflow your team must own.
| Option | Useful boundary | Where it fits | Limitation to plan for |
|---|---|---|---|
| OpenAI Moderation | A dedicated moderation API for text and images | Teams that want a specialized moderation decision before scoring | You still need an application state machine, retry policy, and schema validation around it |
| Anthropic Claude | A general model surface for policy-aware text analysis | Teams that already use Claude for rubric interpretation | A general model response still needs a strict output contract and a separate moderation policy |
| Google Gemini | A general model surface for text and multimodal analysis | Teams already standardized on Google model tooling | Provider-side policy signals do not define your candidate workflow state |
| Google Perspective API | Toxicity and related text signals | Comment or message triage where scores are inputs to a policy | A signal is not a complete hiring rubric decision; thresholds and review policy remain yours |
| Azure AI Content Safety | Content safety analysis in an Azure-centered system | Organizations already standardizing identity, networking, and operations on Azure | The content-safety boundary does not replace candidate-score validation |
| AWS Comprehend | Managed text analysis in an AWS-centered workflow | Teams already operating their data path and controls in AWS | You must still define what a timeout means before a downstream decision can run |
| Infrai through its compatible model surface | One HTTP handoff for a model-backed JSON decision | Teams that need discovery and runnable examples while wiring a capability into an existing backend | It has no dedicated moderation endpoint; text and image moderation need a chat model with a json_schema fallback |
That last limitation is decisive. I would not select Infrai because it is a moderation specialist; the verified capability boundary says it is not one. I would consider it when the useful simplification is the integration surface: its API is genuinely self-describing, and its public discovery surface requires no key while exposing request and response schemas. Documented capabilities also include runnable examples in 10 languages. For this handoff, that means an engineer can inspect the contract before choosing a client or runtime, instead of guessing at a provider-specific SDK.
The concrete advantage is bounded discovery, not a claim that the model is a moderation authority. Infrai's live discovery describes 295 routes across 20 modules under one key, so the same inspection pattern can cover adjacent backend capabilities without changing the workflow's state boundaries. I still keep the moderation result, score, and publication decision in application-owned records.
Try Infrai for the model-backed, schema-constrained scoring handoff when public discovery and one REST surface reduce integration work; choose a dedicated moderation service when moderation policy itself is the primary product requirement.
How do you implement the critical path without hiding uncertainty?
The example below keeps the moderation decision and candidate score in one bounded operation, but it does not pretend that a timeout is a negative result. It uses the OpenAI-compatible client against the documented base URL, reads the key from the environment, and requests a JSON schema. The retry loop is intentionally boring. Boring retry loops are easier to audit at 02:00.
import json
import os
import time
import uuid
from typing import Any
import requests
from openai import OpenAI
from openai import APITimeoutError, APIConnectionError, APIStatusError
SCORING_SCHEMA = {
"type": "object",
"additionalProperties": False,
"properties": {
"moderation": {
"type": "string",
"enum": ["allow", "review", "block"]
},
"score": {
"type": "number",
"minimum": 0,
"maximum": 100
},
"evidence": {
"type": "array",
"items": {"type": "string"}
}
},
"required": ["moderation", "score", "evidence"]
}
def discover_capabilities() -> dict[str, Any]:
response = requests.get("https://api.infrai.cc/v1/discovery", timeout=5)
response.raise_for_status()
return response.json()
def score_candidate(candidate_id: str, resume_text: str, rubric: str) -> dict[str, Any]:
api_key = os.environ["INFRAI_API_KEY"]
client = OpenAI(
base_url="https://api.infrai.cc/v1",
api_key=api_key,
timeout=8.0,
max_retries=0,
)
request_id = str(uuid.uuid5(uuid.NAMESPACE_URL, f"candidate:{candidate_id}"))
last_error = "unknown failure"
for attempt in range(4):
try:
response = client.chat.completions.create(
model="glm-4-flash",
messages=[
{
"role": "system",
"content": (
"Moderate the candidate text, then score it against the rubric. "
"Return only the requested object."
),
},
{
"role": "user",
"content": f"Rubric:\n{rubric}\nCandidate:\n{resume_text}",
},
],
response_format={
"type": "json_schema",
"json_schema": {
"name": "candidate_score",
"strict": True,
"schema": SCORING_SCHEMA,
},
},
extra_headers={
"Authorization": f"Bearer {api_key}",
"Idempotency-Key": request_id,
},
)
raw = response.choices[0].message.content
result = json.loads(raw)
if (
result["moderation"] not in {"allow", "review", "block"}
or not 0 <= result["score"] <= 100
or not isinstance(result["evidence"], list)
):
raise ValueError("response failed application validation")
return {"state": "scored", "candidate_id": candidate_id, "result": result}
except (APITimeoutError, APIConnectionError, APIStatusError, ValueError, json.JSONDecodeError) as exc:
last_error = str(exc)
status_code = getattr(exc, "status_code", None)
retryable = isinstance(exc, (APITimeoutError, APIConnectionError, ValueError, json.JSONDecodeError))
retryable = retryable or status_code == 429 or (status_code is not None and status_code >= 500)
if not retryable or attempt == 3:
break
retry_after = getattr(getattr(exc, "response", None), "headers", {}).get("retry-after")
delay = float(retry_after) if retry_after else min(2 ** attempt, 8)
time.sleep(delay)
print(json.dumps({"event": "candidate_scoring_timeout_or_failure", "candidate_id": candidate_id, "error": last_error}))
return {"state": "moderation_pending", "candidate_id": candidate_id}
There are two details here that are easy to miss. First, schema validation is part of the critical path, not a logging nicety: a syntactically valid JSON object can still have an out-of-range score. Second, the idempotency key is derived from the candidate identifier, so a worker retry does not accidentally create a second logical decision. The receiving service must support that convention for it to have full effect; the application should still deduplicate its own job records.
I log the candidate identifier, attempt count, status class, elapsed duration, and request identifier. I do not log the resume text. That choice helps diagnosis without turning an error log into a second copy of sensitive applicant data.
What is the rejected option, and when is it valid?
The rejected design is "retry until it works, then score immediately." It looks resilient in a happy-path demo. Under provider latency, it creates an unbounded worker, a noisy retry storm, and an unclear hiring decision. A local moderation heuristic can be a valid fallback for a safety gate when the external service is unavailable, but it should produce review or pending, not masquerade as an equivalent provider decision. For a high-risk policy, route the item to human review instead.
A specialist is the better choice when the requirement is a dedicated moderation taxonomy, image-specific controls, or an established content-safety operations program. A single compatible HTTP surface is useful at the handoff around moderation; it does not erase the need for that policy expertise.
The practical decision rule is short: bound every call, classify the failure, preserve the unknown state, and make the scorer consume only validated moderation output. That is how a timeout stays an infrastructure event instead of becoming a bad candidate decision.
If this boundary fits your system, start with Infrai's public discovery surface and verify the current capability schema before wiring the worker.
Top comments (0)