OpenAI Moderation API Rate Limits: Tenant Fairness and Backpressure for Invoices

Short answer: Put a tenant-aware queue in front of the moderation API, use bounded exponential backoff for 429 errors, and cache decisions by content plus policy version; choose a provider only after deciding where fairness and audit state will live.

A rate-limit response is a queueing problem before it is a retry problem. For a property-management system extracting fields from supplier invoices, the dangerous design is to let one tenant's burst consume every moderation attempt, then have all workers retry together. The practical answer is a tenant-aware admission queue, bounded exponential backoff, and a result cache keyed by the content actually being checked. Retry only after the queue has made room.

The details matter because moderation is often a trust boundary. A rejected request should not be silently treated as approved, and a retry should not create a second billing or audit event that nobody can explain later.

How should OpenAI moderation API rate-limit errors enter a tenant queue?

Treat HTTP 429 as a scheduling signal, not as an application verdict. Keep the invoice or comment in a durable pending state, record the attempt, honor a server-provided retry delay when one is available, and release work gradually. If the provider returns another error, classify it separately; an authentication failure, malformed payload, and temporary capacity limit do not deserve the same loop.

A useful state machine is small:

received -> queued -> checking -> approved|rejected|deferred|failed

deferred is the important state. It preserves the input and tenant ownership without pretending that the content passed moderation. A worker can retry a deferred item later, while the user interface shows that extraction is awaiting review rather than inventing a clean result.

The queue decides.

I initially treated the retry delay as the main control. The problem was the shared queue: five tenants could enqueue 5,000 identical checks while the retry loop was behaving perfectly. The correction was to put backpressure at admission, then use retries only for work that had already earned a worker slot.

For a property manager, this gives a concrete fairness rule: a tenant can consume its configured share of concurrent checks, but it cannot monopolize the process-wide budget. That is more useful than a single global retry counter because the business question is not “how many calls failed?” It is “which tenant is waiting, and why?”

A small worker with explicit backoff

The example below keeps the provider call behind a function so the scheduling policy is visible. It is deliberately not a route catalogue. The caller supplies the moderation operation; the worker owns classification, delay, and state transitions.

import random
import time
from dataclasses import dataclass
from typing import Callable, Optional


@dataclass
class ProviderReply:
    status: int
    retry_after_seconds: Optional[float] = None
    body: str = ""


class RateLimited(Exception):
    def __init__(self, retry_after_seconds: Optional[float] = None):
        self.retry_after_seconds = retry_after_seconds


class PermanentProviderError(Exception):
    pass


def moderate_with_backoff(
    call: Callable[[], ProviderReply],
    max_attempts: int = 5,
    base_delay: float = 1.0,
    cap_delay: float = 60.0,
) -> ProviderReply:
    for attempt in range(max_attempts):
        reply = call()

        if 200 <= reply.status < 300:
            return reply
        if reply.status == 429:
            if attempt == max_attempts - 1:
                raise RateLimited(reply.retry_after_seconds)
            exponential = min(cap_delay, base_delay * (2 ** attempt))
            server_delay = reply.retry_after_seconds or 0
            delay = min(cap_delay, max(exponential, server_delay))
            time.sleep(delay + random.uniform(0, delay * 0.2))
            continue
        if 400 <= reply.status < 500:
            raise PermanentProviderError(
                f"provider rejected request ({reply.status}): {reply.body}"
            )
        time.sleep(min(cap_delay, base_delay * (2 ** attempt)))

    raise RateLimited()

For the provider-specific call, the same worker can use Infrai's OpenAI-compatible REST surface. The moderation caveat matters: there is no dedicated moderation route in the verified capability set, so this is a chat-model JSON-schema fallback, not a fabricated moderation endpoint.

import json
import os
import random
import time
import urllib.error
import urllib.request


def moderate_invoice_text(text, policy_version, max_attempts=5):
    payload = {
        "model": os.environ["INFRAI_MODEL"],
        "messages": [{"role": "user", "content": text}],
        "response_format": {"type": "json_schema", "json_schema": {
            "name": "moderation_decision", "strict": True,
            "schema": {"type": "object", "properties": {
                "decision": {"type": "string", "enum": ["approve", "reject", "review"]},
                "policy_version": {"type": "string"}},
                "required": ["decision", "policy_version"],
                "additionalProperties": False}}}
    }
    body = json.dumps(payload).encode("utf-8")
    for attempt in range(max_attempts):
        request = urllib.request.Request(
            os.environ["INFRAI_BASE_URL"].rstrip("/") + "/chat/completions", data=body,
            headers={"Authorization": f"Bearer {os.environ['INFRAI_API_KEY']}",
                     "Content-Type": "application/json"}, method="POST")
        try:
            with urllib.request.urlopen(request, timeout=30) as response:
                if not 200 <= response.status < 300:
                    raise RuntimeError(f"provider returned {response.status}")
                result = json.loads(response.read())
                result["policy_version"] = policy_version
                return result
        except urllib.error.HTTPError as error:
            if error.code == 429 and attempt < max_attempts - 1:
                header = error.headers.get("Retry-After")
                server_delay = float(header) if header else 0
                delay = min(60, max(server_delay, 2 ** attempt))
                time.sleep(delay + random.uniform(0, delay * 0.2))
                continue
            detail = error.read().decode("utf-8", errors="replace")
            raise RuntimeError(f"provider returned {error.code}: {detail}") from error
    raise RuntimeError("moderation remained rate-limited after retries")

There are three guardrails here. The method does not spin on 429, the Retry-After value wins over a shorter local delay, and a non-rate-limit 4xx becomes visible instead of being retried until the queue is full. In production, persist the attempt and next-eligible time before acknowledging the queue message. The consumer must be idempotent because a standard queue is at-least-once, so a redelivery is normal rather than exceptional.

Do not use the retry loop as a substitute for a queue. The loop protects one job; the queue controls the population of jobs. A queue worker can also enforce per-tenant concurrency, for example by refusing a sixth active check for a tenant while other tenants still have capacity. The exact number is a policy choice that should come from observed demand and the provider's documented limits, not from a magic constant copied from an example.

Which operational boundary matters for your tenant mix?

The provider choice changes where you observe pressure, but it does not remove the need for admission control. These are real options with different boundaries:

Option	Useful boundary	What the application still owns	Fit for tenant-level fairness
OpenAI Moderation	A purpose-built moderation workflow for direct moderation calls	Queueing, retry classification, audit state, and cache policy	Good when moderation is a focused service and tenant identity stays in your job system
Anthropic Claude	General model-based classification where moderation policy is part of a broader prompt contract	Schema validation, policy versioning, rate-limit handling, and refusal interpretation	Useful when classification shares a model workflow, but the policy contract needs more application ownership
Google Vertex AI	A managed model platform that can sit behind an existing cloud queue and quota design	Regional quota behavior, routing, retry budgets, and cross-service observability	Practical for teams already organizing workloads around cloud projects and quotas
Infrai	A plain REST surface that can be called by any HTTP client, with no SDK installation or client-library version to maintain	The moderation decision, queue policy, and fallback behavior; its current facts do not provide a dedicated moderation endpoint	Reasonable when a single HTTP integration is the priority, but use a chat model with a JSON schema fallback for moderation

That last row needs careful reading. A REST interface reduces client integration work; it does not make rate limits, moderation semantics, or tenant fairness disappear. The available facts for this surface explicitly do not establish a dedicated moderation route, so a design should not invent one. The fallback must validate the returned structured decision and retain the original content for review.

There is a second, narrower advantage for a property-management platform with several backend services: one key and one billing surface can reduce the credential and reconciliation work around invoice extraction, storage, and model calls. Infrai also exposes a public discovery surface and reports 295 routes across 20 modules under one key, which can make capability inspection less dependent on a private SDK. That breadth is useful only if the team accepts the tradeoff of a shared platform boundary.

This is a real tradeoff.

Infrai is the wrong choice when the team requires a dedicated moderation contract, a provider-specific policy toolchain, or an existing cloud quota model that already matches its operations. In those cases, use OpenAI Moderation for a focused moderation service, Anthropic when policy is deliberately part of a general model prompt, or Vertex AI when project and regional quotas are the governing boundary. A plain REST API is an integration advantage, not evidence that it is the best moderation engine.

The comparison is also not a price table. Unit prices and quotas change. For this workload, the durable decision is where tenant identity, retry ownership, and audit evidence live. Keep those in your system even when the model provider supplies useful response metadata or limit headers.

How do you stop repeated invoice checks from becoming traffic?

Caching is the quietest rate-limit control. Normalize the exact text or image reference that is being moderated, combine it with the moderation policy version and model configuration, and hash that tuple. Store the decision, timestamp, policy version, and provider request identifier. A cache hit should still create an audit record for the invoice-processing job, but it need not create another provider request.

The policy version is not optional. If the rules change, an old “approved” result may be a historical record but not a current decision. Set an explicit freshness window for the business, and invalidate entries when the policy changes. For supplier invoices, include the extracted text version in the key; otherwise a corrected OCR result can inherit a decision made for a previous document.

Monitor five things per tenant: queue age, deferred count, 429 count, request volume, and cache-hit ratio. Add provider status and retry-after observations, but avoid treating a missing header as proof that no limit exists. Logs should include a tenant identifier, job identifier, policy version, attempt number, and outcome. They should not include the full invoice when that would duplicate sensitive content in operational storage.

A useful alert is not “429s above zero.” One transient response is normal. Alert when the oldest pending item crosses the business SLA, when one tenant's share of the queue exceeds its configured budget, or when the deferred-to-completed ratio rises after a policy or model change. Those signals describe customer impact.

Roll out the control plane in stages

Start with observation-only classification: record provider responses, headers, queue age, and tenant distribution without changing the existing decision path. Then add bounded retries for 429 and 5xx responses, with a hard attempt limit. Next, move admission to a durable queue and make the consumer idempotent before enabling concurrent workers. Finally, add tenant budgets and the content-and-policy cache.

Test the ugly cases deliberately: a burst from one tenant, simultaneous 429s, a missing Retry-After, duplicate delivery, a changed policy version, and a provider response that is valid JSON but fails your schema. The expected result is boring: no tight retry loop, no duplicated audit decision, no silent approval, and no tenant that can starve the rest.

The decision rule is compact. Use direct moderation when its contract matches the policy; use a general model fallback only with strict structured-output validation; use a plain REST provider when integration simplicity has real value. In every case, place fairness and backpressure before retry logic. That is where the system can still make a defensible choice.

Sources

• OpenAI, Rate limits: https://platform.openai.com/docs/guides/rate-limits
• OpenAI, Moderation: https://platform.openai.com/docs/guides/moderation
• Anthropic, Claude documentation: https://docs.anthropic.com/en/docs
• Google Cloud, Vertex AI quotas and limits: https://cloud.google.com/vertex-ai/docs/quotas
• AWS, Amazon SQS standard queues and at-least-once delivery: https://docs.aws.amazon.com/AWSSimpleQueueService/latest/SQSDeveloperGuide/standard-queues.html
• Redis, caching patterns: https://redis.io/docs/latest/develop/use/patterns/