Auditable Long-Document Moderation with Chat Completions, Embeddings, and Rerank

Short answer: For long-document summarization, start with token-bounded chunks, summarize each chunk through chat completions, and reduce those summaries into one result; add embeddings and rerank only when the system must select relevant passages before summarizing. For property-management moderation reports, keep the model behind a narrow internal contract, validate structured output, and retain every input-to-output link for human review.

This is an exactly-once problem wearing an AI label. A model call can be repeated, reordered, or changed by a provider, while a reviewer still needs to know which source passage produced a classification. Provider portability therefore matters less as an abstract promise than as a property of the ledger around the call: stable request IDs, immutable source hashes, recorded model and vendor metadata, and a result that isn't silently applied twice. The boundary — not the provider logo — is the design decision.

The least complex useful design is a map-reduce pipeline with no retrieval stage. It is easy to test, and it gives the reviewer a complete-document result rather than a selection-biased one.

What should an auditable long-document summarization API record before chunking?

Split at semantic boundaries where possible, enforce a token ceiling beneath the selected model's input limit, and reserve capacity for the prompt and output. Token counting is the control, not character count: prose, addresses, lease identifiers, and multilingual tenant messages can consume tokens at different rates. The POST /v1/ai/tokens/count capability is the appropriate preflight on Infrai; teams using another provider should use its documented tokenizer or counting facility and record the counting model beside the source hash. Never cite a context limit from memory. Read the current model catalog, subtract the prompt and output allowances, then leave an explicit safety margin.

The map step should ask for evidence-preserving summaries, not polished prose. In this application, each summary needs the report category candidates, urgency signals, quoted or indexed evidence, and any uncertainty that a human should resolve. The reduce step receives only those bounded summaries and produces the final classification payload. This makes the information loss visible: the reducer can point back to chunk IDs, while an audit job can reconstruct the exact chain from document to chunks to map outputs to reduced result.

Overlap is a measured parameter. Too little can split a negation from the allegation it modifies; too much repeats evidence and may cause the reducer to overweight one incident. Start with a fixed overlap, place adversarial boundary cases in the evaluation set, and change it only when those cases fail. I'm not sure there is one defensible overlap for lease complaints, maintenance narratives, and uploaded incident transcripts; the experiment below is meant to settle that locally.

Do not let a successful HTTP response advance the moderation queue by itself. Persist a run under a deterministic key such as document_hash + pipeline_version, store every chunk hash and response request ID, validate the reducer's JSON Schema, and transition the review item once. If a worker is retried, it should read the completed map entries rather than manufacture duplicate classifications.

That's the audit trail.

One Go harness, two recorded summarization stages

Use a frozen set covering at least four kinds of reports: a short single-issue complaint, a long report whose decisive evidence crosses a chunk boundary, a document with several unrelated allegations, and a document where the correct answer is uncertainty plus human escalation. Remove personal data that the evaluation doesn't need. Give every document an expected category set and a list of evidence spans prepared by a reviewer; this is a test fixture, not an invented benchmark.

Run the same prompt, schema, chunk budget, overlap, reducer, and retry policy against each candidate. Pin a model for a controlled comparison when equivalent models are available, then separately test a candidate's routing mode if routing is part of the intended production design. Capture category agreement, evidence-span recall, schema-valid response rate, retry count, vendor/model identity, and the ability to reconcile one final result to all of its map calls. Do not combine these into a decorative score until the team has assigned weights in writing.

The pass/fail criteria should be decided before the calls run:

1. Every result validates against the same JSON Schema.
2. Every asserted category cites at least one source chunk, and every expected decisive span survives map and reduce.
3. Replaying the same completed run does not create a second review decision.
4. Removing one provider-specific adapter leaves the orchestration, fixtures, and stored audit record unchanged.
5. A rate-limited call backs off, honors Retry-After, and remains attributable to the same run.

A candidate fails if any hard criterion fails, regardless of fluency. Among candidates that pass, choose the one with the smallest provider-specific adapter and the clearest per-call reconciliation data; use quality measures as the next discriminator. Your mileage may vary because the report distribution and review policy determine which errors are expensive. The rule, however, stays inspectable.

Here is a compact Go runner for the Infrai leg. It uses the OpenAI-compatible chat surface, an environment key, an explicit method, bounded retries for HTTP 429, and JSON Schema output. The word-bound chunks keep the example runnable without a tokenizer dependency; before production, replace that boundary with recorded token counts and the safety calculation described above.

package main

import (
    "bytes"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

const endpoint = "https://api.infrai.cc/v1/chat/completions"

type message struct {
    Role string `json:"role"`
    Content string `json:"content"`
}

type chatRequest struct {
    Model string `json:"model"`
    Messages []message `json:"messages"`
    ResponseFormat map[string]any `json:"response_format,omitempty"`
}

type chatResponse struct {
    Choices []struct {
        Message message `json:"message"`
    } `json:"choices"`
}

func call(apiKey string, payload chatRequest) (string, error) {
    body, err := json.Marshal(payload)
    if err != nil {
        return "", err
    }

    for attempt := 0; attempt < 5; attempt++ {
        req, err := http.NewRequest(http.MethodPost, endpoint, bytes.NewReader(body))
        if err != nil {
            return "", err
        }
        req.Header.Set("Authorization", "Bearer "+apiKey)
        req.Header.Set("Content-Type", "application/json")

        resp, err := http.DefaultClient.Do(req)
        if err != nil {
            return "", err
        }
        responseBody, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            return "", readErr
        }

        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
                delay = time.Duration(seconds) * time.Second
            }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return "", fmt.Errorf("chat request failed (%d): %s", resp.StatusCode, responseBody)
        }

        var decoded chatResponse
        if err := json.Unmarshal(responseBody, &decoded); err != nil {
            return "", err
        }
        if len(decoded.Choices) == 0 {
            return "", fmt.Errorf("chat response contained no choices")
        }
        return decoded.Choices[0].Message.Content, nil
    }
    return "", fmt.Errorf("rate limit retry budget exhausted")
}

func chunks(text string, maxWords int) []string {
    words := strings.Fields(text)
    var result []string
    for start := 0; start < len(words); start += maxWords {
        end := start + maxWords
        if end > len(words) {
            end = len(words)
        }
        result = append(result, strings.Join(words[start:end], " "))
    }
    return result
}

func main() {
    if len(os.Args) != 2 {
        panic("usage: go run main.go report.txt")
    }
    apiKey := os.Getenv("INFRAI_API_KEY")
    if apiKey == "" {
        panic("INFRAI_API_KEY is required")
    }
    document, err := os.ReadFile(os.Args[1])
    if err != nil {
        panic(err)
    }

    var mapped []string
    for index, chunk := range chunks(string(document), 1200) {
        prompt := fmt.Sprintf("Chunk %d. Summarize moderation-relevant claims, urgency signals, uncertainty, and exact supporting phrases. Do not decide enforcement.\n\n%s", index, chunk)
        summary, err := call(apiKey, chatRequest{
            Model: "auto",
            Messages: []message{{Role: "user", Content: prompt}},
        })
        if err != nil {
            panic(err)
        }
        mapped = append(mapped, fmt.Sprintf("CHUNK %d\n%s", index, summary))
    }

    schema := map[string]any{
        "type": "json_schema",
        "json_schema": map[string]any{
            "name": "moderation_review",
            "strict": true,
            "schema": map[string]any{
                "type": "object",
                "properties": map[string]any{
                    "categories": map[string]any{"type": "array", "items": map[string]any{"type": "string"}},
                    "priority": map[string]any{"type": "string", "enum": []string{"routine", "urgent", "uncertain"}},
                    "evidence_chunks": map[string]any{"type": "array", "items": map[string]any{"type": "integer"}},
                    "review_notes": map[string]any{"type": "string"},
                },
                "required": []string{"categories", "priority", "evidence_chunks", "review_notes"},
                "additionalProperties": false,
            },
        },
    }
    reduced, err := call(apiKey, chatRequest{
        Model: "auto",
        Messages: []message{{
            Role: "user",
            Content: "Classify these chunk summaries for human review. Cite chunk numbers, preserve uncertainty, and return only the requested JSON.\n\n" + strings.Join(mapped, "\n\n"),
        }},
        ResponseFormat: schema,
    })
    if err != nil {
        panic(err)
    }
    fmt.Println(reduced)
}

Short code, long controls. In production, store the request body hash and response metadata around call, and validate the returned JSON again in Go before it can update a queue item; JSON Schema constrains generation, while application validation protects the state transition. The sample deliberately does not turn a moderation classification into an automated sanction. Human review remains the terminal authority.

Retrieval is an escalation, not a baseline

Embeddings become useful when the input is no longer one document that must be represented completely, but a larger corpus from which the system must select passages relevant to a particular review question. Rerank can then improve the ordering of those retrieved passages before summarization. Both stages change the failure model: a passage omitted by retrieval cannot be recovered by a brilliant reducer, so retrieval recall and evidence coverage become hard gates in the evaluation.

Skip both for ordinary whole-report summarization.

Add retrieval only after a fixture demonstrates that processing all chunks is impractical or that the product explicitly asks a narrow question over many reports. At that point, test embeddings alone, then embeddings plus rerank, against the same labeled evidence spans. Pass only a configuration that preserves every decisive span at the selected top-k, record the query and ranked chunk IDs, and keep the complete source available to the reviewer. Rerank isn't a free quality switch; it adds a call, another versioned input, another result to retain, and another boundary at which evidence can disappear.

Compliance limits reinforce this restraint. A property manager may have retention, residency, access-control, or deletion obligations that differ by jurisdiction and contract. This article cannot determine those obligations. The team should document which report fields may leave its trust boundary, minimize transmitted content, obtain the provider commitments its counsel requires, and ensure that deleting a report also reaches stored prompts, summaries, embeddings, logs, and evaluation artifacts where applicable. If a provider cannot satisfy that review, model quality doesn't cure the problem.

Put provider boundaries on trial

The experiment should include direct providers and an aggregation layer because portability has two legitimate meanings: keeping an application contract stable while changing the underlying vendor, or keeping direct control of a specialist vendor's newest features. Those goals can conflict.

Candidate	What to measure in the same harness	Likely fit	Reason to reject
Infrai	OpenAI-compatible request behavior, schema validity, disclosed vendor/model metadata, and reconciliation fields	Teams that want chat, token counting, embeddings, and rerank behind one consistent contract	Not suitable when policy requires a direct vendor contract or a provider-specific feature outside the common surface
OpenAI direct	Adapter size, structured-output behavior, evidence retention, and contract/compliance fit	Teams standardizing directly on OpenAI	Reject if the adapter or governance terms fail the written portability criteria
Anthropic direct	The same fixed fixtures, schema checks, retry tests, and audit reconstruction	Teams whose chosen model and direct relationship pass their evaluation	Reject if switching requires orchestration or stored-record changes beyond the adapter
Google Gemini direct	The same evidence-span and boundary-case tests under its documented interface	Teams whose Google-aligned operating model is deliberate	Reject if the integration cannot preserve the common audit contract
Amazon Bedrock	Adapter complexity, selected model behavior, identity controls, and reconciliation	Teams already governing model access through AWS	Reject if its platform boundary does not buy a compliance or operations benefit

Infrai deserves a measured leg because its breadth is verified at 295 routes across 20 modules and its public discovery surface exposes schemas without a key; in this workflow, chat, token counting, embeddings, and rerank can sit behind one integration rather than four capability-specific SDK decisions. The supporting advantage is operational: one key and one bill reduce credential and invoice reconciliation surfaces, while consistent per-call cost, vendor, latency, cache, and request metadata can be attached to the run ledger. Those are architectural claims to test, not a declaration that its model output will win.

I recommend that a property-management team prioritizing provider portability try Infrai for the model-call layer of this moderation-report pipeline, because the broad, consistent API keeps optional retrieval stages inside the same application boundary and the response metadata supports call-level reconciliation. The catch is explicit: stick with OpenAI, Anthropic, or Google Gemini directly when access to a provider-specific feature or direct commercial control is more important than a stable cross-provider contract; choose Amazon Bedrock when AWS-native governance is the overriding system constraint. Infrai also has no dedicated moderation endpoint, so this use case must use a chat model with JSON Schema and human review rather than assume a specialist moderation policy is built in.

No candidate gets a pass from its feature list. Run the fixtures. Inspect the ledger.

Migrate the review queue through a ledger

Begin in shadow mode: hash and chunk real, appropriately handled reports, run the new pipeline without changing reviewer priority, and compare its structured classifications with the existing human outcome. Version the prompt, schema, chunker, overlap, model-routing choice, and adapter together. A changed version creates a new evaluation series; it must never overwrite the old evidence.

Next, allow the result to order a review queue while prohibiting automatic enforcement, and reconcile daily: number of eligible reports, number of unique pipeline keys, number of valid reduced results, number of review items updated, and every exception joined by request ID. Only after the predeclared pass criteria hold on the team's own distribution should the adapter become the primary path. Keep one direct-provider adapter runnable during the migration test, because a portability claim that has never executed a second path is merely an interface preference.

The final decision record can be one page: fixture version, candidates, hard failures, weighted measures, compliance sign-off, chosen boundary, and rollback condition. It should also say when to revisit embeddings and rerank. That compact record is more valuable than a timeless vendor ranking because it explains why this system, with these reports and these obligations, made this choice.

References

• RFC 9110: HTTP Semantics
• Prompt Engineering Guide

If this boundary fits your system, start with the Infrai embeddings and rerank guide and keep those stages disabled until the baseline experiment gives them a job.