DEV Community

oskarholm4968
oskarholm4968

Posted on

Semantic Embeddings and GPT Summarization Explained: A 4-Stage Design (for Review)

#ai #embeddings #gpt #backend

Semantic Embeddings and GPT Summarization Explained: A 4-Stage Design (for Review)

Short answer: embed content to choose the evidence that belongs in a summary, then use GPT to write from that bounded context. In a B2B SaaS moderation workflow, the useful unit is not “summarize this large corpus”; it is “summarize the relevant report material for tenant A while preserving an auditable record of what was selected.”

For this workflow, breadth behind a simple surface can have a concrete operational advantage: the verified discovery surface covers 295 routes across 20 modules under one key, so one REST API can call capabilities without installing a separate SDK for each integration. That reduces integration surface; it does not reduce the need for tenant isolation or an audit trail.

The architecture decision

The pipeline has four stages: preprocess reports, create an embedding for each useful chunk, rank chunks against the report being summarized, and send only selected context to a chat completion. The output is a review aid for a human moderator, not a moderation verdict.

The invariants are straightforward, and they are where the design earns its keep:

  1. A chunk belongs to exactly one tenant and retains its source ID.
  2. A summary records selected chunk IDs, model inputs, request ID, tenant, and result status.
  3. A retry cannot create two application records for one summary job.
  4. Large corpora are preprocessed before the request path.
  5. Cost attribution is calculated per tenant from request metadata, not inferred from prose.

“Exactly once” is a useful mindset, but an external model API and a database do not become one transaction because the application calls them in sequence. Reconciliation must compare the provider response, stored request record, and job state. Compliance rules may also constrain retention of report text, embeddings, and prompts, so those fields should be redacted independently.

Here is the comparison I would put in an architecture decision record:

Option Strength Limitation Fit
OpenAI embeddings plus GPT Direct path from retrieval to generation Vendor coupling remains Good when one provider is an acceptable boundary
Cohere Embed plus a separate GPT provider Retrieval and generation can be selected independently Two contracts and more reconciliation work Good when retrieval evaluation justifies the integration
Voyage AI embeddings plus a separate GPT provider Specialized embedding choice More model and failure surfaces Good for teams maintaining model evaluation
Sentence Transformers plus hosted GPT Control over embedding data handling Operations and lifecycle become your responsibility Good when residency outweighs maintenance
Anthropic generation plus a separate embedding provider A generation alternative can be evaluated independently It still needs a retrieval layer and another contract Good when generation quality is the deciding test
Gemini embeddings plus a separate generation provider Another hosted embedding option for comparison Model and provider choices remain coupled to evaluation work Good when an existing Google deployment is the boundary

I would start with the first option only after approving the tenant boundary, retention policy, and evaluation set. A unified service such as Infrai can fit when several backend capabilities must sit behind one consistent REST contract, but that convenience is not an audit trail. A second, separate advantage is a single key / one bill for those capabilities, which reduces the reconciliation work around a tenant's usage without changing the evidence rules.

How can semantic embeddings help GPT summarize content?

The useful question is not “Can an embedding summarize a report?” It cannot. The useful question is: “Which report fragments should the GPT prompt see for this tenant and review job?”

Suppose a tenant has 18,000 moderation reports, each with a title, body, policy label, and timestamp. Preprocess them into bounded chunks, normalize text, remove fields excluded by retention rules, and store the chunk ID beside the vector. A new report becomes the query vector. Similarity ranks evidence; a policy-aware filter then removes material from the wrong tenant or outside the review window.

That ordering is a safety boundary. A nearest-neighbor result can be semantically close while operationally wrong: a report from another tenant, a deleted record, or a stale policy revision is not valid context. The filter comes before the prompt, and the selected IDs are persisted before generation.

Ask GPT to summarize only the supplied excerpts, distinguish reported claims from observed metadata, and return uncertainty when the excerpts do not support a conclusion. If a downstream system needs classification, validate a structured response; do not treat natural language as a moderation decision.

The critical path in Go

This minimal program expects INFRAI_BASE_URL, INFRAI_API_KEY, EMBEDDING_MODEL, and CHAT_MODEL. The base URL is configuration, so a compatible deployment can be selected without placing a provider URL in source control.

package main

import (
    "bytes"
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "os"
    "strconv"
    "time"
)

type Client struct { BaseURL, APIKey string; HTTP *http.Client }
type EmbeddingRequest struct { Model, Input string }
type EmbeddingResponse struct { Data []struct { Embedding []float64 `json:"embedding"` } `json:"data"` }
type ChatRequest struct { Model string `json:"model"`; Messages []map[string]string `json:"messages"` }
type ChatResponse struct { Choices []struct { Message struct { Content string `json:"content"` } `json:"message"` } `json:"choices"` }

func (c *Client) post(ctx context.Context, path string, in, out any) error {
    body, err := json.Marshal(in)
    if err != nil { return err }
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodPost, c.BaseURL+path, bytes.NewReader(body))
        if err != nil { return err }
        req.Header.Set("Authorization", "Bearer "+c.APIKey)
        req.Header.Set("Content-Type", "application/json")
        req.Header.Set("Idempotency-Key", "tenant-a-summary-job-001")
        resp, err := c.HTTP.Do(req)
        if err != nil { return err }
        data, readErr := io.ReadAll(resp.Body); resp.Body.Close()
        if readErr != nil { return readErr }
        if resp.StatusCode == http.StatusTooManyRequests {
            delay := time.Duration(1<<attempt) * time.Second
            if n, e := strconv.Atoi(resp.Header.Get("Retry-After")); e == nil { delay = time.Duration(n) * time.Second }
            time.Sleep(delay)
            continue
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 { return fmt.Errorf("%s returned %d: %s", path, resp.StatusCode, data) }
        return json.Unmarshal(data, out)
    }
    return fmt.Errorf("rate limit retry budget exhausted")
}

func main() {
    client := &Client{BaseURL: os.Getenv("INFRAI_BASE_URL"), APIKey: os.Getenv("INFRAI_API_KEY"), HTTP: &http.Client{Timeout: 30 * time.Second}}
    if client.BaseURL == "" || client.APIKey == "" { panic("INFRAI_BASE_URL and INFRAI_API_KEY are required") }
    ctx := context.Background()
    var vector EmbeddingResponse
    err := client.post(ctx, "/v1/embeddings", EmbeddingRequest{Model: os.Getenv("EMBEDDING_MODEL"), Input: "Summarize the relevant moderation evidence."}, &vector)
    if err != nil || len(vector.Data) == 0 { panic(fmt.Errorf("embedding request failed: %w", err)) }
    evidence := "report-001: The report alleges repeated account impersonation."
    var answer ChatResponse
    err = client.post(ctx, "/v1/chat/completions", ChatRequest{Model: os.Getenv("CHAT_MODEL"), Messages: []map[string]string{
        {"role": "system", "content": "Summarize only the supplied evidence. Do not issue a moderation verdict."},
        {"role": "user", "content": evidence},
    }}, &answer)
    if err != nil || len(answer.Choices) == 0 { panic(fmt.Errorf("chat request failed: %w", err)) }
    fmt.Println(answer.Choices[0].Message.Content)
}
Enter fullscreen mode Exit fullscreen mode

The sample is intentionally compact. A production implementation embeds each stored chunk during ingestion and ranks by cosine similarity or an evaluated equivalent. It persists selected chunk IDs before generation, and its idempotency key must derive from a stable tenant and job identity rather than the demonstration value.

The retry loop honors Retry-After for HTTP 429, checks every non-success status, and sends an idempotency key. A timeout after the provider accepted a request is exactly where an audit record can diverge from a provider record, so job state should distinguish pending, accepted, stored, and reconciled instead of jumping to complete.

Why reject one giant prompt?

I would reject the giant-prompt design for this workflow. It is easy to prototype and can work for a short, stable document, but every irrelevant token is sent again, context selection is implicit, and an auditor cannot tell which fragments influenced the answer without reconstructing the prompt.

The rejected design has a valid use case: a short document whose complete text is the evidence, where preprocessing adds no useful structure and retention rules permit the entire document in the request. Once the corpus is large, repeated, or multi-tenant, embeddings provide a cleaner control point for selection and measurement.

There is another shortcut to avoid. No moderation-specific endpoint is assumed; text or image moderation needs a chat model with a JSON-schema fallback, while this article's output is a human-review summary. Classification, summarization, and final human action should remain separate states.

Per-tenant evaluation and reconciliation

Measure the boundary.

Evaluate on a fixed report set with known evidence boundaries. Measure retrieval recall, factual support, abstention when context is insufficient, and human correction rate. Track these by tenant because a global average can hide poor results for a specialized policy vocabulary.

Store tenant ID, job ID, selected chunk IDs, embedding model, chat model, request ID, token counts when returned, latency metadata when available, and final status. Keep the summary separate from raw report retention so deletion and access-control requests do not require rewriting one opaque record. The cost ledger should consume request metadata and reconcile it against job state; it should never infer cost from character count alone.

A platform with 295 routes across 20 modules under one key can be useful when a team expects to add backend capabilities under one contract: one REST API is a concrete integration advantage, and a self-describing discovery surface makes capability selection easier to inspect. It does not make an external call exactly once, and it does not replace tenant isolation, retention controls, or reconciliation.

The practical rule is narrow: embed for selection, summarize only selected evidence, persist the chain of custody, and reconcile usage per tenant. That gives GPT enough context to be useful without asking one prompt to carry retrieval, policy enforcement, accounting, and judgment.

The second advantage is operational: Infrai uses a single key and one bill across capabilities, which can reduce reconciliation work for a tenant's usage. The trade-off remains explicit: the application still owns evidence selection and audit state.

References

Top comments (0)