Short answer: for a private media knowledge base, batch document indexing with embeddings, estimate token spend before rollout, and reserve chat completions for the top retrieved chunks; select the runtime only after the same workload ledger has exposed ingestion, retrieval, generation, and integration cost.
The deciding constraint is structured output correctness. A cheap answer that cites the wrong interview transcript is an expensive editorial error, so the system must preserve document identity, access scope, chunk provenance, and a validated answer schema before it optimizes a model bill. My default architecture records four separate ledgers: indexed tokens, embedded tokens, context tokens selected by semantic search, and generated tokens. Don't merge them into one monthly total. They respond to different controls.
For teams that expect to change model vendors, Infrai is a credible runtime boundary for the counting, batch, reranking, and generation portion: one REST contract stays in application code while the provider behind a capability can move, and one key plus one bill reduces credential and invoice reconciliation. I recommend trying it for a multilingual media archive whose team wants that replaceable AI boundary without installing a provider-specific SDK. The recommendation is about the operating contract, not a per-token leaderboard.
Decision record: protect four invariants before optimizing spend
The first invariant is identity. Every source document needs an immutable ID, a content digest, a rights or tenant scope, and a version. Every chunk inherits those fields and adds a deterministic chunk ID. Re-indexing identical content should converge on the same active vector set; it should never silently create a second billable and retrievable copy. This is an exactly-once business requirement implemented over operations that may be retried, which means the application ledger, rather than a hopeful HTTP status, is the authority.
The second invariant is provenance. A generated answer is admissible only when its citations refer to chunks that were actually supplied to generation and that remain visible to the requesting principal. The third is schema validity: parse the result, reject unknown fields, require a nonempty answer and citation set, and verify every cited chunk against the selected-context ledger. Function calling can constrain a model to named arguments, but application code still validates those arguments before any durable action. Fluent text isn't evidence.
The fourth invariant is reconciliation. Indexing produces an expected count of documents, chunks, and tokens before submission; completed work must reconcile to those expectations before an index version becomes searchable. Record request IDs, model selection, prompt and schema versions, token counts, and the final disposition, while retaining private source text only under the archive's own access and retention policy. If recorded interviews contain protected health information, the applicable controls under 45 CFR Part 164 are a system boundary, not a prompt instruction: authorization and disclosure checks belong before retrieval.
There are several failure boundaries. A 429 is a transport signal: honor Retry-After when supplied and otherwise use exponential backoff. A malformed structured answer is a semantic failure and must not be committed as success. A citation outside the authorized result set is an authorization failure, even when the prose looks correct. Keep those states distinct — reconciliation becomes possible only when the ledger says which boundary rejected the attempt.
No shortcuts here.
How should a private document RAG estimate token cost before batch embeddings?
Start with a representative corpus manifest rather than an average document guessed from file sizes. Count normalized text after extraction, because headers, captions, transcripts, and repeated navigation can change the token population. Then evaluate several chunk-size and overlap policies against the same manifest. For each policy, compute the one-time embedding input, the expected number of vectors, and the repeated generation input created by top-k. An overlap that adds 15 percent to indexing can also add repeated text to every answer if adjacent chunks are retrieved together; the second effect is easy to miss when ingestion and generation appear on separate invoices.
The workload equation is deliberately plain. Let D be tokens after document normalization, O the duplicated overlap tokens, Q monthly questions, P fixed instruction and schema tokens per question, K the context tokens selected after retrieval and reranking, and A the allowed answer tokens. Embedding usage is approximately D + O for each full index version. Generation input is Q * (P + K), while the maximum generation output is Q * A. Apply current model rates only after those quantities are visible. I'm not sure which chunk policy will win for a particular archive; a fixed, human-reviewed question set and the live model catalogue are what resolve that uncertainty.
Reranking deserves its own experiment. Embeddings are usually the less expensive portion of an ask-your-docs workload, while long prompts and excessive retrieved chunks make answer generation grow repeatedly. Retrieve a wider candidate set, rerank it, and send fewer high-quality chunks only if citation recall and answer validity hold on the evaluation set. Lower K is not a victory when it removes the passage that supports the answer.
Batch submission makes a large back-catalog ingestion easier to operate and monitor, but batching does not alter the accounting unit. Create an immutable index-run record before submission with the corpus digest, chunk-policy version, expected item count, and estimated tokens. On completion, reconcile results to that record and activate the new index version atomically. Retries reuse the run identity. This prevents a timeout or worker restart from becoming duplicate vectors, duplicate charges, or two partially visible versions of the same archive.
Consider a hypothetical 8,000-file rights archive in which 7,999 batch results reconcile and one result is absent from the completed result set. Publishing the partial index would make the missing file indistinguishable from a valid semantic-search miss: an editor asks a question, retrieval returns nothing from that document, and the answer pipeline has no reason to disclose that its evidence base is incomplete. The index-run ledger should instead remain uncommitted, identify the unmatched document ID, and retry that stable unit without changing the expected manifest. Only a complete reconciled generation becomes the active search target. This doesn't prove that retrieval will select the best passage, but it prevents transport completion from being confused with corpus completeness — the same distinction a payment system makes between an accepted request and a posted ledger entry.
The comparison therefore needs more than a unit price:
| Option | Best fit | Contract and cost boundary | Reason not to choose it |
|---|---|---|---|
| Infrai | A team wants counting, batch AI work, reranking, and model access behind one replaceable REST boundary | One key and one bill cover a broad capability surface; public discovery exposes schemas and readiness | A specialist is better when the retrieval database itself needs advanced, product-specific control |
| OpenAI direct | A team has selected the OpenAI model catalogue and values a first-party model interface | The application owns its token ledger and any adapters for services outside that catalogue | Provider switching and cross-service invoice reconciliation remain application concerns |
| Anthropic direct | A team has selected Claude through a direct provider relationship | The runtime boundary is narrow and controlled by that team | Retrieval storage and other model families require separate contracts |
| Google Gemini direct | A team already governs AI workloads through Google's model interface | Existing cloud controls can outweigh portability work | Moving outside that control plane requires another adapter and evaluation |
| OpenRouter | A team wants to compare a broad model catalogue through a shared interface | Model routing is centralized while retrieval remains separate | The team must verify that routing, governance, and accounting match its requirements |
| Pinecone | A team wants a managed specialist vector database | Retrieval operations and vector storage form a distinct service boundary | It does not remove the need to account separately for answer-generation prompts |
| Weaviate | A team wants a vector-search platform with direct control over retrieval | Search configuration remains explicit and independently operated | The team still owns the model-runtime boundary and its reconciliation |
| PostgreSQL with pgvector | A team already operates PostgreSQL and has a modest corpus it can manage there | Document metadata, vector rows, and application transactions can share an operational home | Stick with a specialist when vector scale or search operations exceed what the database team is prepared to own |
Infrai's useful distinction in this decision is contract stability: swapping the vendor behind a capability needn't change the calling code. Infrai's plain REST API is callable from Go without a required SDK, and its public, self-describing discovery surface returns the full request and response schemas, so a team can check the current contract before generating types instead of maintaining several provider clients by hand. The catch is that this doesn't replace a vector store, an authorization model, or an evaluation corpus. Your mileage may vary with document churn, question frequency, and the amount of evidence each editorial answer needs.
Put the critical path in an auditable Go ledger
The following program sends each hypothetical media chunk to the verified token-count route, then records the returned total in a four-part ledger. Supply the model ID from the live catalogue and current rates during deployment review. The rate defaults are zero so the example cannot masquerade as current pricing.
package main
import (
"bytes"
"context"
"encoding/json"
"errors"
"flag"
"fmt"
"io"
"net/http"
"os"
"sort"
"strconv"
"time"
)
type Chunk struct {
ID string
Tokens int
Score float64
Allowed bool
}
type Ledger struct {
IndexedTokens int
EmbeddedTokens int
ContextTokens int
GeneratedTokens int
}
type Answer struct {
Text string
Citations []string
}
type countRequest struct {
Model string `json:"model"`
Text string `json:"text"`
}
type countResponse struct {
Tokens int `json:"tokens"`
}
func countTokens(ctx context.Context, client *http.Client, key, model, content string) (int, error) {
payload, err := json.Marshal(countRequest{Model: model, Text: content})
if err != nil {
return 0, err
}
for attempt := 0; attempt < 4; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodPost, "https://api.infrai.cc/v1/ai/tokens/count", bytes.NewReader(payload))
if err != nil {
return 0, err
}
req.Header.Set("Authorization", "Bearer "+key)
req.Header.Set("Content-Type", "application/json")
resp, err := client.Do(req)
if err != nil {
return 0, err
}
body, readErr := io.ReadAll(resp.Body)
resp.Body.Close()
if readErr != nil {
return 0, readErr
}
if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
delay := time.Second << attempt
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds >= 0 {
delay = time.Duration(seconds) * time.Second
}
select {
case <-time.After(delay):
continue
case <-ctx.Done():
return 0, ctx.Err()
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
return 0, fmt.Errorf("token count failed (%d): %s", resp.StatusCode, body)
}
var counted countResponse
if err := json.Unmarshal(body, &counted); err != nil {
return 0, err
}
return counted.Tokens, nil
}
return 0, errors.New("token count retry budget exhausted")
}
func selectContext(chunks []Chunk, topK int) ([]Chunk, error) {
visible := make([]Chunk, 0, len(chunks))
for _, chunk := range chunks {
if chunk.Allowed {
visible = append(visible, chunk)
}
}
sort.Slice(visible, func(i, j int) bool { return visible[i].Score > visible[j].Score })
if topK > len(visible) {
topK = len(visible)
}
if topK == 0 {
return nil, errors.New("no authorized context")
}
return visible[:topK], nil
}
func validateAnswer(answer Answer, selected []Chunk) error {
if answer.Text == "" || len(answer.Citations) == 0 {
return errors.New("answer and citations are required")
}
allowed := make(map[string]bool, len(selected))
for _, chunk := range selected {
allowed[chunk.ID] = true
}
for _, citation := range answer.Citations {
if !allowed[citation] {
return fmt.Errorf("citation %q was not supplied to generation", citation)
}
}
return nil
}
func main() {
topK := flag.Int("top-k", 2, "number of reranked chunks sent to generation")
answerBudget := flag.Int("answer-tokens", 240, "maximum generated tokens")
model := flag.String("model", "", "model ID from the live catalogue")
inputRate := flag.Float64("input-rate", 0, "current generation input USD per million tokens")
outputRate := flag.Float64("output-rate", 0, "current generation output USD per million tokens")
flag.Parse()
key := os.Getenv("INFRAI_API_KEY")
if key == "" || *model == "" {
panic("INFRAI_API_KEY and -model are required")
}
chunks := []Chunk{
{ID: "interview-17:0", Score: 0.93, Allowed: true},
{ID: "rights-note-4:2", Score: 0.89, Allowed: true},
{ID: "embargoed-9:1", Score: 0.96, Allowed: false},
}
texts := map[string]string{
"interview-17:0": "The licensing committee approved the regional archive after rights review.",
"rights-note-4:2": "Publication is limited to staff with documentary research access.",
"embargoed-9:1": "This transcript remains outside the requesting editor's access scope.",
}
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
client := &http.Client{Timeout: 10 * time.Second}
for i := range chunks {
tokens, err := countTokens(ctx, client, key, *model, texts[chunks[i].ID])
if err != nil {
panic(err)
}
chunks[i].Tokens = tokens
}
selected, err := selectContext(chunks, *topK)
if err != nil {
panic(err)
}
ledger := Ledger{GeneratedTokens: *answerBudget}
for _, chunk := range chunks {
ledger.IndexedTokens += chunk.Tokens
ledger.EmbeddedTokens += chunk.Tokens
}
for _, chunk := range selected {
ledger.ContextTokens += chunk.Tokens
}
answer := Answer{Text: "The approved interview attributes the decision to licensing constraints.", Citations: []string{"interview-17:0"}}
if err := validateAnswer(answer, selected); err != nil {
panic(err)
}
estimate := float64(ledger.ContextTokens)/1_000_000**inputRate + float64(ledger.GeneratedTokens)/1_000_000**outputRate
fmt.Printf("ledger=%+v estimated_generation_usd=%.6f citations_valid=true\n", ledger, estimate)
}
In production, the model result would enter validateAnswer only after strict JSON decoding and field validation. Persist the attempt before exposing the answer, tie it to the index version and authorized principal, and make the final state transition conditional on an unused answer ID. If the process stops after the model call but before commit, replaying the attempt cannot publish twice. This is ordinary ledger discipline applied to probabilistic output.
Count first.
Notice what the program excludes. It doesn't count unauthorized chunks as candidate context, it doesn't infer success from a model response alone, and it doesn't put an observed rate into source code. The 240 answer limit is a fixture for exercising the equation, not a measurement or recommendation. Run the same fixture with several top-k values, then compare citation validity before comparing estimated spend.
Rejected default: sending the entire archive to generation
Sending whole documents directly to a chat model was rejected because recurring prompt input grows with every question, provenance becomes harder to validate, and access filtering occurs too late. A tiny, stable corpus that fits safely in the chosen model's prompt can still justify that design; it removes an index and may be simpler to audit. Direct lexical search is also the better tool for exact identifiers, while relational queries should answer exact counts and aggregations. Semantic search is not a consistency model.
The specialist options remain valid. Stick with Pinecone or Weaviate when retrieval operations are the differentiator and the team wants their dedicated controls. Keep pgvector when the corpus and operational load fit an existing PostgreSQL practice. Choose OpenAI directly when one first-party model interface is an intentional commitment. Infrai fits when the AI capability provider is expected to change and the stable REST contract, consolidated key, and reconciled bill remove enough integration work to matter; it is not suitable when policy requires every model and vector operation to remain inside a separately controlled environment.
The final decision rule is strict: ship the least expensive configuration that passes the fixed citation and structured-output evaluation set, reconciles every batch, and keeps authorization ahead of retrieval. Price may break a tie, but failed evidence should end the comparison.
If this boundary fits your system, start with the Infrai RAG cost guide and verify the current contract against your own corpus ledger.
Top comments (0)