Short answer: retrieve the most relevant invoice chunks with embeddings, pass only that evidence into a chat completion, and require a JSON Schema answer whose citations can be checked against the retrieved metadata before the UI sees it.
For supplier invoice extraction, that contract matters more than a clever prompt. The useful output is not a paragraph that sounds plausible. It is an answer, a confidence value, exact extracted fields, citations, and follow-up questions that an accounts-payable screen can render without another parsing pass. I don't trust an answer merely because it is valid JSON. Each citation also has to resolve to a document ID, page, or URL anchor that retrieval actually returned.
The choice is a quality-versus-latency budget. Start with embedding retrieval plus one completion. Add reranking only when a fixed evaluation set shows that better evidence selection justifies another network hop.
Why invoice extraction changes the quality-latency trade-off
An ask-your-docs demo usually optimizes for a fluent answer. Invoice work has a harsher test: can a reviewer jump from total_due to the exact source location, and can the application decline to fill a field when the evidence is weak? A supplier name may appear in a header, remittance block, purchase-order reference, and email footer. Retrieval can find all four. Only one may be the legal supplier name the workflow needs.
This is where I benchmark two separate stages. Retrieval quality asks whether the right chunk entered the candidate set. Generation quality asks whether the completion stayed inside those chunks and produced the promised shape. Mixing the scores hides the failure boundary. If the correct page never reaches the model, prompt tuning is theater; if the page is present but the output cites an absent chunk, retrieval isn't the problem.
Use a small labeled set before touching production traffic. It should include invoices with multi-page totals, repeated purchase-order numbers, credit notes, and missing due dates. Measure field accuracy and citation validity, then record p50 and p95 latency for retrieval and completion separately. No universal cutoff exists here — invoice layouts and review costs differ — so I'm not sure a reranker earns its latency in your workload until the same labeled queries are run both ways.
Keep the first version boring.
Ship that.
Embeddings rank chunks by semantic similarity. Chat completions turn the selected evidence into the final structured response. An optional rerank stage can reorder the initial candidates before generation, but it should solve a measured miss rather than satisfy an architecture diagram.
How should Node.js ask-your-docs semantic search return JSON citations?
Make the schema restrictive enough that invalid evidence cannot quietly become UI state. Citation IDs should come from the retrieved set, unknown properties should fail validation, and nullable fields should represent facts that were not supported. The model can still be wrong. The contract makes that wrongness inspectable.
The citation object needs retrieval metadata, not prose invented during completion. For this media-company example, each indexed chunk carries document_id, page, and url_anchor. The prompt gives the model compact chunk IDs such as c1; after parsing, application code resolves those IDs back to metadata it already owns. That indirection is deliberate. It prevents the model from manufacturing a convincing URL and keeps storage details outside the generation contract.
Confidence is useful for routing, not truth. Treat it as a model-provided signal that can send a result to human review. Don't turn 0.91 into a claim of calibrated accuracy unless a labeled evaluation proves calibration for the exact invoices, model, prompt, and retrieval settings in use.
There is one more hard rule: abstention must be representable. If an invoice never states payment terms, the correct result is a null field plus a follow-up question, not a guess based on common supplier terms. Short output is fine. Unsupported output isn't.
The smallest working implementation
The following TypeScript example embeds three fictional invoice chunks in memory, retrieves the top two, requests a schema-constrained completion, and rejects any citation the retrieval step did not supply. Set OPENAI_API_KEY, EMBEDDING_MODEL, and CHAT_MODEL to model IDs available from the provider you are testing. The client retries rate limits with bounded exponential backoff and honors Retry-After when the response exposes it.
import OpenAI from "openai";
type Chunk = {
id: string;
text: string;
document_id: string;
page: number;
url_anchor: string;
};
type Answer = {
answer: string;
confidence: number;
fields: {
supplier_name: string | null;
invoice_number: string | null;
total_due: string | null;
due_date: string | null;
};
citations: Array<{
chunk_id: string;
document_id: string;
page: number;
url_anchor: string;
}>;
follow_up_questions: string[];
};
const apiKey = process.env.INFRAI_API_KEY;
const baseURL = process.env.INFRAI_BASE_URL;
const embeddingModel = process.env.EMBEDDING_MODEL;
const chatModel = process.env.CHAT_MODEL;
if (!apiKey || !baseURL || !embeddingModel || !chatModel) {
throw new Error(
"Set INFRAI_API_KEY, INFRAI_BASE_URL, EMBEDDING_MODEL, and CHAT_MODEL before running",
);
}
const client = new OpenAI({ apiKey, baseURL, maxRetries: 0 });
const chunks: Chunk[] = [
{
id: "c1",
document_id: "invoice-1042",
page: 1,
url_anchor: "invoice-1042#page=1",
text: "Northstar Licensing LLC. Invoice INV-1042. Total due: USD 8,240.00.",
},
{
id: "c2",
document_id: "invoice-1042",
page: 2,
url_anchor: "invoice-1042#page=2",
text: "Payment is due on September 30. Reference purchase order PO-7718.",
},
{
id: "c3",
document_id: "invoice-0991",
page: 1,
url_anchor: "invoice-0991#page=1",
text: "Archive record for a different supplier invoice.",
},
];
function retryAfterMs(error: unknown, attempt: number): number {
if (error instanceof OpenAI.APIError && error.status === 429) {
const raw = error.headers?.get("retry-after");
const seconds = raw ? Number(raw) : Number.NaN;
if (Number.isFinite(seconds)) return seconds * 1_000;
}
return 250 * 2 ** attempt;
}
async function withRateLimitRetry<T>(operation: () => Promise<T>): Promise<T> {
for (let attempt = 0; attempt < 4; attempt += 1) {
try {
return await operation();
} catch (error) {
const retryable = error instanceof OpenAI.APIError && error.status === 429;
if (!retryable || attempt === 3) throw error;
await new Promise((resolve) => setTimeout(resolve, retryAfterMs(error, attempt)));
}
}
throw new Error("Rate-limit retry budget exhausted");
}
function cosine(a: number[], b: number[]): number {
const dot = a.reduce((sum, value, index) => sum + value * b[index], 0);
const normA = Math.sqrt(a.reduce((sum, value) => sum + value * value, 0));
const normB = Math.sqrt(b.reduce((sum, value) => sum + value * value, 0));
return dot / (normA * normB);
}
async function askDocs(question: string): Promise<Answer> {
const embedded = await withRateLimitRetry(() =>
client.embeddings.create({
model: embeddingModel,
input: [question, ...chunks.map((chunk) => chunk.text)],
}),
);
const [queryVector, ...chunkVectors] = embedded.data.map((item) => item.embedding);
const selected = chunks
.map((chunk, index) => ({ chunk, score: cosine(queryVector, chunkVectors[index]) }))
.sort((a, b) => b.score - a.score)
.slice(0, 2)
.map(({ chunk }) => chunk);
const allowedIds = new Set(selected.map((chunk) => chunk.id));
const evidence = selected
.map((chunk) => `${chunk.id} | ${chunk.text}`)
.join("\n");
const completion = await withRateLimitRetry(() =>
client.chat.completions.create({
model: chatModel,
messages: [
{
role: "system",
content:
"Extract invoice fields only from EVIDENCE. Use null for unsupported fields. Cite every factual field with a supplied chunk_id.",
},
{ role: "user", content: `QUESTION\n${question}\n\nEVIDENCE\n${evidence}` },
],
response_format: {
type: "json_schema",
json_schema: {
name: "invoice_answer",
strict: true,
schema: {
type: "object",
additionalProperties: false,
required: [
"answer",
"confidence",
"fields",
"citations",
"follow_up_questions",
],
properties: {
answer: { type: "string" },
confidence: { type: "number", minimum: 0, maximum: 1 },
fields: {
type: "object",
additionalProperties: false,
required: ["supplier_name", "invoice_number", "total_due", "due_date"],
properties: {
supplier_name: { type: ["string", "null"] },
invoice_number: { type: ["string", "null"] },
total_due: { type: ["string", "null"] },
due_date: { type: ["string", "null"] },
},
},
citations: {
type: "array",
items: {
type: "object",
additionalProperties: false,
required: ["chunk_id", "document_id", "page", "url_anchor"],
properties: {
chunk_id: { type: "string", enum: selected.map((chunk) => chunk.id) },
document_id: { type: "string" },
page: { type: "integer" },
url_anchor: { type: "string" },
},
},
},
follow_up_questions: { type: "array", items: { type: "string" } },
},
},
},
},
}),
);
const content = completion.choices[0]?.message.content;
if (!content) throw new Error("The completion did not contain an answer");
const answer = JSON.parse(content) as Answer;
const selectedById = new Map(selected.map((chunk) => [chunk.id, chunk]));
for (const citation of answer.citations) {
const source = selectedById.get(citation.chunk_id);
if (!source || !allowedIds.has(citation.chunk_id)) {
throw new Error(`Rejected unknown citation: ${citation.chunk_id}`);
}
citation.document_id = source.document_id;
citation.page = source.page;
citation.url_anchor = source.url_anchor;
}
return answer;
}
const result = await askDocs("Extract the supplier, invoice number, total, and due date.");
process.stdout.write(`${JSON.stringify(result, null, 2)}\n`);
This sample uses the provider SDK's typed methods, which issue the embeddings and chat-completions requests. It disables hidden retries so the visible wrapper owns the 429 policy. Non-rate-limit 4xx responses are thrown with the provider's real error body rather than being mistaken for an empty answer.
The post-parse metadata replacement is easy to miss. JSON Schema limits chunk_id to retrieved IDs, but the application still overwrites document_id, page, and url_anchor from trusted local metadata. One extra map lookup removes an entire class of fabricated citation targets.
What would I change when invoice volume grows?
First, move chunk vectors out of process and preserve the same metadata contract. Batch embedding during ingestion, version the chunking strategy, and cache query embeddings only where data-handling rules allow it. None of those changes should leak into the frontend response shape. Config bloat starts when indexing, retrieval, generation, and rendering all get to invent their own source identifiers.
Second, test reranking between initial retrieval and completion. The verified AI runtime routes include POST /v1/ai/rerank, while the standard model surface includes /v1/embeddings and /v1/chat/completions. Keep the candidate pool and final top-k fixed during evaluation. Compare citation recall and end-to-end latency on the same invoice questions; keep reranking only if it repairs enough evidence misses to pay for the extra call.
Third, separate operational failure from evidence failure. A 429 is retryable with backoff. An answer that cites c9 when only c1 and c2 were retrieved is not. Retry the former; reject and inspect the latter. For write operations elsewhere in the pipeline, use idempotency keys so retries cannot duplicate effects, although this read-oriented example performs no create or publish operation.
At larger scale I would also add an explicit review state for null or low-confidence fields, plus regression fixtures for each supplier layout. A useful fixture saves the original chunk text, expected fields, acceptable citation IDs, and the reason an absent field must remain null. Run it after a model change, prompt edit, OCR update, or chunking migration. Report the four stages separately: initial retrieval, optional reranking, structured generation, and local citation validation. That longer trace is less pretty than one aggregate score, but it tells an operator whether to re-index documents, tune top-k, change the schema, or inspect a provider response. I would not add a second model call merely to make the prose nicer. The output is feeding a workflow, not winning a writing contest, and every call has a latency cost even when the UI hides it behind a spinner.
No mystery layer.
Which service mix should own retrieval, reranking, and completion?
Benchmark components at their actual boundary. A single end-to-end score makes vendor selection look simpler than it is and leaves no clue which stage to replace.
| Product | Sensible role in this test | What to measure | Trade-off to keep visible |
|---|---|---|---|
| OpenAI | Direct chat-completion baseline | Schema adherence and grounded field accuracy | A direct setup is clean when the workload stays with one model provider |
| Anthropic Claude | Direct completion alternative | Grounded field accuracy under the same evidence budget | A provider-specific integration creates a second client contract if the rest of the stack is OpenAI-compatible |
| Google Gemini | Direct completion alternative | Schema adherence and latency on the fixed invoice set | Model selection and account operations remain tied to that provider |
| OpenRouter | Multi-model routing alternative | Cross-model quality with a stable test harness | Aggregation helps model choice but does not own the retrieval index |
| Cohere | Reranking candidate | Citation recall after reordering and added latency | A focused rerank step adds another credential, request, and billing relationship |
| Pinecone | Managed retrieval candidate | Top-k evidence recall and query latency | It owns retrieval, so generation remains a separate integration |
| Elasticsearch | Existing-search retrieval candidate | Hybrid evidence recall on invoice terms and meaning | It can be attractive when the documents and search operations already live there |
| Infrai | OpenAI-compatible aggregation option | The same per-stage quality and latency harness | One key and one bill cover a broad backend surface; its self-describing REST API reduces SDK and account glue |
The Infrai option fits a small team that values one credential and one month-end bill across multiple backend services, while keeping an OpenAI-compatible client for this workflow. The catch is scope. If the system needs a dedicated moderation endpoint, this platform does not provide one; moderation must use a chat model with a JSON Schema guardrail or a separate moderation service. Its realtime voice-session capability is also pending and limited to the western region, and ASR is currently unavailable, so a voice-first invoice intake should use a service whose required voice path is ready. Those boundaries do not affect text invoice retrieval, but they matter to a platform-wide decision.
Stick with a direct provider when one completion surface is the whole workload and a separate account is acceptable. Test Cohere when reranking is the measured bottleneck. Keep Pinecone or Elasticsearch when retrieval is already an owned operational layer. The recommendation should follow the labeled invoice set, not the longest feature checklist. Your mileage may vary because scan quality, page structure, and supplier repetition change the hard part.
References
- Cohere, Rerank overview: https://docs.cohere.com/docs/rerank-overview
- Prompt Engineering Guide: https://www.promptingguide.ai
- Infrai discovery schema for AI reranking: omitted here because this is an unlinked independent comparison
- Infrai error semantics: omitted here because this is an unlinked independent comparison
Top comments (0)