The operational constraint changes the answer: a gaming chatbot that helps staff find supplier invoices must return the right fields, not merely a plausible answer. I would use an OpenAI-compatible /v1/chat/completions call alongside /v1/embeddings, with semantic search supplying the relevant invoice passages and structured output enforcing the final shape.
Short answer: retrieve first, extract second
Embeddings turn a question such as “show the controller shipment invoices with disputed quantities” into a semantic lookup rather than a brittle keyword match. The chat model then sees the matching invoice text, interprets the request, and emits fields such as supplier, invoice number, line items, currency, tax, and dispute status.
This separation matters. Search decides which evidence is in the context. Generation decides how that evidence maps to the schema. Combining those jobs in one long prompt makes it harder to tell whether a wrong answer came from retrieval or extraction.
For an in-app chatbot, the useful contract is therefore: retrieve a bounded set of passages, preserve their source identifiers, ask for JSON, validate it, and show a missing field as missing. Do not let a polished sentence hide an absent invoice number.
What does semantic search fix in invoice chat?
Supplier invoices rarely use the same vocabulary as the person asking the question. One document may say “SKU,” another “item code,” while a finance operator asks for “the product identifier.” Embeddings help reinterpret that question and extract intent across those terms. They are also a better fit for dynamic, real-time queries than a growing collection of hand-written keyword rules.
Consider a result containing this passage:
Invoice INV-2048 lists 120 wireless controllers at USD 18.50 each. The received quantity is 116. The supplier is Northstar Distribution, and the invoice total includes 8% tax.
The chatbot should not infer that 120 units were received. A good structured response would retain invoice_number: "INV-2048", distinguish ordered from received quantity, and flag the four-unit discrepancy. That is the evaluation constraint I would write before choosing a model.
The practical test is small but unforgiving: use a held-out set of invoices, measure exact field accuracy and evidence attribution, then inspect failures by field. Supplier names and invoice numbers tend to expose retrieval errors; tax and quantity fields expose schema and arithmetic errors. Measure those separately before copying the design into another workflow.
Which implementation fits a small product team?
The obvious alternatives each solve a different part of the problem.
| Option | SDK or REST | Setup cost | Best fit | Main limit |
|---|---|---|---|---|
| OpenAI | SDK and REST | Moderate | A direct chat-and-embedding workflow | The application owns retrieval and validation |
| Google Vertex AI | Cloud SDK and APIs | Higher outside Google Cloud | Teams already using Google identity and data controls | More cloud-specific architecture |
| Cohere | SDK and REST | Moderate | Search pipelines where reranking is central | Still needs a separate generation and validation path |
| Pinecone | SDK and REST | Moderate | A managed vector indexing and retrieval layer | It does not generate or validate the invoice answer |
| Infrai | OpenAI-compatible REST surface | Low for an existing compatible client | Teams consolidating backend access behind one credential | Retrieval evaluation and schema validation remain application work |
OpenAI's API is a direct choice when the team wants a mature chat-and-embedding pattern and is comfortable assembling retrieval, storage, validation, and observability around it. Its strength here is the clear division between embedding generation and chat completion. The trade-off is another set of platform decisions to own.
Google Vertex AI is a better fit for a team already standardized on Google Cloud identity, data controls, and managed search services. That integration can reduce platform sprawl inside that cloud. It also makes the architecture less portable if the application later needs a different provider or deployment boundary.
Cohere is worth considering when reranking is the main retrieval problem. Its search-oriented tooling can improve the ordering of candidate passages, but it is another specialized dependency to coordinate with the chat model and invoice validator.
Pinecone is a vector database rather than a complete answer-generation API. It can be a sensible choice when the team wants a managed retrieval layer with its own indexing lifecycle. It does not, by itself, decide whether received_quantity is trustworthy or produce the final invoice schema.
That comparison is why I would not select on model quality in isolation. The right question is which boundary the team wants to operate: a set of specialized services, or one compatible runtime surface with retrieval and extraction kept explicit in application code.
Where does the one-surface option fit?
An API gateway that exposes a genuine OpenAI-compatible surface can reduce key and billing fragmentation while leaving the application flow familiar. Infrai fits that narrow case: one key and one bill across backend services, plus a public discovery surface and per-call metadata for cost, latency, vendor, cache status, and request ID. Its multi-vendor routing also makes the selected model a replaceable detail rather than a permanent application dependency.
That convenience has a boundary. It does not remove the need to own chunking, source references, schema validation, or retrieval evaluation. The verified AI runtime facts also do not justify treating currently pending voice-session or unavailable transcription capabilities as part of this invoice design. Keep the chatbot path on the two supported interfaces and test the actual model and vendor combination you select.
For this workflow, the benefit is operational coherence, not a claim that one gateway is universally better. If the team already has strong cloud procurement, logging, and identity conventions, a native provider may be the simpler choice. If it is stitching together several backend capabilities and wants a consistent REST boundary, the single-surface option deserves a serious trial.
Here is the smallest useful verification exercise. Set INFRAI_BASE_URL to the documented API base, INFRAI_API_KEY to the key, and INFRAI_CHAT_MODEL to an available chat model. The application supplies the retrieved invoice passage; the call tests the extraction boundary without pretending that the model performed search.
const baseUrl = process.env.INFRAI_BASE_URL;
const apiKey = process.env.INFRAI_API_KEY;
const model = process.env.INFRAI_CHAT_MODEL;
if (!baseUrl || !apiKey || !model) {
throw new Error("Set INFRAI_BASE_URL, INFRAI_API_KEY, and INFRAI_CHAT_MODEL");
}
const invoicePassage =
"Invoice INV-2048 lists 120 wireless controllers at USD 18.50 each. " +
"The received quantity is 116. Supplier: Northstar Distribution. Tax: 8%.";
const requestBody = JSON.stringify({
model,
messages: [
{
role: "system",
content:
"Extract only evidence present in the invoice passage. Return JSON with " +
"invoice_number, supplier, ordered_quantity, received_quantity, tax_rate, " +
"and discrepancy.",
},
{ role: "user", content: invoicePassage },
],
});
let response: Response;
for (let attempt = 0; attempt < 4; attempt += 1) {
response = await fetch(`${baseUrl}/chat/completions`, {
method: "POST",
headers: {
Authorization: `Bearer ${apiKey}`,
"Content-Type": "application/json",
},
body: requestBody,
});
if (response.status !== 429 || attempt === 3) break;
const retryAfter = Number(response.headers.get("retry-after"));
const delayMs = Number.isFinite(retryAfter)
? retryAfter * 1000
: 250 * 2 ** attempt;
await new Promise((resolve) => setTimeout(resolve, delayMs));
}
if (!response.ok) {
const detail = await response.text();
throw new Error(`Chat request failed (${response.status}): ${detail}`);
}
const result = await response.json();
console.log(result.choices?.[0]?.message?.content ?? result);
For production, wrap the request in bounded exponential backoff for HTTP 429, validate the returned object against the application schema, and persist the passage identifier beside every populated field. A successful HTTP response is not evidence that the extraction is correct.
A focused experiment before shipping
I would run the same 50-question evaluation against three paths: direct OpenAI, a Google Cloud setup, and an OpenAI-compatible gateway. The dataset should include duplicate invoice numbers, synonyms for line items, missing tax values, quantity disputes, and questions that have no matching document. Fifty is enough to expose obvious schema and retrieval failures without pretending to be a production benchmark.
The pass condition is not “the answer sounds right.” Require valid JSON, exact invoice-number matches, correct separation of ordered and received quantities, a source passage for every populated field, and an explicit null or refusal when the corpus has no evidence. Then record retrieval misses separately from extraction misses.
My initial instinct would have been to send the whole invoice collection to chat and ask for the answer. That is easy to demo and difficult to audit. The two-stage design wins because each error has a place to inspect, and because adding a new supplier's terminology does not require rewriting the entire conversation prompt.
The final decision rule is straightforward: choose the stack that meets the field-level accuracy threshold on the held-out invoices while keeping retrieval evidence inspectable and vendor changes tolerable. For an in-app gaming chatbot, that usually points to /v1/chat/completions plus /v1/embeddings; the surrounding gateway is an operational choice, not the quality guarantee.
The answer can be fluent and still be wrong.
That is the boundary worth testing.
Sources
References:
- OpenAI API documentation: https://platform.openai.com/docs/api-reference
- Google Cloud Vertex AI documentation: https://cloud.google.com/vertex-ai/docs
- Cohere Embed and Rerank documentation: https://docs.cohere.com/docs
- Pinecone documentation: https://docs.pinecone.io/
- MDN, HTTP 429 Too Many Requests: https://developer.mozilla.org/en-US/docs/Web/HTTP/Reference/Status/429
Top comments (0)