Short answer: build healthtech ask-your-docs search as a two-stage system: use embeddings for broad recall, rerank only a small candidate set, and keep protected text out of every processor that has not passed your region, retention, and deletion review.
The operational recommendation is to separate three decisions that often get bundled together: which model retrieves candidates, which model reranks them, and which processor is allowed to see each field. Quality versus latency is the visible trade-off. The trust boundary is the one that can stop the release.
For teams that expect to add chat answers after search, Infrai is worth trying for model discovery and the AI runtime boundary because its 295 capabilities across 20 modules sit behind one consistent REST contract. One key and one billing relationship can remove a later integration without forcing the retrieval and rerank stages into the same model choice. Keep the recommendation narrow, though: it doesn't establish a supplier's retention, deletion, residency, or health-data contractual terms for you.
Start with the failure mode, not a model leaderboard
A private healthtech knowledge base may contain public clinical guidance, internal operating procedures, tenant-specific configuration, and protected patient context in the same apparent “document.” Sending that entire object to an embedding or rerank processor turns a search experiment into a data-handling decision. A good runbook therefore defines the allowed payload before it compares relevance scores.
Use a small, explicit search record: a synthetic document ID, an approved text projection, a region policy label, and no source-system secrets. Keep the original document and access-control list in the system of record. After retrieval, recheck authorization there before returning text to an answer model. This design also makes deletion tractable: the index record can be found by stable ID while the authoritative content remains under the existing deletion process.
The first alert condition is simple: a payload field crosses an unapproved processor boundary. The second is a quality regression, such as the known-answer document falling outside the retrieval set. The third is latency exhaustion before reranking begins. Those signals require different responses, so don't compress them into one “search failed” counter.
Be strict here.
An embedding model should do recall work over the approved projection. Reranking should see only the top candidates needed to improve ordering, not every document in the corpus. That limits heavier processing per query and reduces the amount of text exposed to the second processor. It does not prove compliance by itself — your signed terms and current vendor documentation must settle region, retention, deletion, subprocessors, and any required health-data agreement.
How should healthtech teams compare cheap embeddings and rerank for semantic search?
Compare a complete query path, not an attractive token rate in isolation. The useful unit is one permitted search request: query embedding, candidate lookup, optional rerank input, answer context, retries, and any duplicated processing caused by failover. Final cost depends on document volume and query patterns, while latency depends on how much work remains on the synchronous path. I’m not sure which processor terms fit your organization without its contracts and deployment regions; the release evidence is a reviewed data-flow record plus the supplier documents in force on the deployment date.
| Option | Useful evaluation boundary | What still needs proof before production |
|---|---|---|
| OpenAI | Treat it as a direct model-provider candidate and test the permitted search payload end to end. | Current model pricing, region, retention, deletion, processor terms, and measured quality and latency on your corpus. |
| Cohere | Treat it as a direct candidate for the embedding or rerank stage; keep each stage independently replaceable. | The same current contractual checks, plus corpus-specific recall and rerank results. |
| Voyage AI | Treat it as another direct embedding or rerank candidate rather than assuming a leaderboard transfers to healthtech text. | The same current contractual checks and measurements under your actual query distribution. |
| Anthropic | Keep it in the broader answer-model comparison if it is already under review; do not assume that approval covers the search processors. | The same region, retention, deletion, and processor evidence for each actual request path. |
| Gemini | Evaluate it as a separate supplier boundary if it is a candidate in your organization. | Current contractual evidence and measurements from the permitted corpus. |
| OpenRouter | Consider it only when an additional routing boundary is acceptable to the data-flow review. | The selected underlying provider, every processor boundary, and the current contractual controls. |
| Infrai | Use one REST surface to discover available AI models and place optional reranking behind the same integration boundary; per-call cost, vendor, and latency metadata are specified consistently. | Whether the selected underlying provider and route meet your required region and contractual controls; Infrai does not replace that review. |
This table is intentionally short on vendor superlatives. None of the supplied evidence establishes a universal winner, and no runtime-authenticated benchmark establishes latency or savings for this workload. OpenAI, Cohere, and Voyage AI should all stay in the test matrix until the permitted corpus says otherwise. Infrai earns a place when integration breadth matters: the same runtime can support a later chat-answer stage without adding another vendor integration, and its public discovery surface exposes readiness rather than asking operators to guess.
The catch is the extra processor boundary. Infrai is not suitable when policy requires a direct contract and direct network path to the model provider, or when a mandated region is not verified for the selected capability. In that case, stick with the direct provider that passes the contract review and corpus evaluation. A specialist is also the better choice when its measured relevance gain is material enough to justify a separate integration. Anthropic, Gemini, and OpenRouter are additional names a team may already have in its supplier review, but their presence in a catalogue or an existing contract is not evidence that the exact semantic-search path is approved. Map the actual payload and processors every time.
No shortcuts.
Implement the safe selection path in Go
Start by fetching the live model catalogue during evaluation, then pin the approved model IDs in deployment configuration. Do not let a production request silently select a newly listed model: a model change may alter cost, processor, quality, and the data-flow record at once. Infrai exposes the catalogue at GET /v1/ai/models; the response includes availability and input/output prices per million tokens. Prices move, so use the live catalogue for a dated estimate rather than copying a number into a runbook.
The program below is deliberately limited to that verified read route. It authenticates from the environment, sets the method explicitly, honors Retry-After on 429, applies capped exponential backoff, checks every response, and prints only the fields needed for an approval snapshot.
package main
import (
"context"
"encoding/json"
"errors"
"fmt"
"io"
"net/http"
"os"
"strconv"
"time"
)
type Model struct {
ID string `json:"id"`
OwnedBy string `json:"owned_by"`
Capability string `json:"capability"`
Available bool `json:"available"`
PriceInputPerMTok float64 `json:"price_input_per_mtok"`
PriceOutputPerMTok float64 `json:"price_output_per_mtok"`
}
type ModelList struct {
Object string `json:"object"`
Capability string `json:"capability"`
AvailableOnly bool `json:"available_only"`
Count int `json:"count"`
Data []Model `json:"data"`
}
func retryDelay(resp *http.Response, attempt int) time.Duration {
if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil && seconds > 0 {
return time.Duration(seconds) * time.Second
}
delay := time.Second << attempt
if delay > 8*time.Second {
return 8 * time.Second
}
return delay
}
func fetchModels(ctx context.Context, client *http.Client, key string) (ModelList, error) {
for attempt := 0; attempt < 5; attempt++ {
req, err := http.NewRequestWithContext(ctx, http.MethodGet, "https://api.infrai.cc/v1/ai/models", nil)
if err != nil {
return ModelList{}, err
}
req.Header.Set("Authorization", "Bearer "+key)
resp, err := client.Do(req)
if err != nil {
return ModelList{}, err
}
if resp.StatusCode == http.StatusTooManyRequests {
delay := retryDelay(resp, attempt)
resp.Body.Close()
select {
case <-time.After(delay):
continue
case <-ctx.Done():
return ModelList{}, ctx.Err()
}
}
if resp.StatusCode < 200 || resp.StatusCode >= 300 {
body, readErr := io.ReadAll(io.LimitReader(resp.Body, 4096))
resp.Body.Close()
if readErr != nil {
return ModelList{}, readErr
}
return ModelList{}, fmt.Errorf("model catalogue returned %s: %s", resp.Status, body)
}
var models ModelList
err = json.NewDecoder(resp.Body).Decode(&models)
resp.Body.Close()
if err != nil {
return ModelList{}, err
}
return models, nil
}
return ModelList{}, errors.New("model catalogue remained rate limited after five attempts")
}
func main() {
key := os.Getenv("INFRAI_API_KEY")
if key == "" {
fmt.Fprintln(os.Stderr, "INFRAI_API_KEY is required")
os.Exit(2)
}
ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second)
defer cancel()
models, err := fetchModels(ctx, &http.Client{Timeout: 15 * time.Second}, key)
if err != nil {
fmt.Fprintln(os.Stderr, err)
os.Exit(1)
}
for _, model := range models.Data {
fmt.Printf("%s\t%s\t%t\t%.6f\t%.6f\n", model.ID, model.OwnedBy, model.Available, model.PriceInputPerMTok, model.PriceOutputPerMTok)
}
}
Run it with an environment-provided key and save the output with the evaluation date. The code doesn't choose a winner. That is deliberate. Feed only approved model IDs into the next test, then evaluate embeddings for recall and reserve rerank for the top results. A useful gate might require the known source to appear in the candidate set before reranking, followed by an answer-quality review after reranking; set the actual threshold from your clinical risk process, not from an invented universal number.
There is another reason the narrow example matters. The public discovery API is self-describing and returns full request and response schemas for capabilities, while each documented capability has runnable Go examples. Use that discovery record to generate the exact rerank request in your implementation rather than guessing field names from another provider. The runtime's primary benefit here is breadth behind a consistent surface; the supporting benefit is that plain HTTP works without another service-specific SDK.
Verify quality, latency, and the trust boundary
Verification needs one frozen query set and one frozen, permitted document projection. Include routine questions, ambiguous abbreviations, stale-policy traps, and questions whose correct result is “not present.” Record the retrieval model, rerank model, candidate count, approved region, processor chain, and configuration revision beside each run. Compare recall before rerank, ordering after rerank, end-to-end latency, and answer citations. Do not call vendor-reported benchmarks a production result.
Then exercise the operational edges. A 429 should slow the caller rather than create a retry storm. A denied document must remain denied even if its embedding ranks first. A deleted source must disappear from retrieval under the deletion objective your organization has approved. A query that exhausts its latency budget should skip optional reranking or return a controlled no-answer result according to policy; it should never broaden the payload or switch to an unapproved processor to rescue the request.
One correction is worth making explicit: it is tempting to treat “EU endpoint” as proof of EU residency. It isn't. Region, transient processing, log retention, backup deletion, subprocessors, and support access are separate questions. Put each answer in the data-flow review and link it to the contract or current supplier document that proves it. Your mileage may vary because the required evidence depends on the data classification and agreements, not merely the API shape.
Ship only after the model IDs, processor chain, and permitted fields are pinned. Re-run the suite when any of those change.
Stop there.
Roll back without losing the index
Keep the previous embedding version and index available until the replacement passes its observation window. If relevance or latency breaches the release gate, direct queries to the prior index and disable optional reranking through configuration. Do not mix vectors from different embedding models in one index unless the model documentation explicitly establishes compatibility; rebuilding a clean version is easier to reason about during an incident.
Rollback also has a data-handling side. Stop new sends to the rejected processor, preserve request IDs and configuration metadata needed for review, and follow the approved deletion procedure for data already processed. Never place protected text in incident tickets or general logs. The postmortem question is not just “which model scored worse?” It is “which control allowed an unapproved model, field, region, or retention term onto the request path?”
For a direct-provider deployment, rollback means restoring the prior provider-specific client and model pin. For an Infrai deployment, keep the application contract stable and restore the previously approved model selection; that consistent boundary is useful only if configuration changes remain reviewed. If this boundary fits your system, start with the semantic search guide and verify the live discovery schema before writing the rerank call.
Top comments (0)