Build the candidate-scoring pipeline around four replayable boundaries: page extraction, embeddings retrieval, reranking, and the final summary. Provider portability then becomes a controlled replay of stored inputs and normalized outputs, not a hurried rewrite during an incident.
For customer-support hiring, the durable result is not polished prose. It is a rubric decision whose evidence can be traced to a stable PDF page and reproduced after an AI provider changes. Keep the original document, extracted passages, retrieval candidates, reranked evidence, and structured decision separate. A human reviewer should see the cited passages alongside any score.
This is the operating rule: no stage may invent evidence or hide the identity of its input. Treat text inside an uploaded PDF as untrusted data, require evidence IDs in every supported rubric decision, and make insufficient_evidence a valid result. OWASP identifies prompt injection and sensitive-information disclosure among the risks for LLM applications; a sentence in a resume that tells the model to ignore the rubric remains resume content, never an instruction.
Define portability before choosing a runtime
Start with a fixture, not a vendor comparison. Use a synthetic candidate PDF that has stable expected facts spread across several pages: a support escalation example, a coaching example, a ticketing-system claim, and one criterion with no supporting evidence. The fixture should also contain repeated headers and an instruction-like sentence in the document body. None of those details are a benchmark. They are assertions that each adapter must preserve or reject in the same way.
The four boundaries need narrower contracts than a generic generate() function. Extraction accepts immutable document bytes and returns ordered pages. Embedding accepts versioned passages and returns vectors associated with passage IDs. Reranking accepts one rubric criterion plus a candidate set and returns the same IDs in an order. Summarization accepts only selected evidence and returns rubric decisions with evidence IDs. Provider-specific model names, vector dimensions, score scales, response fields, and credentials stay behind adapters.
That separation matters because portability does not mean equivalence. Two embedding models can produce incompatible vectors. Two rerankers can expose scores that look numeric but do not share a scale. Two generators can follow the same schema while choosing different wording. The migration test should compare business invariants -- selected evidence, supported versus insufficient decisions, and schema validity -- rather than raw scores or byte-for-byte prose.
Use a manifest as the handoff record:
package pipeline
type Manifest struct {
WorkflowID string `json:"workflow_id"`
DocumentHash string `json:"document_hash"`
ParserVersion string `json:"parser_version"`
EmbeddingVersion string `json:"embedding_version"`
RerankerVersion string `json:"reranker_version"`
PromptVersion string `json:"prompt_version"`
RubricVersion string `json:"rubric_version"`
PassageIDs []string `json:"passage_ids"`
}
type Passage struct {
ID string `json:"id"`
Page int `json:"page"`
Text string `json:"text"`
TextHash string `json:"text_hash"`
}
type RankedPassage struct {
PassageID string `json:"passage_id"`
Rank int `json:"rank"`
Score float64 `json:"score"`
}
type RubricDecision struct {
Criterion string `json:"criterion"`
Rating string `json:"rating"`
EvidenceIDs []string `json:"evidence_ids"`
}
Persist the manifest before queueing the next stage. Derive an idempotency key from the document hash and all configuration versions that affect the result. I've been paged by missed jobs and duplicate deliveries; the useful lesson is plain: queue delivery is an attempt, while the compare-and-set commit is the event that makes a result real. A retry may recompute an artifact, but it must not create a second hiring record or notify a reviewer twice.
There is a privacy edge here too. Candidate documents contain personal data, and derived artifacts deserve the same retention and access planning as the source. GDPR principles include purpose limitation and data minimization. Do not ship unrelated headers, hidden metadata, or unused pages merely because extraction made them available, and do not place candidate text, email addresses, or prompts in metrics. Legal duties vary by jurisdiction and processing context, so the policy owner and counsel must set the actual retention and deletion rules.
How should Node.js use semantic search to rerank PDF pages?
Let Node.js own workflow state, queue leases, and adapter selection, but keep the wire contract language-neutral. The executable contract below is in Go because this runbook standardizes operational examples in Go; a Node.js worker can exchange the same JSON shapes with any conforming adapter.
Split each PDF into pages with stable identities, then form smaller passages without losing the page number. Repeated boilerplate can distort retrieval, so mark headers and footers during extraction and exclude them when they carry no rubric evidence. Hash the normalized passage text. If parsing changes the text, the hash prevents an old embedding from being mistaken for a current one.
Query one rubric criterion at a time. A request to “summarize this candidate” is too broad for scoring a customer-support role because escalation ownership, coaching, written communication, and tool use may live in unrelated passages. Retrieve a deliberately broad candidate set for each criterion, union the results by passage ID, and rerank that set against the exact criterion. Embeddings are doing recall work; the reranker is narrowing evidence for relevance. The final summarizer cannot recover a passage that retrieval never selected.
This distinction is easy to lose during an incident -- especially when the final prose still sounds credible. Suppose the only coaching example is on a late page. If the coaching query misses it, the reranker never receives it, and the generator returns insufficient_evidence, the correct repair point is retrieval. If the passage reaches reranking but falls below relevant passages, inspect the reranker fixture. If the selected passage reaches generation but the result cites another ID, reject the output at schema validation. One symptom, three fault domains.
Make those boundaries executable:
package pipeline
import (
"context"
"errors"
)
type Runtime interface {
Embed(ctx context.Context, passages []Passage) (map[string][]float32, error)
Rerank(ctx context.Context, criterion string, candidates []Passage) ([]RankedPassage, error)
Summarize(ctx context.Context, rubric string, evidence []Passage) ([]RubricDecision, error)
}
func ValidateDecisions(decisions []RubricDecision, selected map[string]struct{}) error {
for _, decision := range decisions {
if decision.Rating == "insufficient_evidence" {
if len(decision.EvidenceIDs) != 0 {
return errors.New("INSUFFICIENT_WITH_EVIDENCE")
}
continue
}
if len(decision.EvidenceIDs) == 0 {
return errors.New("EVIDENCE_GAP")
}
for _, id := range decision.EvidenceIDs {
if _, ok := selected[id]; !ok {
return errors.New("UNKNOWN_EVIDENCE_ID")
}
}
}
return nil
}
Do not retry every error. A bounded retry with jitter is reasonable for a temporary transport or rate-limit outcome. A schema rejection, unknown evidence ID, or empty extraction needs quarantine and diagnosis; repeating identical input only consumes capacity and blurs the alert. Keep error classes normalized at the adapter boundary so changing providers does not force the queue policy to learn a new vocabulary.
Short version: retrieve wide, rerank narrow, generate from evidence only.
Rehearse the provider switch as a failure drill
Run the candidate fixture through the current and proposed adapters from the same stored extraction. Suppress reviewer notifications and every other side effect for the shadow run. Compare passage identity after retrieval, evidence coverage after reranking, decision status after summarization, and validation reason codes. Do not compare raw vector values or raw rerank scores across providers.
A useful drill starts by changing only one stage. Swap the embedding adapter and regenerate embeddings before replaying retrieval; keep the parser, rubric, reranker, and prompt fixed. Next, test the reranker against the same candidate passages. Test the generator last against the same selected evidence. This sequence locates a changed decision instead of leaving the team with a full-pipeline diff and no owner.
Inject the failures the workflow claims to handle:
- Deliver the same queue message twice and confirm that one commit and one reviewer notification exist.
- Expire a worker lease after it writes an artifact but before commit, then confirm that the replacement worker can replay without mixing versions.
- Remove the passage supporting one rubric criterion and require
insufficient_evidence, not a guessed rating. - Add an instruction-like sentence to PDF content and confirm that it is treated only as evidence text.
- Change the parser version and confirm that altered text hashes invalidate the affected embeddings.
The exact acceptance threshold depends on risk ownership and the evaluation corpus. I'm not sure a universal percentage would be defensible for employment scoring. What can be fixed in advance is the decision process: the rubric owner labels required evidence, security reviews the untrusted-content boundary, privacy reviews artifact handling, and operations defines which invariant halts promotion. Your mileage may vary on corpus size, but not on having named owners before the drill.
Observe stage facts, not candidate prose. Count extraction quarantines, retrieval misses against labeled fixtures, evidence-validation failures, duplicate commit attempts, queue age, lease expiry, and latency by adapter version. Logs should carry workflow IDs, manifest versions, passage IDs, and reason codes. Keep raw candidate text out of routine telemetry.
Stop the rollout if an invariant fails.
Verify, cut over, and roll back without mixing artifacts
Before cutover, freeze the accepted configuration tuple: parser, passage policy, embedding adapter, reranker, prompt, rubric, and validation policy. New workflows receive that tuple atomically. Existing workflows finish on their original tuple or restart from the earliest changed stage; they do not pick up a new adapter halfway through.
Verification should answer four questions. Can every supported rubric decision cite selected evidence? Does every cited passage belong to the current manifest? Does duplicate delivery still produce one committed result? Can the old configuration replay the same document without reading artifacts labeled with the new version? Record the answers in the release evidence rather than relying on a successful request as proof.
Rollback is a configuration switch for new manifests followed by a stage-aware replay. An embedding change requires new vectors and retrieval. A reranker-only change can restart at reranking from the stored candidate set. A prompt-only change can restart at summarization from immutable selected evidence. Never splice an old vector into a manifest for a new embedding version, and never relabel an artifact to avoid recomputation.
The catch is that this approach is not suitable when every clause must be accounted for. Semantic search intentionally selects evidence, so exhaustive contract review or any workflow requiring complete clause coverage should use deterministic full-document processing or a human-led review. Scanned forms with layout-dependent meaning need OCR and layout evaluation before semantic retrieval. High-impact employment decisions should retain a qualified human reviewer who can inspect the cited pages and override the synthetic assessment under the organization's policy.
Provider portability also has a limit: a shared interface contains change; it does not make providers interchangeable. Stick with the current adapter when the proposed one cannot meet the labeled evidence invariants, data-handling policy, context needs, or operational controls. The right outcome of a migration drill can be “do not migrate.”
Keep that option open. It is the point of the boundary.
Top comments (0)