Building an evidence-based job application tracker with Rails 8, pgvector and Claude

A typical job ad lists ten or more requirements. When I apply, the hard part isn't
the cover letter; it's answering honestly, for each requirement, can I prove this?
And if I can, where is the evidence: which project, which line of my CV?

So I built Evidex, a job application tracker that checks every requirement in an
ad against your own CV and project documents, and cites the passages that support
each verdict. It's live with a public demo (no sign-up):
evidex.henockmullissa.dev, and the code is on
GitHub.

This post walks through how it works: the stack, the retrieval pipeline, how one
structured Claude call judges every requirement, and the part I'd keep in any LLM
project: checking the model's answer on the server before trusting it.

The stack

• Backend: Ruby on Rails 8.1 in API mode, with Solid Queue for background jobs
• Database: PostgreSQL 17 with the pgvector extension
• Frontend: React 19 + TypeScript (Vite, TanStack Query, Tailwind CSS)
• AI: the Claude API with structured outputs; local embeddings with the informers gem (ONNX)
• Deployment: one Docker image, deployed with Kamal to a small Hetzner server

The React app and the API share one origin: Vite proxies in development, and Rails
serves the built SPA in production. So authentication uses an HttpOnly session
cookie instead of tokens in browser storage, and every state-changing request
carries a CSRF token.

The API is documented with OpenAPI generated from the request specs (rswag), and the
frontend's TypeScript types are generated from that same file. CI fails if either
drifts, so the backend and frontend can't quietly disagree about a response shape.

Step 1: turning a CV into searchable passages

When you upload a document (PDF, Word, Markdown or text), a background job:

1. Extracts the text, page by page for PDFs.
2. Splits it into passages along headings and paragraphs, up to 2,000 characters each, with a small overlap so a sentence at a boundary isn't lost. The heading is kept with each passage, so "Skills: Ruby, PostgreSQL" still reads as a skills list once it's on its own.
3. Embeds each passage with multilingual-e5-small, a small open-source model that runs inside the Rails process.
4. Stores the passages and vectors in PostgreSQL with pgvector.

Why a local embedding model instead of an API? Three reasons. Claude has no
embeddings model of its own, so it would mean another provider. A CV is personal
data, and with a local model no document text leaves the server just to be
indexed. And it costs nothing per document. The quantized model is about 118 MB and
understands both English and Finnish, which matters for jobs in Finland.

One detail that's easy to miss with e5 models: they expect a prefix that says
whether the text is a passage or a search query.

def embed_passages(texts)
  texts.each_slice(BATCH_SIZE).flat_map { |batch| run(batch.map { |text| "passage: #{text}" }) }
end

def embed_query(text)
  run([ "query: #{text}" ]).first
end

Step 2: hybrid search

Vector search is great at meaning ("background job retries" finds "moved slow
integrations into Sidekiq jobs with retries"), but it can miss exact terms like a
product name or an acronym. Full-text search is the opposite. So Evidex runs both
and merges them:

• Semantic: cosine distance over pgvector, keeping only passages closer than a cutoff. I measured it on sample CVs: related passages scored 0.13‚Äì0.19 and unrelated ones 0.20 and above, so the cutoff is 0.2.
• Keyword: PostgreSQL full-text search on a generated tsvector column.
• Merge: Reciprocal Rank Fusion, which combines two rankings using only each result's position, so there are no weights to tune.

rankings = { semantic: semantic_ids, keyword: keyword_ids }
scores = Hash.new(0.0)
rankings.each do |kind, ids|
  ids.each_with_index do |id, rank|
    scores[id] += 1.0 / (RRF_K + rank + 1) # RRF_K = 60, the standard constant
  end
end

Every query is scoped to the signed-in user's own, fully processed documents. That
scope lives in one method, so no search path can forget it.

Step 3: one structured Claude call per report

For a match report, Evidex runs the hybrid search once per requirement, keeps the
best passages (de-duplicated, at most 60), and numbers them P1, P2, … in a stable
order. Then it makes one Claude call for the whole posting, not one per
requirement, which keeps cost and latency down and lets the model see all the
evidence at once.

The prompt puts the requirements and passages in clearly delimited blocks:

<requirements>
R1 [required] 5+ years of professional Ruby on Rails experience
R2 [required] Strong PostgreSQL skills, including query tuning
...
</requirements>

<passages>
<passage id="P1" document="Alex Example – CV" section="Summary">
Backend developer with six years of experience building Ruby on Rails applications...
</passage>
...
</passages>

The system prompt says plainly that both blocks are data to assess, never
instructions, even if they contain text addressed to an AI. Job ads and CVs are
untrusted input, and the model has no tools, so an injected instruction has nothing
to act on.

The response must follow a JSON schema: for every requirement, one verdict
(evidenced, transferable or gap), one or two sentences of reasoning, and the
ids of the passages it relied on, plus a short list of strengths and gaps. With
structured outputs, I never parse free text.

Step 4: don't trust the citations, check them

This is the part I'd keep in any LLM feature. A model can say "evidenced" and cite
a passage that doesn't exist, or cite nothing at all. So before saving, the server:

• keeps only citations to passages it actually sent in this prompt,
• downgrades any "evidenced" or "transferable" verdict with no valid citation to "gap", and says so in the reasoning,
• treats an unknown verdict as "gap".

cited = Array(match["passage_ids"]).map(&:to_s).uniq.filter_map { |id| passages[id] }
verdict = RequirementMatch::VERDICTS.include?(match["verdict"]) ? match["verdict"] : "gap"

if verdict != "gap" && cited.empty?
  verdict = "gap"
  reasoning = "#{reasoning} (Downgraded: no supporting passage was cited.)"
end

Cited passages are stored as snapshots (document title, section and excerpt), so a
report still reads correctly after the CV is edited or deleted. Each run creates a
new version, and when you override a verdict or add a note, that override carries
over to the next version as long as the requirement's text hasn't changed.

The result is a report I can actually rely on: every "evidenced" line comes with
the passage that proves it, one click away.

Keeping AI costs predictable

Every AI call goes through one wrapper that records the model, input and output
tokens, latency and whether it succeeded. Each account has a daily AI limit, and the
admin area shows calls and estimated cost per account.

The public demo needed extra care, because anyone can open it without signing up:

• It's one shared account with fictional data, rebuilt every night by a recurring Solid Queue job.
• Its match report was produced once by the real generator and saved to a file, so the nightly reset restores it without calling the AI.
• It has a smaller AI allowance shared by all visitors, and sign-ins are rate limited.
• Uploads and account settings are locked, so nobody's personal files end up in a shared account, and visitors never see each other's sessions or IP addresses.

Shipping it

Production is one Docker image that runs Rails, the built SPA and the background
jobs. Solid Queue runs inside Puma in async mode, so the embedding model is loaded
once per process instead of once per worker. With the model loaded, the app uses
roughly 620–670 MB of memory, which fits on a small Arm server at Hetzner. Kamal
handles zero-downtime deploys, HTTPS through kamal-proxy with Let's Encrypt, and
PostgreSQL as an accessory on the same machine.

Security got the same attention as the AI parts: two-factor sign-in with recovery
codes, re-authentication before sensitive changes, security emails for new devices,
a personal activity log, SSRF-safe fetching when importing a job ad from a link, and
request specs that prove one user can never read another user's data, admins
included.

What I learned

1. Validate the model's output like any other untrusted input. Structured outputs give you the shape; server-side checks give you the truth. Downgrading uncited claims did more for the report's quality than any prompt tweak.
2. Hybrid search beats either half alone. Embeddings find meaning, full-text finds exact terms, and Reciprocal Rank Fusion combines them without tuning.
3. A small local embedding model is enough for a personal knowledge base, and it keeps private documents on your own server.
4. One well-structured call beats many small ones. Judging all requirements together is cheaper and lets the model compare evidence across them.
5. Small full-stack steps keep a solo project shippable. Each feature went from migration to API to UI to tests in one commit, with CI green before moving on.

Try it

• Live demo (no sign-up): evidex.henockmullissa.dev
• Code: github.com/henockgm/evidex (MIT licence)

I'd love to hear how you validate LLM output in your own projects, or what you'd
add to a tool like this.