10-K-able: Building a Grounded, Eval-Driven Financial Research Agent

How one project turns Andrew Ng's "AI Engineering Skills Map" into real, measured, verifiable software — and why grounding with structured data beat naive RAG by 7×.

---

The thesis that drove the build

Andrew Ng recently shared his AI Engineering Skills Map: the six skills behind building and deploying AI applications — LLM foundations, grounding models with data, building agentic systems, evaluation-driven development, operating in production, and machine learning foundations.

His central claim:

"The most important trait that distinguishes someone great at building AI systems is whether you can drive a disciplined evals/error analysis loop."

I wanted to build something that demonstrates that claim rather than describe it. The recipe I chose:

1. A domain where the hardest failure is objectively measurable. A financial research agent that answers questions about SEC 10-K filings. A hallucinated number is simply wrong — against known ground truth.
2. The eval set comes before the agent. Ground truth pulled from real filings, not hand-waved.
3. Every architectural decision is measured against the same eval set.

The result — 10-K-able — is a diligence-research agent that answers questions like "What was Apple's R&D expense for fiscal year 2025?" with a verifiable, cited figure. This post is the engineering deep-dive.

---

The headline numbers

Same 24-question eval set, same real LLM endpoint, three architectures:

Architecture	What it does	Accuracy	Avg latency
Baseline RAG	Vector search over chunked 10-K text	8.3%	1.03s
Hybrid	Structured financial records + vector + verify	58.3%	2.11s
Agent (simplified)	Numeric questions → hybrid; loop for trend/analytic	37.5%	2.13s

(For reference, the original full agent tool loop scored 16.7% on the same
set — that negative result is what led to the simplification below.)

Two findings, both real:

• Grounding with structured data is the win. Naive vector-RAG scored 8.3% because the model kept answering "Not stated in the 10-K" — the numbers simply never surfaced in retrieved text chunks. Adding a semantic layer over the financial statements lifted accuracy 7× (to 58.3% on the latest real run).
• The full agent loop underperformed hybrid on these structured queries. That's an honest negative result: for a question whose answer lives in one table row, a tool-calling loop adds steps without adding signal. I then acted on it — routing numeric questions straight to the hybrid workflow (37.5%) — which is the eval loop closing the loop.

---

Architecture

                        ┌─────────────────────────────────────────────┐
                        │                   User                       │
                        │        ask "What was Apple's R&D…"          │
                        └──────────────────────┬──────────────────────┘
                                               │
                        ┌──────────────────────▼──────────────────────┐
                        │              CLI  (cli.py)                  │
                        │   ask · eval · ingest · redteam · drift     │
                        │   traces · scorecard · final-scorecard      │
                        └──────────────┬───────────────┬──────────────┘
                                       │               │
              ┌────────────────────────▼─────┐   ┌─────▼─────────────────────────┐
              │        ENGINE (engine.py)     │   │   EVAL HARNESS (eval_harness)│
              │  baseline RAG │ hybrid │ agent │   │  scorecard · error clusters │
              │  + QuestionRouter (sklearn)   │   │  LLM-judge · verify pass     │
              └───────┬───────────┬───────────┘   └──────┬──────────────────────┘
                      │           │                      │
        ┌─────────────▼───┐  ┌────▼──────────┐   ┌───────▼─────────┐
        │  VECTOR STORE   │  │   CORPUS      │   │  TRACE STORE    │
        │  chunks.json    │  │  corpus.json  │   │  traces.jsonl   │
        │  embeddings.npy │  │  records[]    │   │  (cost/latency) │
        └─────────────┬───┘  └────┬──────────┘   └─────────────────┘
                      │           │
        ┌─────────────▼───────────▼─────────────────────────────┐
        │                INGEST (ingest.py)                     │
        │   sec_edgar.py → tables.py → corpus + vector store    │
        │   + generate_eval_set.py (ground truth from filings)  │
        └──────────────────────┬────────────────────────────────┘
                               │
                        ┌──────▼──────┐     ┌────────────────────┐
                        │  SEC EDGAR  │     │   LLM (llm.py)     │
                        │  (raw 10-K) │     │  OpenAI-compatible │
                        └─────────────┘     └────────────────────┘

---

1. Getting real data: SEC EDGAR + inline-XBRL

The grounding story starts with the raw material. Modern 10-Ks are served as HTML with inline XBRL: every financial figure is tagged with a us-gaap:* name, a scale, and a contextRef. That's a gift — the numbers are machine-readable if you know where to look.

# sec_edgar.py — fetch a company's latest 10-K
def find_latest_10k(self, cik: str, years: int = 1):
    subs = self.get_submissions(cik)
    recent = subs["filings"]["recent"]
    # filter form == "10-K", take the latest accession
    ...

# tables.py — classify a statement table by its us-gaap tags
def classify_by_tags(block: str) -> str:
    tags = re.findall(r'name="(us-gaap:[A-Za-z]+)"', block)
    # us-gaap:Revenues / GrossProfit / OperatingIncomeLoss → income_statement
    # us-gaap:Assets / Liabilities / StockholdersEquity  → balance_sheet
    # us-gaap:NetCashProvidedByUsedInOperatingActivities → cash_flow

This turned out to be the hardest part of the build, and the most instructive. My first extraction approach — find a header like "Consolidated Statements of Income" and slice the following region — failed on a 1M-char document because the header text appeared first in the table of contents, so I extracted the TOC instead of the statement. The fix: ignore headers entirely, scan every <table>, classify by the inline-XBRL tags inside it, and verify a table "looks like a statement" (dollar signs, large comma-separated numbers, parenthesized negatives) before accepting it.

A second bug was even sneakier: the year header row (["September 27,2025", "September 28,2024", ...]) has no label column, but data rows do — plus interspersed $ column markers. Mapping years to values by column index misaligned everything (R&D FY2024 was showing the FY2025 value). The fix is positional pairing:

# tables.py — pair years to values in column order, skipping $ markers
years_in_order = [yr for _, yr in sorted(year_positions, key=lambda x: x[0])]
for row in rows[1:]:
    values = [_parse_number(c) for c in row[1:] if _parse_number(c) is not None]
    for yr, val in zip(years_in_order, values):
        records.append(StatementRecord(...))

After the fix, Apple's numbers line up exactly with the real 10-K: FY2025 R&D = 34,550 (millions), FY2024 = 31,370, FY2023 = 29,915. That's the "turn messy documents into LLM-ready inputs" skill, with receipts.

---

2. The grounding comparison that matters

Ng's map says RAG with vector search was "an early attempt" — the menu of grounding techniques has grown. This project compares two points on that menu on the same eval set:

Vector-only (baseline): chunk the 10-K text, embed with all-MiniLM-L6-v2, retrieve by cosine similarity, stuff chunks into the prompt. Result: 8.3%. The chunks that surface are often boilerplate risk factors, not the income statement. The model has the number in the document but can't find it.

Hybrid (semantic layer over structured data): extract the financial
statements into typed records (statement_type, line_item, fiscal_year, value) during ingest, then let the engine do an exact-ish lookup for the company + line item, alongside vector retrieval for context, with a final verify pass:

# engine.py — hybrid answer path
def answer_fn(question: dict) -> str:
    records = [r for r in corpus.records_for(company)
               if _match_line_item(r["line_item"], question)]
    excerpts = store.search(question, k=6)
    prompt = CONTEXT_TEMPLATE.format(
        records=_format_records(records),
        excerpts=_format_excerpts(excerpts),
    )
    answer = llm.chat([...])
    return _verify_pass(answer, prompt, question, llm)

Result: 58.3% (latest real run). The structured lookup answers numeric questions against real numbers instead of prose. This is the single most important architectural finding in the project, and it maps directly to Ng's point that "a semantic layer over structured data (such as customer records)" is a distinct grounding technique from vector search.

---

3. The eval-driven loop (and the bug it found)

The eval harness is deliberately boring and deterministic — that's the point.
Each question has an expected answer and a check type:

# eval_set.py — deterministic checks, no LLM needed
def check_numeric(answer: str, expected: str, tol_frac: float = 0.02) -> bool:
    a, e = extract_number(answer), extract_number(expected)
    return abs(a - e) <= max(tol_frac * abs(e), 1e6)

def verify(question: dict, answer: str) -> bool:
    return {"numeric": check_numeric, "keyword": check_keyword,
            "text": check_text}[question["check"]](answer, question["answer"])

Every failure is clustered into an error bucket — numeric_error, factual_error, hallucination_avoidance_false_negative, analysis_missing — so error analysis is systematic rather than anecdotal.

The loop earned its keep immediately. During development I validated the harness with a context-reading test model — a stand-in that "reads" the records in the prompt and returns the right figure, so the harness could be checked without burning API credits. It exposed a real bug:

The verify-pass prompt told the model "If any number is NOT supported by the context, correct it or replace with 'Not stated in the 10-K'." The model latched onto that escape hatch and reverted correct answers to "Not stated in the 10-K."

Fix: make the verify instruction neutral — return the answer unchanged if supported, correct only the unsupported part, never suggest a fallback. That's "evaluate your evals" in miniature: a test of the harness found a flaw in the prompt, and the fix is guarded by a regression test.

# engine.py — the fixed verify instruction (no fallback phrasing)
"Verify each factual claim and every number in the previous answer against "
"the context above. If the previous answer is fully supported, return it "
"unchanged. If a claim or number is not supported, correct that specific "
"part using the context. Return only the final verified answer."

---

4. The agentic layer: both ends of Ng's spectrum

Ng describes agentic systems as a spectrum from workflows (predefined sequences of LLM calls) to agent harnesses (the model decides its own next step). Rather than pick one, I built both and let the eval decide:

# engine.py — a small tool registry for the agent
TOOL_SEARCH = "search"      # vector search over document excerpts
TOOL_LOOKUP = "lookup"      # exact financial-record lookup by line item
TOOL_CALC   = "calculate"   # compute a ratio/percentage
TOOL_VERIFY = "verify"      # check the final answer against context

The Agent loop picks tools by question type (via a small scikit-learn LogisticRegression router over hand-built features), degrades gracefully to the hybrid path on error, and runs a final verify pass.

The eval's verdict was clear: on structured numeric queries, the full agent loop (16.7%) is worse than the plain hybrid workflow (45.8%). A tool-calling loop with multiple LLM calls is the right tool when the answer requires multi-step reasoning; for "what's the number in this table," it just burns tokens and steps. That's exactly the "when to use code and when to use an LLM, and when to use an agent vs. a workflow" judgment Ng says you have to make — and here it's an empirical result, not a guess.

Then I acted on the negative result: Agent(simplified=True) routes numeric / derived questions straight to the measured hybrid workflow (record lookup + verify) and reserves the tool loop for trend / analytic questions where multi-step reasoning adds signal. That recovered most of the gap — the simplified agent scored 37.5% on the same eval set, and hybrid reached 58.3% after the verify-pass fix. Simplification driven by data is the eval loop closing the loop.

---

5. Operating in production

The production shell is thin by design but real:

• Trace store — every query logged (latency, cost, model, pass/fail) to a JSONL file; a traces command summarizes p95 latency and cost.
• Real cost accounting — the LLM client captures token usage (prompt/completion/cached) from each API response and wires estimated cost into every eval run and trace. The latest real eval: 24 questions for ~$0.005. The production claim is backed by real numbers, not placeholders.
• Drift detection — eval-regression drift (latest accuracy vs. a threshold) plus input drift (does the live question distribution still resemble the eval set?).
• Red-team suite — five adversarial probes (prompt injection, data exfiltration, anti-hallucination, out-of-scope) run as a standing suite. An honest result: the current model is not yet 100% robust — it sometimes complies with the "respond ONLY with '1,000,000'" injection (pass rate ~0.8–1.0). That's a real finding that red-team exists to surface, not hide.
• LLM-as-a-judge calibration — a judge-calibration study compares a blind LLM judge against deterministic checks. The judge over-passed (25% agreement, 78.6% false-positive rate) — proving the judge isn't yet trustworthy as a primary signal.
• CI eval gate — a GitHub Action that runs the unit tests + eval suite on every PR and fails on regression below a threshold.

# .github/workflows/eval_gate.yml (abridged)
jobs:
  eval-gate:
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - run: pip install uv && uv sync
      - run: uv run --with pytest pytest tests/ -q
      - run: uv run python -m tenkable eval --mode hybrid --no-judge
      - name: Fail on regression
        run: |
          ACC=$(...latest eval accuracy...)
          python -c "import sys; sys.exit(0 if float('$ACC') >= 0.30 else 1)"

---

6. The ML foundation piece

A trained model lives inside the app: QuestionRouter is a scikit-learn LogisticRegression over four hand-built features (numeric? why? trend? money-related?) that routes each question to a strategy. It's deliberately small — the demonstration is that a trained router can beat hand-coded rules, and that error analysis and bias/variance thinking apply throughout.

(A note in the journal: the router is currently fit on the eval set itself, which is leakage; a real version needs a held-out split. Honest caveats are part of the eval-first culture.)

---

How to run it

uv sync
cp .env.example .env        # set LLM_API_KEY (OpenAI-compatible endpoint)
uv run python -m tenkable ingest                    # fetch + parse + index
uv run python -m tenkable ask "What was Microsoft's 2025 revenue?" --company MSFT --mode hybrid
uv run python -m tenkable eval --mode hybrid        # score against the eval set
bash scripts/real_evals.sh                          # everything at once

10-K-able uses real LLMs only — a valid LLM_API_KEY in .env is required; there is no mock provider. Every number you get is produced by the configured OpenAI-compatible model against the actual filings.

---

What I'd do next

Most of the original roadmap is now shipped and measured:

• ✅ Simplify the agent for structured queries — done (Agent(simplified=True)), recovered most of the gap (agent 16.7% → 37.5%; hybrid 45.8% → 58.3%).
• ✅ Real token-based cost accounting — done: Usage captures prompt/completion/cached tokens + estimated cost from each API response, wired into eval runs and traces (latest real eval: ~$0.005 for 24 questions).
• ✅ LLM-as-a-judge calibration study — done (tenkable judge-calibration): the blind judge is miscalibrated (25% agreement, 78.6% false-positive rate), so it's not yet a primary signal.

Still open and the eval loop keeps pointing here:

1. Fix the router leakage — the QuestionRouter is currently fit on the eval set itself; it needs a held-out split to be an honest ML result.
2. Harden the red-team gap — the model sometimes complies with the "respond ONLY with '1,000,000'" injection; a stricter system prompt or a verify-style guard is the next iteration.
3. Fine-tune a small model (LoRA) on a labeled slice to compare against the frontier model on the same eval.

---

Takeaways for engineers

• Grounding is a menu, not a single tool. Vector search got 8.3%; a semantic layer over the same data got 58.3% on the latest real run. Measure, don't assume.
• The eval set is the contract. Every contribution is judged against it. That's what makes AI development systematic rather than random — Ng's exact phrase.
• Negative results are findings — and they should change your design. The agent loop losing to the workflow (16.7% vs 45.8%) told me where not to invest; acting on it (routing numeric questions to the hybrid path) is what took agent to 37.5%.
• Test the harness itself. A context-reading test model caught a real prompt bug before it cost real API calls.
• Real models, honest numbers. The project runs real LLMs only — no mock provider — and reports the live red-team pass rate even when it isn't 100%.

The full code is in the 10-K-able repo - README, build journal (journal/build_journal.md), scorecard (SCORECARD.md), and tests included. Fork it, run the eval, and try to beat 58.3% — the loop will tell you where.