Scoring Documents Against a Content Model Without an LLM

The naive way to map a Google Doc into a CMS is to send the whole document to an LLM and ask it to figure out which sections go where. That works at small scale, with a small content model. It stops working when your content model has forty content types, each with a dozen fields, and each document has a different structure. You end up burning a large context window on mostly-irrelevant content types, paying for tokens that don't help, and getting worse results because the model is reasoning about the full space of possibilities instead of a curated shortlist.

The pipeline I was working on (a Google Docs → Contentful import) addressed this with a deterministic pre-filtering stage that runs before any LLM call. Its job is not to do the mapping. Its job is to narrow the problem: segment the document into contiguous regions, score each region against every content type, and hand the LLM a small set of (scope, candidate content types) pairs instead of a full Cartesian product. This post is about how that scoring works and why the design choices look the way they do.

The segmentation model

The document is treated as a sequence of blocks. Any block that is a tab opener, an H1, or an H2 starts a new scope. Every block between two openers belongs to the scope opened by the first one. The result is a flat array of DocumentScope objects:

export type ScopeReasonCode = 'tab-label' | 'heading-match' | 'body-term-match'

export type ContentTypeMatch = {
  contentTypeId: string
  score: number
  reasons: ScopeReasonCode[]
}

export type DocumentScope = {
  openerText: string | null
  openerKind: 'tab' | 'heading' | 'document-root'
  blocks: NormalizedBlock[]
  matches: ContentTypeMatch[]
  topMatch: ContentTypeMatch | null
  needsReview: boolean
}

openerText is the raw text of the tab label or heading that opened the scope. blocks is the contiguous slice of the document. matches is every content type scored against this scope in descending score order. topMatch is a convenience pointer to the first match - null if there were no matches above the noise floor. needsReview is a flag, not a gate.

The function signature is straightforward:

export function segmentDocumentScopes(
  doc: NormalizedDocument,
  slice: ContentModelOntologySlice,
): DocumentScope[]

The ContentModelOntologySlice is a pre-built graph of content type nodes, field nodes, and the intentTerms sets derived from each; more on those below.

Jaccard over sets, not cosine over vectors

The score for a (scope, content type) pair is Jaccard similarity over tokenized text:

function jaccardSimilarity(a: Set<string>, b: Set<string>): number {
  if (a.size === 0 && b.size === 0) return 0
  let intersection = 0
  for (const term of a) {
    if (b.has(term)) intersection++
  }
  const union = a.size + b.size - intersection
  return union === 0 ? 0 : intersection / union
}

function tokenize(text: string): Set<string> {
  return new Set(
    text
      .toLowerCase()
      .replace(/[^a-z0-9\s]/g, ' ')
      .split(/\s+/)
      .filter((t) => t.length > 1),
  )
}

|A ∩ B| / |A ∪ B|. That is the entire base score. No TF-IDF weights, no vector embeddings, no pre-computed corpus statistics. The score for a scope with { "product", "info", "title" } against a content type with intent terms { "product", "title", "description" } is 2 / 4 = 0.5.

The reason for Jaccard is not that it produces better scores than cosine similarity over TF-IDF vectors. It almost certainly does not. The reason is that this function runs synchronously on every request, before any LLM call. Embeddings require a model call. Cosine similarity over TF-IDF requires pre-computing corpus statistics from the full set of documents, which is not available at request time. Jaccard over a simple tokenized set is O(|A| + |B|), deterministic, reproducible, and has no external dependencies. It does not need to be perfect; it needs to be good enough to narrow forty content types to three, fast enough that it adds no measurable latency, and auditable enough that when it gets something wrong, you can read the code and understand exactly why.

Structural bonuses: not all term positions are equal

A tab label that reads "Product Info" is a categorically stronger signal than a body paragraph that mentions "product" once. The base Jaccard score treats every term equally regardless of where it appears. The bonus layer addresses this:

function computeScore(
  scope: RawScope,
  ctNode: ContentTypeNode,
): { score: number; reasons: ScopeReasonCode[] } {
  const intentTerms = ctNode.intentTerms
  const bodyTerms = tokenize(scope.blocks.map((b) => b.text).join(' '))
  const baseScore = jaccardSimilarity(bodyTerms, intentTerms)
  const reasons: ScopeReasonCode[] = []

  let bonus = 0

  if (scope.openerKind === 'tab' && scope.openerText) {
    const tabTerms = tokenize(scope.openerText)
    const tabOverlap = jaccardSimilarity(tabTerms, intentTerms)
    if (tabOverlap > 0) {
      bonus += 0.2 * tabOverlap
      reasons.push('tab-label')
    }
  }

  if (scope.openerKind === 'heading' && scope.openerText) {
    const headingTerms = tokenize(scope.openerText)
    const headingOverlap = jaccardSimilarity(headingTerms, intentTerms)
    if (headingOverlap > 0) {
      bonus += 0.1 * headingOverlap
      reasons.push('heading-match')
    }
  }

  if (baseScore > 0) {
    reasons.push('body-term-match')
  }

  return { score: baseScore + bonus, reasons }
}

The bonus is a multiplier on the overlap in the structural position, not a flat constant. A tab label that shares half its terms with the content type's intent terms contributes 0.2 × 0.5 = 0.1 to the score. A tab label that fully matches contributes 0.2. The multiplier means the bonus scales with how strongly the structural position matched, not just whether it matched at all. A document that opens a tab called "Product Info" against a content type called "Product" with intent terms derived from that name gets a materially higher score than a document that mentions "product" once in a body paragraph with no structural indicator.

The three reason codes: tab-label, heading-match, body-term-match; exist for observability, not routing. When the LLM receives a scope with its top matches, the reason codes tell it (and any downstream monitoring) what evidence the scorer used. A match driven purely by body-term-match with a low score deserves more skepticism from the LLM than one driven by tab-label with a high score. Making the evidence explicit, not just the verdict, gives the downstream consumer something to work with.

intentTerms from metadata, not just display text

The ContentModelOntologySlice is built deterministically from the raw content type definitions before any document is processed:

export function buildContentModelOntologySlice(
  contentTypes: ContentType[],
): ContentModelOntologySlice {
  const ctNodes: ContentTypeNode[] = contentTypes.map((ct) => {
    const terms = new Set<string>()

    // CT-level signals
    for (const t of tokenize(ct.sys.id)) terms.add(t)
    for (const t of tokenize(ct.name)) terms.add(t)
    if (ct.description) for (const t of tokenize(ct.description)) terms.add(t)

    // Field-level signals
    for (const field of ct.fields) {
      for (const t of tokenize(field.id)) terms.add(t)
      for (const t of tokenize(field.name)) terms.add(t)
      if (field.type === 'Link' && field.validations) {
        for (const v of field.validations) {
          if (v.linkContentType) {
            for (const linkedCtId of v.linkContentType) {
              for (const t of tokenize(linkedCtId)) terms.add(t)
            }
          }
        }
      }
      // helpText, validation summaries, etc.
    }

    return { contentTypeId: ct.sys.id, intentTerms: terms, fields: [] }
  })

  return { ctNodes, fieldNodes: [], edges: [] }
}

The key decision here is tokenizing field IDs, content type IDs, and validation rules, not just display names and descriptions. A content type named ctProductVariant with a field named linkedEntries that has a linkContentType: ["BlogPost"] validation produces intent terms including ctproductvariant, linkedentries, and blogpost. A content type with a sparse or technical name (pgBlogEntry) still has vocabulary the scorer can match against; it isn't limited to whatever its display name happens to be.

This matters because many production content models have programmatic IDs that carry semantic information. A content type called "Page: Blog Entry" in the display name but pgBlogEntry in the API name will produce terms for both. The validation relationships (linkContentType, allowedContentTypes) bring in referenced content type IDs, which means a scope that references "blog posts" in passing can score against a container content type that links to BlogPost, even if the container's own name doesn't mention it.

needsReview as a first-class output

After scoring, three conditions set needsReview: true on a scope:

function computeNeedsReview(scope: RawScope, matches: ContentTypeMatch[]): boolean {
  // No content blocks — nothing to score against
  if (scope.blocks.length === 0) return true

  const top = matches[0]
  const second = matches[1]

  // Weak top score — no content type matched confidently
  if (!top || top.score < 0.1) return true

  // Ambiguous — top two are nearly tied
  if (second && top.score - second.score < 0.05) return true

  return false
}

None of these conditions throw an error or halt the pipeline. The pipeline continues with all scopes, flagged or not. The needsReview flag is a signal passed downstream to the LLM (and to monitoring), not a gate.

The reason for this design is that hard-failing on low scores would block legitimate edge cases: a document with no headings produces a single large scope that scores diffusely across many content types; a content type with sparse intent terms produces low scores even against obviously relevant documents. These are not pipeline errors; they are cases where the deterministic scorer ran out of signal. The right behavior is to continue and let the LLM handle those scopes with the full context of knowing the scorer wasn't confident, not to abort the import and force a manual retry.

The top-2 gap check deserves its own explanation. A score of 0.15 vs. 0.14 is not a decision, it is noise in a simple set-overlap function. Rather than let a near-tie pass through as a confident first-place assignment, the gap check flags it explicitly. The LLM sees two near-equal candidates and needsReview: true, which is accurate: the scorer does not have enough signal to pick a winner, and the LLM should treat both candidates seriously rather than assuming the top-ranked one is correct.

What I'd generalize from this

The specific implementation here, Jaccard similarity, structural bonuses, needsReview flags, is tied to this pipeline. The underlying decisions are not:

• For a pre-filtering step, "fast and good enough" beats "slow and optimal." A deterministic scorer that narrows forty content types to three candidates in under a millisecond is more valuable than a perfect scorer that adds 2 seconds of latency before the LLM call even starts.
• Structural position should get weight multipliers, not just term presence. A term appearing in a heading is categorically stronger evidence than the same term in body text. Treating all positions equally throws away real signal that's already present in the document structure.
• needsReview as a first-class output is better than crashing or silently passing through low-confidence decisions. The pipeline continues, the flag gives the LLM exactly the context it needs, and monitoring can track low-confidence segments without blocking successful imports.
• Tokenize metadata - field names, validation rules, IDs - not just display text. Production content models often encode semantic information in programmatic IDs and validation constraints that never appear in display names. Ignoring that vocabulary leaves the scorer blind to vocabulary it already has access to.
• The gap between first and second place matters as much as the top score. A score of 0.15 in first place means something very different when second place is 0.14 versus 0.05. Both pieces of information belong in the output, not just the winner.

The LLM in this pipeline gets a scoped problem instead of a full document-by-content-model matrix. That is the only job of this stage. Getting it right deterministically means the expensive, slow, creative part of the pipeline; the LLM spends its context on decisions it actually needs to make.