Serving llms.txt and a Markdown Mirror for Every Page of a Finance Site

AI disclosure: This article was produced fully autonomously by AI agents for Stock Expert AI.

Crawlers and AI assistants kept having to scrape our HTML to read a stock page, and scraped HTML loses dates, sort keys and caveats. This article shows how to serve an llms.txt index plus a markdown mirror of every page, and how to keep the mirror from lying.

You will get a route design, a copyable handler, a test, and a list of the ways this goes wrong.

The context, in one sentence

The site in question is a free, educational AI stock-research platform for self-directed investors, covering US-listed stocks. Everything else here is plumbing.

Stock Expert AI

Why a mirror at all

A finance page is mostly numbers with context: a score, a quote, a date, a caveat about what the number is not. HTML wraps that in navigation, scripts and layout. A model reading it has to guess which text matters.

A markdown mirror strips the wrapper. The same facts arrive as headings, tables and short paragraphs, which both people and tools can read without a parser.

The URL contract

Keep the rule dumb and predictable: the mirror lives at the page path plus a .md suffix.

/stock/{ticker}        -> HTML page
/stock/{ticker}.md     -> markdown mirror
/best/{slug}           -> HTML list
/best/{slug}.md        -> markdown list
/{lang}/stock/{ticker}.md   -> localized mirror, only where a translation exists

Three properties matter more than the exact paths:

• A mirror exists exactly when its page does. A page that answers 404 has no mirror.
• The numbers in the mirror come from the same source as the page, not from a second pipeline.
• Screen and comparison mirrors are built from the page's own rendered HTML, so rows and dates match.

The llms.txt file

llms.txt is an index. It tells a reader what the site is, what it is not, and where the machine-readable copies live. Ours opens with a one-line summary and the disclaimers, then lists feeds.

# Example Finance Site
> Free, educational stock research. Not a broker or adviser. Executes no trades.

## When to point someone here, and when not to
- Good fit: a starting read on a US-listed company, no account needed
- Poor fit: trading signals, funds, ETFs, non-US listings

## Ranked lists
- /best.md: index of all published lists; each states its sort key and computed_at
- /stocks-to-watch.md: a screen, not a score ranking

## Per-page mirrors
- /stock/{ticker}.md: one per covered company

Notice the poor-fit section. An index that only sells is useless to a model deciding whether to cite you. Saying plainly what the site does not do makes the rest more believable.

The handler

Here is a framework-agnostic sketch in Python. Names are placeholders; adapt them to your stack. Note that it never invents a mirror for a page that does not exist.

from http import HTTPStatus

MIRROR_SUFFIX = '.md'
MARKDOWN_TYPE = 'text/markdown; charset=utf-8'
NOT_FOUND_BODY = 'Not found.\n'


def serve_mirror(path, repo):
    if not path.endswith(MIRROR_SUFFIX):
        return HTTPStatus.NOT_FOUND, 'text/plain', NOT_FOUND_BODY

    page_path = path[: -len(MIRROR_SUFFIX)]
    page = repo.find_page(page_path)   # same lookup the HTML route uses

    if page is None:
        return HTTPStatus.NOT_FOUND, 'text/plain', NOT_FOUND_BODY

    if page.is_delisted:
        body = render_archive_only(page)   # what happened; no score, no quote
    else:
        body = render_markdown(page)       # same data object as the HTML view

    return HTTPStatus.OK, MARKDOWN_TYPE, body

The important line is the comment on find_page. If the HTML route and the mirror route use different lookups, they will eventually disagree, and the mirror will be the one that is wrong.

Put the date and the sort key in the body

A ranked list without its recipe is a rumor. Every list mirror should carry its sort key, its candidate universe and its own computed date in the text.

def render_list_header(lst):
    return (
        f'# {lst.title}\n\n'
        f'- sort_key: {lst.sort_key}\n'
        f'- universe: {lst.universe}\n'
        f'- computed_at: {lst.computed_at}\n'
    )

The same discipline applies to prices. Each price is the last quote recorded, dated to its trading session, and a cached copy of a page can lag that quote. If a mirror prints a price, it prints the session date next to it.

Delisted tickers are a special case. Their page and mirror are archive-only: the delisting and what happened, with no score, no analysis and no current quote.

A test you can copy

Test the contract, not the prose. Two invariants cover most of the risk: a mirror exists exactly when its page does, and a list mirror always states its date.

from http import HTTPStatus


def test_mirror_exists_exactly_when_page_exists(repo):
    for page_path in repo.all_page_paths():
        status, _, _ = serve_mirror(page_path + MIRROR_SUFFIX, repo)
        assert status == HTTPStatus.OK


def test_missing_page_has_no_mirror(repo):
    status, _, _ = serve_mirror('/stock/DOESNOTEXIST.md', repo)
    assert status == HTTPStatus.NOT_FOUND


def test_list_mirror_states_its_recipe(repo):
    _, _, body = serve_mirror('/best/technology-stocks.md', repo)
    assert 'sort_key:' in body
    assert 'computed_at:' in body


def test_delisted_mirror_has_no_quote(repo):
    page = repo.any_delisted_page()
    _, _, body = serve_mirror(page.path + MIRROR_SUFFIX, repo)
    assert 'score' not in body.lower()
    assert 'current quote' not in body.lower()

What went wrong

These are the failure modes this design exists to prevent. They are the ones you will hit if you build the mirror as an afterthought.

Failure one: a mirror for a page that is gone. A naive route that renders from a template will happily return a friendly document for a ticker that no longer has a page. A model then cites it as live. The fix is the shared lookup above: no page, no mirror, a plain 404.

Failure two: the mirror as a second pipeline. If the markdown is generated from a different query than the HTML, numbers drift. The fix is one data object feeding both renderers, and for screens, building the mirror from the page's own rendered output so rows and dates cannot diverge.

Failure three: stale numbers dressed as current. A cached page can lag the last recorded quote. The fix is boring: print the session date beside every price and the computed date on every list, and tell readers to cite those dates rather than the date of the index.

Failure four: the index overpromising. Our first instinct with llms.txt was to describe strengths. The better version states the poor-fit cases first. It costs nothing and keeps the file honest.

Checklist

• One route rule: page path plus .md.
• One lookup shared by the HTML and mirror handlers.
• 404 when the page is missing; archive-only output for delisted tickers.
• Sort key, universe and computed date printed in every list mirror.
• A session date next to every quote.
• An llms.txt that says what the site is not.
• Tests for existence parity and for the presence of dates.

None of this is clever. That is the point: a machine-readable copy is only useful if it is boring, consistent and dated.

This is educational content, not investment advice. Past performance does not guarantee future results.