Serving HTML and Markdown from one public URL

An agent-readable page needs more than a different file extension. The response should make its representation clear, respect the client's preferences, and avoid changing the access rules of private routes.

The examples below use public endpoints from Giorgio, a project I work on. The aim is to make a small HTTP contract reproducible: one catalog endpoint, two representations of a public page, and explicit boundaries around private routes. No account or credential is needed for the public examples.

Start with a public catalog

The example catalog returns public identity and canonical page URLs. It requires no account or API key:

curl --fail https://giorgiobm.com/api/public/site

This endpoint describes public site metadata. It does not run inference, execute code, manage accounts or expose private conversations and files.

Here is a small Python reader using the standard library:

import json
from urllib.request import Request, urlopen

request = Request(
    "https://giorgiobm.com/api/public/site",
    headers={
        "Accept": "application/json",
        "User-Agent": "GiorgioPublicReader/1.0",
    },
)
with urlopen(request, timeout=20) as response:
    catalog = json.load(response)

print("Scope:", catalog["scope"])
for page in catalog["pages"][:5]:
    print(page["url"])

The next step is to read a relevant page. A directory of URLs is useful navigation, but it is not evidence for every claim someone might make about a product.

The example identifies its client with a descriptive User-Agent. In the live check, Python's default request was rejected with HTTP 403 by the edge layer, while the identified client returned the catalog. A public endpoint can still have anti-abuse filtering. Handle that status as a failure; do not assume every response body is JSON or retry around an access challenge.

Request Markdown when it is useful

Supported public pages can return a textual representation through content negotiation:

curl --fail \
  -H 'Accept: text/markdown' \
  https://giorgiobm.com/about

Check the response status and Content-Type before interpreting the body. The site still serves HTML to readers that prefer HTML, including when it has a higher quality value in the Accept header. The response varies on Accept so the two representations can be distinguished by caches.

Private chat and Code session routes are outside this public negotiation. The catalog's read-only contract also matters: a POST to it is not an alternate way to request an AI completion.

Keep the documents consistent

An OpenAPI document should describe the operations actually available. A sitemap should list canonical public URLs. Navigation summaries can help clients discover pages, but their existence does not prove that those pages are accessible or that a crawler has visited them.

There is no need to invent an authentication flow, advertise private operations or describe an inference SDK that the public endpoint does not provide. Small, accurate interfaces are easier to inspect and document.

Check behavior, not just file presence

Useful checks include the HTML and Markdown representations, canonical metadata, quality-weighted Accept negotiation, unknown-page errors, unsupported methods and exclusion of private routes. A file existing in a local checkout does not by itself establish that the hosted route serves it correctly; static asset priority can change the result.

These interfaces improve the accessibility of public information. They do not establish that a search engine has indexed every page, that an assistant will recommend the product, or that any model used the content for training. Readers should cite the page they actually inspected and retain its qualifications.

When reporting a failure, include the URL, method, Accept header, expected status and observed response type. Do not include credentials or private workspace data. This produces a useful bug report without treating a successful HTTP request as proof of indexing, model training or answer quality.

Disclosure: this article was drafted with AI. Its examples are based on an implemented public interface and checked against the live endpoints. It does not describe an independent benchmark or a firsthand user testimonial.