DEV Community: sai pramod upadhyayula

Links that don't break when your docs move

sai pramod upadhyayula — Wed, 10 Jun 2026 03:04:36 +0000

Here's a problem that sounds trivial until you've felt it at scale: you publish a documentation page, an agent cites it, someone bookmarks it — and then the author renames the file or moves it into a different folder. The content is the same. The link is dead.

In a docs-as-code world this happens constantly. Files get reorganized, folders get restructured, a getting-started.md becomes onboarding/intro.md. Every one of those moves quietly breaks links, citations, and any agent answer that pointed at the old path. If your link is just "the file path," then the link is only as stable as the file path — which is to say, not stable at all.

We needed links that stay valid even when content moves in source control. Here's how we built them.

The core idea: identity that isn't the path

A path is a location, not an identity. The fix is to give every doc a stable identity that travels with the content, and to build the public link on that identity instead of the path.

So a deeplink looks like this:

https://{host}/cid/{contentId}/fid/{fileId}

No folder structure. No filename. No extension. Just two stable identifiers: which content source the doc came from, and a fileId that uniquely and durably names the document itself. Move the file anywhere you like — the link still resolves, because the link was never about where the file lived.

The whole trick is making fileId stable across moves. That's where git earns its keep.

Deriving a stable file id from git

The naive version is easy: hash the file path.

fileId = sha256(repositoryFilePath);

That gives you a clean, deterministic id — but it has the exact flaw we're trying to avoid: move the file and the hash changes, so you get a new id and a new link. Useless.

The real work is detecting when a "new" path is actually an existing document that simply moved, and carrying its id forward. Git already knows this — every commit diff records renames as a source → target pair. So instead of hashing paths in isolation, we walk the commit history and let the diffs tell us what moved.

Walking from one commit to the next, for every change we ask: did this file exist before under a different path?

const changes = commitDiff.changes || [];
for (const change of changes) {
  const previousPath = change.sourceServerItem; // where it used to live
  const currentPath  = change.item?.path;       // where it lives now

  // If we already had an id for the old path, carry it forward.
  // Otherwise, mint a fresh one from the path hash.
  const fileId = previousPathToId.has(previousPath)
    ? previousPathToId.get(previousPath)
    : sha256(previousPath);

  if (fileId && currentPath) {
    pathToId.set(currentPath, fileId);
  }
}

The key line is the carry-forward: when a file moves, its new path inherits the old path's id. The document keeps its identity through the move, and therefore keeps its link.

Files that didn't change in this commit simply keep whatever id they already had:

for (const item of allFilesAtThisCommit) {
  if (!pathToId.has(item.path)) {
    pathToId.set(item.path, previousPathToId.get(item.path) ?? sha256(item.path));
  }
}

Do this commit-by-commit across the history and you end up with a path → fileId map where the id is anchored to the document's lineage, not its current location.

Handling the awkward cases

Two real-world wrinkles are worth calling out, because they're where naive implementations fall over:

Collisions. Two different documents can, in edge cases, resolve to the same hash-derived id. We detect duplicates and disambiguate the colliding entries by prefixing the commit id, so two distinct docs never share a link:

if (duplicateFileIds.has(fileId) && sha256(repoFilePath) === fileId) {
  pathToId.set(repoFilePath, targetCommitId + fileId);
}

Cost. Walking commit diffs across a large repository is not free — it's the same lesson from selective fetching, where commit data, not file content, is the expensive part. So this derivation rides on cached, immutable commit metadata and only processes the diff between the last processed commit and the current one. You pay for history once, then move incrementally.

Closing the loop: embedding the link where it's used

A stable id is only useful if it actually reaches the consumer. When we publish a doc into the knowledge store that backs our agents, we stamp the deeplink directly into the content and alongside it as metadata:

const deeplink = `https://${host}/cid/${contentId}/fid/${fileId}`;
fileContents = `${fileContents}\n\ndocument_link:${deeplink}`;
filePathToDeepLink[blobPath] = deeplink;

We also write a small sidecar record per document — {fileId}.metadata.json — carrying who last touched it and when:

const sidecar = {
  metadataSchemaVersion: '1',
  lastUpdatedBy: change.sourceLastUpdatedBy ?? '',
  lastUpdatedOn: change.sourceLastUpdatedTimestamp ?? '',
};

Now when an agent answers a question from one of these docs, it can cite a link that (a) points at the live page regardless of where the file currently sits in source control, and (b) comes with provenance — last editor, last edit time — so the answer can be trusted and traced.

Why it matters

The shift is small but the payoff is large: stop treating the file path as the link. Give each document a durable identity derived from its git lineage, build the public link on that identity, and embed it where your consumers — humans and agents alike — actually pick it up.

Content can move freely in source control. Authors can reorganize without a second thought. And every link, citation, and bookmark keeps pointing at the right page — because it was never pointing at a path in the first place.

This is part 3 of the **Docs-as-code at scale* series:*

Stop cloning entire repos for your doc builds
The expensive part of selective doc fetching isn't the files — it's the commits
Links that don't break when your docs move (you are here)

Sai Pramod Upadhyayula is a Senior Software Engineer at Microsoft working on AI-powered enterprise knowledge platforms, and a contributor to the DocFX open-source ecosystem.

The expensive part of selective doc fetching isn't the files — it's the commits

sai pramod upadhyayula — Wed, 10 Jun 2026 03:04:02 +0000

In my last post I argued for resolving before you fetch: use the doc manifest to figure out which 50 files matter before you pull 200,000. That solves the obvious waste. But once we put it into production across dozens of large repos — working closely with the Azure DevOps and content-version provider teams — we found the real cost was hiding somewhere I didn't expect.

It wasn't the file content. It was the commit data.

The surprise: walking commit history is the bottleneck

When you ask a git provider "what changed, and at what version?", you're not making one cheap call. Commit data spans levels — trees reference trees, commits reference parents, and a naive walk fans out into a deep recursive traversal. For a build that needs 50 markdown files, we were spending most of our time not downloading those files, but resolving the commit graph around them.

The content fetch was the easy part. The version resolution was the tax.

The fix: cache the metadata, not the content

The instinct is to cache file content. We did the opposite.

Commit data has one beautiful property: it's immutable. A commit SHA points to exactly one tree, forever. So instead of caching the bytes of intro.md (which change, and which we want fresh), we cache the commit metadata — the tree structure, the file-to-blob mappings, the version graph.

The pattern looks like this:

One full recursive fetch, once. Pay the deep-walk cost a single time per repo and store the resulting commit/tree metadata.
Build incrementally with diffs. After that, never walk the full history again. Ask only for the commit diff since the last known SHA, and patch your cached metadata forward.

An expensive recursive traversal becomes a cheap incremental update. And because the cached layer is immutable metadata, there's no invalidation headache — you're only ever appending knowledge of new commits, never invalidating old ones.

The unexpected bonus: honest freshness

This caching layer turned out to do more than save time. It made freshness honest.

If you're feeding docs into a RAG pipeline or an agent's knowledge base, the scariest failure mode is an answer sourced from a stale or irrelevant file. The commit diff is the cleanest possible signal for this: it tells you precisely which docs moved, were added, or were deleted — at the version level. You stop re-indexing on a blind schedule and start re-indexing on actual change. "Is this doc still current?" becomes a cheap lookup against {repo, commitSha, path} instead of a re-fetch.

Where this bites differently: monorepo vs. many small repos

The same problem shows up in opposite shapes depending on how your org structures things.

In a monorepo, the hard part is precision. One commit graph, but docs are buried among 200k files. The commit diff is your friend — one cached tree plus a diff tells you exactly which of those files changed, and the manifest's src scoping and exclude patterns keep a greedy **/*.md from sweeping in changelogs and archived folders. Freshness is comparatively easy because everything shares one version.

Across many repos under one org, it's a coordination and provenance problem. Fifty repos, fifty commit graphs, fifty manifests — some with no manifest at all. This is where the cached commit metadata earns its keep: you can answer "what changed across all of them?" by diffing each cheaply, instead of re-walking fifty histories on every build. A few things that helped us:

Trust the per-repo manifest as the definition of "what is a doc here." Don't impose a global glob — that's how irrelevant files leak across repo boundaries.
Pin everything to {repo, commitSha, path} and re-resolve on a commit diff, not a timer. Staleness is almost always a re-index-cadence problem, not a fetch problem.
Treat no-manifest repos as out-of-scope, explicitly. The "just index everything" fallback is the single biggest source of irrelevant answers downstream.

The takeaway

Selective fetching gets you from 200,000 files to 50. But the speed and the trustworthiness of the whole pipeline come from a layer underneath it: caching the immutable commit metadata once, then riding commit diffs forever. Monorepos push you toward better filtering; many-repos push you toward better provenance. The commit-diff layer is what makes both fast — and what lets an agent know its docs are actually current.

That same cached commit history powers something else, too: links to your docs that don't break when files get moved or renamed. That's the subject of the next post in this series.

This is part 2 of the **Docs-as-code at scale* series:*

Stop cloning entire repos for your doc builds
The expensive part of selective doc fetching isn't the files — it's the commits (you are here)
Links that don't break when your docs move

Sai Pramod Upadhyayula is a Senior Software Engineer at Microsoft working on AI-powered enterprise knowledge platforms, and a contributor to the DocFX open-source ecosystem.

docfx-remote-include: inline remote markdown into DocFX, and govern the assembled page

sai pramod upadhyayula — Tue, 09 Jun 2026 23:19:27 +0000

docfx-remote-include is a Markdig extension and a dotnet CLI tool for DocFX. It does two things: it inlines markdown fetched from an HTTP service at build time, and it can send each fully-assembled page to a transform service for centralized governance.

It is not a fork of DocFX. It plugs into DocFX's public
BuildOptions.ConfigureMarkdig seam, so it tracks upstream DocFX releases as a regular
NuGet dependency. It targets .NET 8, 9, and 10.

If you read my earlier post on this project, it had a per-include AI rewrite feature —
you tagged an individual include and a model rewrote that fragment in place. That's
gone now. Governance moved up to a single page-level transform that runs once on the
assembled page, which is both simpler and a better fit for the problem. More on that
below.

The include directive

In any markdown file processed by DocFX:

Some local content.

[!remoteinclude[Welcome](snippets/welcome.md)]

Inline usage also works: today's status is [!remoteinclude[s](status/prod.md)].

At build time the extension performs GET {baseUrl}/{source}, parses the response as
markdown, and inlines it. The directive has two shapes that share one syntax:

Block — the directive is the only thing on its line. The fetched markdown is inlined as block content (headings, lists, paragraphs).
Inline — the directive appears mid-paragraph. The fetched markdown must reduce to a single paragraph; only its inline content is spliced in, with no <p> wrapper.

Nested directives, cycle detection (via an AsyncLocal source stack, max depth 8), an
in-process per-build cache, and concurrency capped at 8 in-flight requests are all
built in. By default a missing source fails the build; --allow-missing renders a
visible error placeholder instead.

If your content service uses a non-trivial URL scheme, set urlTemplate with a
{source} placeholder:

{
  "baseUrl": "https://api.example.com/",
  "urlTemplate": "content/GetFile?path={source}"
}

Optional page transform

After all includes are resolved and a page is fully assembled, the build can send that
page to a page transform service. The service owns the rules; pages only declare
intent via YAML frontmatter:

---
transform:
  audience: engineer
  intent: onboarding
  overrides:
    prerequisites: "target macOS users"
---

The extension extracts this transform: block, sends the assembled page plus the
metadata to your configured endpoint, and uses the response. Governance happens once,
at the page level. The library only defines the contract:

public interface IPageTransformService
{
    Task<PageTransformResponse> TransformAsync(
        PageTransformRequest request, CancellationToken ct = default);
}

Omit the transform config to disable it. The library itself has no Azure dependency —
implement IPageTransformService with an LLM or with deterministic rules.

The reference content-and-transform service

The repo ships a reference service (samples/knowledge-service) that plays both roles
from one endpoint surface:

Endpoint	Method	Purpose
`/content/{path}`	GET	Serves markdown for `[!remoteinclude]` directives
`/transform`	POST	Transforms the assembled page
`/health`	GET	Health check

Point both at the same service in remoteinclude.json:

{
  "baseUrl": "http://localhost:8080/content/",
  "transform": {
    "endpoint": "http://localhost:8080/transform",
    "auth": { "mode": "none" }
  }
}

The /transform endpoint has three behaviors depending on how you configure it:

With central guidance (markdown under content/guidance/): it treats that guidance as the source of truth, compares each assembled page against it, and inserts > [!NOTE] **Team Override** callouts above content that deviates. It also harmonizes tone and structure for the page's audience/intent.
Without guidance content: it is a passthrough to the LLM — harmonizing tone and structure for the declared audience/intent, with no comparison or override callouts.
Without an AiEndpoint configured: it returns the page unchanged. /content still works.

Running it as a sidecar

The container binds 0.0.0.0:8080 (set via ASPNETCORE_URLS in the Dockerfile), reads
all configuration from environment variables, and has no required dependencies, so it
runs cleanly as a sidecar next to a docs build. With no content/guidance and no
AiEndpoint it still serves /content and returns pages unchanged from /transform.

Kubernetes — run it in the same Pod and probe /health:

containers:
  - name: docs-build
    image: your-docs-builder:latest
    # baseUrl/transform.endpoint point at http://localhost:8080
  - name: knowledge-service
    image: ghcr.io/saipramod/knowledge-service:latest
    ports:
      - containerPort: 8080
    env:
      - name: Transform__AiEndpoint
        value: ""            # empty = passthrough/no-op transform
    readinessProbe:
      httpGet: { path: /health, port: 8080 }

Because the sidecar shares localhost with the build container, point both baseUrl
and transform.endpoint at http://localhost:8080. A Docker Compose example is in the
sample's README.

Using it

As a CLI (dotnet tool), with a remoteinclude.json next to your docfx.json:

dotnet tool install -g Documentation.DocfxRemoteInclude.Cli
docfx-ri build docs/docfx.json

As a library, for hosts that call Docset.Build(...):

using Docfx;
using Docfx.RemoteInclude;

using var client = new HttpRemoteContentClient(
    baseUri: new Uri("https://internal.example.com/"),
    authHandler: async (request, ct) =>
        request.Headers.Authorization = new("Bearer", await GetJwtAsync(ct)));

await Docset.Build("docs/docfx.json", new BuildOptions
{
    ConfigureMarkdig = pipeline => pipeline.UseRemoteInclude(client, new RemoteIncludeOptions
    {
        PageTransformService = myTransformService, // optional IPageTransformService
    }),
});

Provide your own IRemoteContentClient for non-HTTP sources, custom auth (mTLS, signed
URLs), or on-disk caching.

Auth and credentials

Both the content and transform auth blocks accept
{ "mode": "none" | "default" | "managedIdentity" | "jwt" | "key", "value": "...", "scope": "..." }.
value can indirect through environment variables with $VAR / ${VAR}, and scope
overrides the OAuth audience for default/managedIdentity modes. Credentials are read
from environment variables or a host-supplied callback — never from docfx.json, and
never written to disk.

Try it end to end

The repo includes a runnable samples/basic DocFX site wired to the
samples/knowledge-service reference service. Run the service, build the sample, and
you get shared snippets and page-level governance working together:

# terminal 1
cd samples/knowledge-service
dotnet run

# terminal 2
docfx-ri build samples/basic/docfx.json

MIT-licensed. Issues and PRs welcome.

GitHub: github.com/saipramod/docfx-remote-include
NuGet: Documentation.DocfxRemoteInclude · Documentation.DocfxRemoteInclude.Cli

Sai Pramod Upadhyayula is a Senior Software Engineer at Microsoft working on
AI-powered enterprise knowledge platforms, and a contributor to the DocFX
open-source ecosystem.

Stop Cloning Entire Repos for Your Doc Builds

sai pramod upadhyayula — Wed, 27 May 2026 01:27:33 +0000

Your docs live next to your code. That's the docs-as-code promise — version control, pull request reviews, CI/CD pipelines. It works beautifully.

Until your repo hits 100,000 files.

The problem nobody talks about

Our team runs a documentation portal that pulls content from dozens of large repositories. Each doc build needs a handful of markdown files and images from repos containing hundreds of thousands of files. The naive approach — git clone — is painfully slow and wasteful.

We tried sparse checkout. We tried shallow clones. We tried the git provider APIs directly. Each came with its own problems:

Full clones: Minutes of download time for a build that needs 50 files
API file downloads: Hit rate limits after a few hundred files
Sparse checkout: Still requires git history negotiation and doesn't help with API-based pipelines

The irony? The manifest already declares exactly which files are needed. The docfx.json (or whatever config your static site generator uses) lists every content glob, every resource pattern. We just weren't using that information early enough.

Why this matters even more now: AI agents

This isn't just a build-speed problem anymore. If you're building AI agents that answer questions about your product, help onboard developers, or assist with internal processes — they need access to your documentation. Not your code. Not your tests. The docs.

The challenge scales fast:

RAG pipelines need to ingest documentation from dozens of repos — cloning all of them is absurd
Incremental indexing requires knowing which files are documentation vs. code — the manifest already tells you
Multi-repo knowledge bases need a fast, selective way to pull only content files across many repos

The faster and more precisely you can extract documentation from your repositories, the fresher and more accurate your agents' knowledge becomes. Solving the selective fetch problem unlocks both faster builds and reliable AI-powered documentation experiences.

The idea: resolve before you fetch

What if we flipped the order?

Instead of: clone everything → build → throw away 99% of the files

We do: get the file listing → match against manifest → fetch only what matches

┌─────────────────┐     ┌──────────────────────┐     ┌─────────────────┐
│  Git Provider    │     │ selective-repo-fetch  │     │  Doc Pipeline   │
│  (file listing)  │────▶│  (manifest matching   │────▶│  (build only    │
│                  │     │   + reference filter) │     │   matched files)│
└─────────────────┘     └──────────────────────┘     └─────────────────┘

A file tree listing from GitHub/Azure DevOps/GitLab is a single, cheap API call — it returns metadata, not file contents. Match that listing against your manifest patterns, and you know exactly what to fetch.

Introducing selective-repo-fetch

We open-sourced this logic as a TypeScript library: selective-repo-fetch. It's MIT-licensed and provider-agnostic.

npm install github:microsoft/selective-repo-fetch

Here's the core workflow:

import { resolveFileMatches, filterReferencedResources } from 'selective-repo-fetch';

// Your manifest declares what your doc site needs
const manifest = {
  build: {
    content: [{ files: ['**/*.md'], src: 'docs' }],
    resource: [{ files: ['**/*.{png,jpg,svg}'], src: 'docs/images' }],
  },
};

// Step 1: Get file listing from any git API (one cheap metadata call)
const repoFiles = await getTreeListing(); // returns [{ path: '/docs/intro.md' }, ...]

// Step 2: Resolve manifest → content + resource matches
const matched = resolveFileMatches(repoFiles, manifest, '/', '/docfx.json');
// matched.contentMatches → only the markdown files your build needs
// matched.resourceMatches → only images/videos matching resource globs

From 200,000 files down to the 50 that matter. One function call.

Going further: filtering unreferenced resources

Glob matching is great, but it can be too generous. A **/*.png pattern in your resource section will match every image under that folder — even the ones no markdown file actually references.

For large repos, this matters. Unreferenced images can be megabytes of wasted downloads.

So we added a second pass:

// Step 3: Fetch the content files (small text — fast and cheap)
const contentFileTexts = {};
for (const filePath of matched.contentMatches) {
  contentFileTexts[filePath] = await fetchFileContent(filePath);
}

// Step 4: Filter resources to only those actually referenced
const referencedResources = filterReferencedResources(
  matched.resourceMatches,
  contentFileTexts
);
// Scans markdown/HTML for ![](path), <img src="path">, [text](path), etc.
// Drops any resource not referenced by any content file

This scans your content files for markdown image references (![](path)), links ([text](path)), and HTML attributes (src="path", href="path"). If a resource file isn't referenced anywhere in your content, it gets dropped.

The full pipeline

Here's what it looks like end-to-end with the GitHub API:

import { Octokit } from '@octokit/rest';
import { resolveFileMatches, filterReferencedResources } from 'selective-repo-fetch';

const octokit = new Octokit({ auth: token });

// 1. One API call to get the full file tree (metadata only, no content)
const { data } = await octokit.git.getTree({
  owner, repo, tree_sha: 'HEAD', recursive: 'true'
});

const files = data.tree
  .filter(item => item.type === 'blob')
  .map(item => ({ path: '/' + item.path }));

// 2. Resolve manifest patterns
const manifest = JSON.parse(/* your docfx.json */);
const matched = resolveFileMatches(files, manifest, '/', '/docfx.json');

// 3. Fetch content files (small text)
const contentTexts: Record<string, string> = {};
for (const path of matched.contentMatches) {
  const { data } = await octokit.repos.getContent({ owner, repo, path: path.slice(1) });
  contentTexts[path] = Buffer.from(data.content, 'base64').toString();
}

// 4. Filter resources to only referenced ones
const resources = filterReferencedResources(matched.resourceMatches, contentTexts);

// 5. Fetch only referenced resources
// You now have the exact list — nothing wasted

What it handles

The manifest matching is thorough:

Glob patterns with brace expansion (*.{md,yml})
src path resolution relative to manifest location
Per-section excludes (exclude: ["**/draft/**"])
Templates, metadata files, .order files — auto-included
External references via src: "../other-folder" — discovered before you fetch

The reference filter handles:

Markdown images and links: ![alt](path), [text](path)
HTML attributes: <img src="path">, <video src="path">, <a href="path">
Path normalization: strips ~/, leading /, query strings, anchors
Skips external URLs, data URIs, mailto:, javascript:
Case-insensitive filename matching

When to use this

Documentation portals pulling from multiple repos — resolve before you clone
Monorepo doc builds — your manifest knows what matters, use it
CI/CD pipelines — cut build times by fetching only what changed
Any static site generator (DocFX, MkDocs, Sphinx, Docusaurus) that uses a manifest

Why this matters for AI agents

There's a downstream benefit we didn't anticipate when we first built this: making documentation efficiently available to AI agents.

If you're building agents that answer questions about your product, help onboard developers, or assist with internal processes — they need access to your documentation. But they don't need your entire codebase. They need the docs.

The manifest-driven approach gives you exactly that separation:

Selective ingestion — only pull documentation into your RAG pipeline, not code, tests, or CI configs
Incremental updates — when a doc changes, you know it's a doc (not a code file) because the manifest says so
Multi-repo knowledge bases — pull docs from 50+ repos without cloning any of them

// Feed docs from multiple repos into your agent's knowledge base
for (const repo of repositories) {
  const files = await getTreeListing(repo);
  const matched = resolveFileMatches(files, repo.manifest, '/', '/docfx.json');

  // Only index documentation — not code, not tests, not configs
  for (const docPath of matched.contentMatches) {
    const content = await fetchFile(repo, docPath);
    await knowledgeBase.ingest({ path: docPath, repo: repo.name, content });
  }
}

The faster and more precisely you can extract documentation from your repos, the fresher and more accurate your agents' knowledge becomes. Efficient content fetching is the foundation of a reliable AI-powered docs experience.

Try it out

The library is MIT-licensed and has zero opinions about your git provider — it works with any API that can give you a file listing.

npm install github:microsoft/selective-repo-fetch

GitHub: microsoft/selective-repo-fetch

If your doc builds are slow because of large repos, give it a try. And if you have ideas for improvements, PRs are welcome.

What's the worst monorepo doc build experience you've had? I'd love to hear about it in the comments.

Building a Markdig Extension for DocFX: Remote Content Inclusion with AI Rewriting

sai pramod upadhyayula — Fri, 22 May 2026 22:35:09 +0000

The Problem Every Documentation Team Hits

If you've worked on a documentation platform at any scale, you've hit this
problem: content lives in multiple places, and you need to compose it into
a single, coherent site at build time.

Maybe your API reference is generated from code. Maybe your troubleshooting
guides live in a separate service. Maybe different teams own different sections,
and you need to pull them together into one DocFX site without copy-pasting
content that goes stale the moment it's duplicated.

DocFX — Microsoft's open-source documentation
generator — is excellent at building static documentation from local markdown
files. But it doesn't natively support fetching and inlining content from remote
sources at build time.

I needed exactly that capability. So I built it.

Introducing docfx-remote-include

docfx-remote-include is
a standalone Markdig extension and CLI tool
that adds remote content inclusion to DocFX.

It's not a fork of DocFX. It hooks into DocFX's public BuildOptions.ConfigureMarkdig
extension point, so it tracks upstream releases as a regular NuGet dependency. When
DocFX updates, your remote include capability doesn't break.

The Directive

In any markdown file processed by DocFX, you can write:

Some local content.

[!remoteinclude[Welcome](path/to/snippet.md)]

More local content.

At build time, the extension fetches {baseUrl}/path/to/snippet.md via HTTP,
parses the response as markdown, and inlines the result. It works in two modes:

Block mode — when the directive is the only thing on its line, the fetched content is inlined as full block content (headings, lists, paragraphs).
Inline mode — when the directive appears mid-paragraph, only inline content is spliced in (no wrapping <p> tags).

The AI Twist

Here's where it gets interesting. You can optionally add a rewrite hint:

[!remoteinclude[Install](snippets/install.md "match this page's tone and tense")]

When a hint is provided, the fetched content is passed through a pluggable
IRewriteService — backed by any LLM you choose (Azure OpenAI, local models,
anything) — which adapts the content to match the surrounding page's voice
and style.

Without a hint, the content is inlined verbatim. The AI capability is entirely
opt-in and has zero vendor lock-in.

Architecture Decisions

Why an Extension, Not a Fork?

Forking DocFX would mean maintaining a parallel codebase and falling behind on
upstream improvements. Instead, docfx-remote-include uses the public
ConfigureMarkdig seam that DocFX exposes:

await Docset.Build("docs/docfx.json", new BuildOptions
{
    ConfigureMarkdig = pipeline => pipeline.UseRemoteInclude(client, options),
});

This means:

Zero maintenance burden from DocFX internals
Works with any DocFX version that exposes ConfigureMarkdig
Can be combined with other Markdig extensions

Auth Flexibility

Enterprise documentation often lives behind authentication. The extension
supports multiple auth modes out of the box:

Mode	Use Case
`none`	Public content services
`default`	Azure Default Credential (local dev, CI/CD)
`managedIdentity`	Azure Managed Identity (production)
`jwt`	Bearer token (custom auth)
`key`	API key header

All credentials are read from environment variables or host callbacks — never
from config files committed to source control.

Safety Features

When you're pulling remote content into a build pipeline, things can go wrong:

Cycle detection — an AsyncLocal source stack prevents infinite recursion when remote content includes other remote content. Max depth defaults to 8.
Concurrency control — in-flight requests are capped at 8 by default to avoid overwhelming the content service.
In-process caching — each source URL is fetched once per build, regardless of how many pages reference it.
Hard fail by default — if a remote source returns 404, the build fails. Use --allow-missing to render a visible error placeholder instead.

Getting Started

As a CLI Tool

# Add the NuGet source (one-time)
dotnet nuget add source "https://nuget.pkg.github.com/saipramod/index.json" \
  --name "docfx-tools" --username YOUR_GITHUB_USERNAME --password YOUR_GITHUB_PAT

# Install the tool
dotnet tool install -g Docfx.RemoteInclude.Cli --source "docfx-tools"

# Build your docs
docfx-ri build docs/docfx.json

Configuration

Create remoteinclude.json next to your docfx.json:

{
  "baseUrl": "https://your-content-service.com/",
  "allowMissing": false,
  "urlTemplate": "api/content/GetFile?path={source}",
  "auth": {
    "mode": "managedIdentity",
    "scope": "api://your-app-id/.default"
  },
  "ai": {
    "endpoint": "https://your-aoai.openai.azure.com/",
    "deployment": "gpt-4o-mini",
    "contextStrategy": "section"
  }
}

As a Library

For full control, use the library directly:

using Docfx;
using Docfx.RemoteInclude;

using var client = new HttpRemoteContentClient(
    baseUri: new Uri("https://your-content-service.com/"),
    authHandler: async (request, ct) =>
    {
        request.Headers.Authorization =
            new("Bearer", await GetJwtAsync(ct));
    });

await Docset.Build("docs/docfx.json", new BuildOptions
{
    ConfigureMarkdig = pipeline => pipeline.UseRemoteInclude(client,
        new RemoteIncludeOptions
        {
            RewriteService = myRewriter, // optional
        }),
});

Implement IRemoteContentClient for non-HTTP sources (file systems, databases,
signed URLs). Implement IRewriteService to plug in any LLM.

Why This Matters

Documentation platforms at scale need to compose content from multiple
authoritative sources. Copy-pasting creates drift. Git submodules add complexity.
Custom build scripts are fragile.

docfx-remote-include solves this with a clean, declarative syntax that works
within DocFX's existing pipeline. The optional AI rewriting capability means
content from different sources can read as if it was written for the page it
appears on.

The project is MIT-licensed, open source, and accepting contributions.

GitHub: github.com/saipramod/docfx-remote-include

Sai Pramod Upadhyayula is a Senior Software Engineer at Microsoft, where he
works on AI-powered enterprise knowledge platforms. He co-authored "AutoTSG:
Learning and Synthesis for Incident Troubleshooting" (ESEC/FSE 2022) and
contributes to the DocFX open-source ecosystem.