Caleb Han

Posted on Jan 30

DocuMate - Eliminate Documentation Drift

#devchallenge #githubchallenge #cli #githubcopilot

GitHub Copilot CLI Challenge Submission

What I Built

DocuMate - An AI-powered documentation assistant that helps developers maintain high-quality code documentation using GitHub Copilot CLI.

Documentation drift is a silent killer in codebases. You refactor a function, add new parameters, change behavior - but the docs stay frozen in time. DocuMate solves this by combining static analysis with GitHub Copilot's AI capabilities to detect drift, generate accurate documentation, and maintain documentation health across TypeScript, JavaScript, Python, and Markdown files.

For me, this project represents the intersection of developer productivity and code quality. Good documentation shouldn't be an afterthought - it should be as automated and intelligent as the code itself.

Demo

Live Project: github.com/calebyhan/documate

YouTube Video Demo

DocuMate Demo

What Makes DocuMate Special

Drift Detection - Analyzes git history to find code that changed without doc updates
Multi-Language Support - TypeScript, JavaScript, Python, and Markdown
Transparency - --explain flag shows all Copilot prompts/responses
Session Metrics - --stats tracks API usage, cache hits, and performance
Intelligent Prioritization - Focuses on public APIs and complex code first

My Experience with GitHub Copilot CLI

Building DocuMate was a masterclass in working with GitHub Copilot CLI as an API rather than just an autocomplete tool. Here's what I learned:

The Integration Journey

Subprocess Architecture
I built a Node.js wrapper around gh copilot suggest and gh copilot explain to make them programmable. The key challenge was parsing JSON from AI responses that often included markdown, code blocks, and explanatory text. I implemented multi-strategy parsing (regex extraction, code block detection, fallback cleaning) to handle this robustly.

// Core wrapper that powers all AI features
async function callCopilot(prompt: string, type: 'explain' | 'suggest') {
  const result = await execPromise(
    `gh copilot ${type} "${escapedPrompt}"`
  );
  return parseAIResponse(result.stdout);
}

Prompt Engineering
The quality of Copilot's output depends entirely on prompt structure. I discovered that:

Including code context before and after the target function improved accuracy by ~40%
Asking for specific JSON schemas with examples reduced parsing errors
Breaking complex tasks into multiple smaller Copilot calls worked better than one large prompt

Caching Strategy
To reduce API calls (and costs), I implemented a semantic cache. If you run documate drift twice on the same code, the second run uses cached Copilot responses. This improved performance by 60% on repeated scans.

Features Powered by Copilot

Semantic Drift Detection - Copilot analyzes if code changes altered the semantic meaning, not just syntax
Context-Aware Doc Generation - Generates docs that understand the surrounding codebase patterns
Interactive Chat Mode - Full conversational AI for documentation questions
Intelligent Fix Suggestions - Understands what changed and updates docs accordingly

What Surprised Me

The Good:

Copilot CLI's responses are remarkably consistent when given structured prompts
The explain command is perfect for semantic code analysis
Error handling in Copilot is excellent - it rarely crashes, even with malformed prompts

The Challenges:

No streaming API - each call is synchronous, which can feel slow for large files
JSON parsing requires defensive programming due to variable output formats
Rate limiting isn't clearly documented, so I added exponential backoff

Transparency & Developer Trust

One thing I'm particularly proud of is the --explain and --stats flags. They show:

Exact prompts sent to Copilot
Raw responses received
API call counts, cache hit rates, response times

Developers should understand what AI is doing under the hood. These flags turn DocuMate from a "magic box" into an observable, debuggable tool.

What I'd Build Next

If I continue this project:

VS Code Extension - Real-time drift warnings as you code
CI/CD Integration - Block PRs with documentation health below threshold
Team Dashboards - Track documentation metrics across teams
Custom Models - Support for other LLM providers (Claude, GPT-4)

Final Thoughts

GitHub Copilot CLI is often seen as a terminal assistant, but it's so much more. It's a programmable AI API that can power entire applications. DocuMate proves that Copilot can analyze, understand, and improve codebases at scale.

The future of developer tools isn't just AI-assisted coding - it's AI-integrated workflows that make quality and maintenance as easy as writing code in the first place.

Tech Stack:

TypeScript with pure ESM
GitHub Copilot CLI (gh copilot)
TypeScript Compiler API for AST parsing
Python AST parser for .py files
Remark for Markdown analysis
Simple-git for history tracking
Inquirer for interactive CLI
Chalk, Ora, Boxen for beautiful terminal UI

Try it yourself:

git clone https://github.com/calebyhan/documate.git
cd documate
npm install && npm run build && npm link
documate scan messy-project/src

DEV Community