DEV Community

Cover image for How I Built a VS Code AI Coding Assistant That Runs Under 3.2K Tokens Per Request
alilo113
alilo113

Posted on

How I Built a VS Code AI Coding Assistant That Runs Under 3.2K Tokens Per Request

 Most AI coding assistants feel heavy fast. Between injecting open tabs, large file trees, and dozens of tool schemas on every turn, a simple coding request can easily balloon to 30K–100K+ tokens—driving up latency and burning through API credits.

When I built Sidekick Code AI, I wanted to test the opposite approach: how useful can an in-editor coding assistant be if you keep context retrieval tight and predictable?

The Token & Latency Baseline

In recent trace runs (sidekick_cloud_chat_completion), a real code-generation turn used 3.13K total tokens and completed in ~2 seconds[cite: 2]:

  • Input: 2.75K tokens (88%) — system prompt, user request, and targeted local context[cite: 2]
  • Output: 383 tokens (12%) — focused code generation and next-step guidance[cite: 2]

For BYOK (Bring Your Own Key) workflows, keeping routine turns around ~3K tokens makes a big difference: on mid-tier models, a request costs under two cents, and on lightweight models it costs a fraction of a tenth of a cent.

3 Decisions That Kept Context Lean

1. Local Indexing Over Full-File Dumping

Instead of stuffing entire files or every open editor tab into the prompt by default, Sidekick indexes locally and pulls only the relevant snippets needed for the active request.

2. Minimal Prompt & Schema Overhead

Every unused tool definition or verbose system instruction gets tokenized on every single call. Keeping the base prompt compact prevents context bloat before the user even types a question.

3. Targeted, Incremental Edits

Rather than asking the model to rewrite large modules in one pass, prompts are structured to generate focused changes step by step—keeping output tokens low (~300–500 tokens) and latency around 2 seconds[cite: 2].

Try It in VS Code

If you prefer lightweight, BYOK tools and want to test it on a real repo, you can install Sidekick Code AI directly inside VS Code via Quick Open (Ctrl+P / Cmd+P):

ext install alidendenne.sidekick-code-ai
Enter fullscreen mode Exit fullscreen mode

Or via terminal:

code --install-extension alidendenne.sidekick-code-ai
Enter fullscreen mode Exit fullscreen mode

I'm actively refining the local indexing and context budgeting based on real developer workflows—if you try it out and hit any friction or edge cases, I'd genuinely appreciate your technical feedback in the comments.

Top comments (0)