DEV Community

Cover image for Agent Mode With Third-Party Models in Copilot
Bas Steins
Bas Steins

Posted on • Originally published at bas.codes

Agent Mode With Third-Party Models in Copilot

#ai #vscode #vibecoding

How to Use Third Party Models in GitHub Copilot Agent Mode

GitHub Copilot has become an indispensable coding assistant for millions of developers. While it's incredibly powerful out of the box, many developers wonder: "Can I use other AI models like Claude, GPT-4, or Grok with Copilot's advanced Agent Mode?"

The answer is yes – but it requires a clever workaround.

The Problem: Tool Support Limitations

OpenRouter is fantastic for accessing various AI models from OpenAI, Anthropic, and others on a pay-per-use basis. It's perfect for testing different models without committing to expensive subscriptions. You can use powerful models like:

  • Grok - X.AI's conversational model
  • Kimi K-2 - Moonshot AI's capable model

However, there's a critical limitation: GitHub Copilot's Agent Mode requires models to support function calling (tools), but OpenRouter's API doesn't announce tool support for its models.

This means that even though models like Claude and GPT-4 natively support function calling, Copilot won't enable Agent Mode when using them through OpenRouter.

The Solution: A Proxy Chain

The solution is elegant: create a local proxy chain that preserves tool calling capabilities while routing requests through OpenRouter.

Here's the architecture:

VSCode Copilot → oai2ollama (port 11434) → LiteLLM (port 4000) → OpenRouter → AI Models
Enter fullscreen mode Exit fullscreen mode

The magic happens in three stages:

  1. VSCode Copilot sends requests to what it thinks is a local Ollama server
  2. oai2ollama translates Ollama API calls to OpenAI format
  3. LiteLLM proxies OpenAI-compatible requests to OpenRouter
  4. OpenRouter routes to the actual AI model providers

Throughout this chain, tool/function calling capabilities are preserved, enabling full Agent Mode functionality.

Setting Up the Proxy: Step-by-Step Guide

I've created an open-source project called copilot-ollama that handles all the complex proxy setup automatically.

Prerequisites

Before we start, make sure you have:

  • uv package manager installed
  • An OpenRouter API key (get one here)
  • VSCode with the GitHub Copilot extension

Installation

  1. Clone the repository: 
   git clone https://github.com/bascodes/copilot-ollama.git
   cd copilot-ollama
Enter fullscreen mode Exit fullscreen mode
  1. Set your OpenRouter API key: 
   export OPENROUTER_API_KEY=your_openrouter_api_key_here
Enter fullscreen mode Exit fullscreen mode
  1. Start the proxy servers: 
   ./run.sh
Enter fullscreen mode Exit fullscreen mode

This single command starts both proxy servers and handles all the configuration.

  1. Configure VSCode:

    • Open VSCode settings
    • Set github.copilot.chat.byok.ollamaEndpoint to http://localhost:11434
    • Click "Manage Models" → Select "Ollama"

  2. Start coding! 🎉

Your OpenRouter models are now available in Copilot Agent Mode with full tool calling support.

Customizing Available Models

The proxy comes with sensible defaults, but you can easily customize which models are available by editing the config.yaml file:

model_list:
  - model_name: claude-3-sonnet  # Name that appears in VSCode
    litellm_params:
      model: openrouter/anthropic/claude-3-sonnet  # Actual OpenRouter model

  - model_name: gpt-4-turbo
    litellm_params:
      model: openrouter/openai/gpt-4-turbo

  - model_name: kimi-k2
    litellm_params:
      model: openrouter/moonshotai/kimi-k2
Enter fullscreen mode Exit fullscreen mode

Popular Models to Try

Here are some excellent models to experiment with:

Model OpenRouter Path Best For
Claude 3 Sonnet openrouter/anthropic/claude-3-sonnet Code generation and reasoning
GPT-4 Turbo openrouter/openai/gpt-4-turbo Latest OpenAI capabilities
Mixtral 8x7B openrouter/mistralai/mixtral-8x7b-instruct Fast open-source model
Llama 3 70B openrouter/meta-llama/llama-3-70b-instruct Meta's powerful open model

Why This Matters

This solution opens up exciting possibilities:

  • Cost Control: Pay per use instead of monthly subscriptions
  • Model Diversity: Access to cutting-edge models from multiple providers
  • Full Agent Mode: Complete tool calling and function support
  • Local Control: Everything runs locally, maintaining privacy

Technical Details

For those interested in the implementation details:

  • The proxy uses LiteLLM for OpenAI-compatible proxying
  • oai2ollama handles Ollama API compatibility
  • Tool calling metadata is preserved throughout the proxy chain
  • The solution works around VSCode's current limitation of only supporting Ollama endpoints for custom models

Looking Forward

There's an ongoing discussion about enabling custom API endpoints directly in Copilot: microsoft/vscode-copilot-release#7518. Once that's implemented, this workaround won't be necessary.

However, until then, this proxy solution provides a robust way to unlock the full potential of GitHub Copilot with your favorite AI models.

The restrictions of the Copilot API can be found in the VSCode Copilot source code, which is where I discovered the Ollama endpoint configuration option.

Get Started Today

Ready to supercharge your coding experience? Check out the copilot-ollama repository and start experimenting with powerful AI models in GitHub Copilot's Agent Mode.

Top comments (0)