Freeloading gpt-4o, llama, and many more llms on Github Model in AutoGen.Net

Github recently introduced Github Models, which allows us to try and test from a wide range of model types, sizes, and specializations FOR FREE. Considering the wide range of available models (gpt-4o, gpt-4o-mini, llama-3.1 8B, llama-3.1 405B), maybe now it's the second best time to have a github account. In this blog post, we'll explore how to freeload these models using AutoGen.Net.

Why GitHub Models?

• Free Access: All you need is a GitHub account to get started.
• Diverse Model Selection: From GPT-4 to Llama-3.1, GitHub offers a variety of models to suit different needs.

Getting Started

• Create a GitHub Account: If you don't already have one, sign up for a free GitHub account.
• Generate an API Key: Navigate to Settings > Developer settings > Token (classic) and generate a new token.
• Step Up Your Environment: Save your API key securely in your local environment.

Integrating with AutoGen.Net

Here's a simple example of how to use the Llama-3.1 405B model with AutoGen.Net:

Create a dotnet console app, and add package AutoGen 0.2.0

dotnet new console -n GithubLLMFreeLoader
cd GithubLLMFreeLoader
dotnet add package AutoGen --version 0.2.0

Create a simple chat sample using llama-3.1-405B

using AutoGen.AzureAIInference;
using AutoGen.AzureAIInference.Extension;
using AutoGen.Core;
using Azure.AI.Inference;
var githubKey = Environment.GetEnvironmentVariable("GH_API_KEY") ?? "YOUR_KEY";
var endpoint = "https://models.inference.ai.azure.com";
var client = new ChatCompletionsClient(new Uri(endpoint), new Azure.AzureKeyCredential(githubKey));

var agent = new ChatCompletionsClientAgent(
    chatCompletionsClient: client,
    name: "assistant",
    modelName: "Meta-Llama-3.1-405B-Instruct")
    .RegisterMessageConnector()
    .RegisterPrintMessage();

await agent.SendAsync("Tell me a long joke");

The code snippet demostrates how to

• Connect to Github Model
• Create an AutoGen.Net agent using llama-3.1-405B
• Start a conversation with agent.

Limitation

Okay, here's the not-so-great part: there's a limit on how much you can use this service. Since it's free, they don't let you go crazy with it. It's fine for trying things out or playing around, but it's not going to cut it if you need it for serious, heavy-duty work. But hey, it's free, so we can't really complain, right?