DEV Community: Xiaoyun Zhang

LLM Canvas: The Story Behind

Xiaoyun Zhang — Sun, 17 Aug 2025 05:33:20 +0000

Project Repository: https://github.com/LittleLittleCloud/llm-canvas

Project Website: https://littlelittlecloud.github.io/llm-canvas/

The Beginning: The Struggle with Linear Logs

The initial inspiration for LLM Canvas came from a very practical pain point I encountered while developing Large Language Model (LLM) applications: how to effectively and clearly record and review LLM conversation histories.

We know that many LLM conversation flows, especially in complex Agent applications, are not inherently linear. The entire workflow is more like a continuously growing tree, where each LLM call or Agent decision can be a new branch, and the final result is often an integration of wisdom from multiple branches.

In such scenarios, traditional print logging methods fall short. Linear text output forces developers to painfully "reconstruct" what should be an intuitive call tree in their minds. When dealing with multi-turn conversations and complex Agent interactions, this experience is undoubtedly painful. You need to guess and restore the model's real thinking path from a long series of flat, sequential records.

It was then that an idea emerged: if there could be a tool that not only records but also visualizes these messages and clearly organizes the parent-child and branching relationships between messages, then the entire debugging and understanding process would become incredibly intuitive.

This was the initial idea of LLM Canvas.

From Idea to Reality: The Seven-Day Challenge

After having the initial idea, I began thinking about how to make it a reality. I set several core principles for this project:

Rich UI is Essential: To intuitively display multimodal content (such as images) and complex tree structures, a powerful frontend interface was indispensable. I chose React because of its mature ecosystem and excellent handling of complex states and views. Ultimately, I decided to package the frontend as static resources hosted by the Python backend, so users wouldn't need to install additional Node.js environments, greatly lowering the barrier to entry.
Python First: The core of the LLM ecosystem is in Python, so providing a simple and easy-to-use Python SDK was paramount. All functionality should be accessible through just a few lines of Python code.
Minimal Dependencies: I wanted users to be able to complete the entire installation with a single pip install command. This meant I had to minimize external dependencies as much as possible, creating a truly "plug-and-play" tool.
Rapid Validation: This was an unvalidated market idea, and I didn't want to invest too much energy in the early stages. I set myself a goal: with the help of AI programming (Vibe Coding), complete a usable prototype (POC) within 7 days. This was both a challenge and a rapid iteration strategy.

Fortunately, AI programming tools greatly improved development efficiency, allowing me to focus on implementing core logic. In the end, I completed the first version within the planned 7 days, validating the feasibility of the core concept of "visualized branching conversations."

Core Value: Managing Conversations Like Git

If LLM Canvas has one core value, it's definitely conversation branch management.

When designing the API, I faced a core challenge: how to design an API that could elegantly handle simple linear conversations while also easily managing complex multi-thread conversation integration?

This required creativity and was work that AI couldn't replace. After much deliberation, I suddenly realized that this need was remarkably similar to Git, the version control tool in software development:

A Git Repository is like a complete conversation Canvas.
A Git Branch is like an independent conversation Thread.
Different branches can develop independently and then be merged, which perfectly mirrors the pattern in LLM applications where multiple Agent branches work in parallel and then consolidate results.
Most importantly, Git's distributed nature allows branches to operate in parallel, just like how LLM or Agent calls can run concurrently across different conversation paths without blocking each other.

This analogy was a good fit. Git is one of the most familiar tools for programmers, and borrowing its mental model could greatly reduce the learning curve for developers.

So I designed LLM Canvas's API based on Git's concepts. Developers no longer need to manually manage complex message IDs and parent-child relationships, but can instead focus on the higher-level "conversation threads" just like operating Git branches. Through familiar concepts like checkout and commit, they can easily create, switch, and manage complex conversation flows. This is the core idea behind the API design.

Looking Forward: From Tool to Interaction Paradigm

For the future, I have two main directions of thinking:

First, horizontal expansion to create a more universal visualization tool.
LLM Canvas was designed with "universality" in mind from the beginning. Currently, we only provide a Python SDK, but the LLM ecosystem is multilingual. Therefore, I will consider adding support for more languages in the future, such as providing official client SDKs for C# and TypeScript developers, making LLM Canvas a standard tool in every LLM developer's debugging toolkit.

Second, vertical deepening to explore new interaction paradigms.
The "branched conversations" provided by LLM Canvas is not only a powerful debugging tool, but also represents a novel, non-linear way of interacting with LLMs. Traditional chatbots follow a linear "question-answer" model, while with Canvas, users can start from any conversation node and explore in different directions, like freely navigating through a mind map.

Therefore, I might build a chatbot directly into the product, allowing users to interact with LLMs directly on the canvas, initiating calls from different branches and different contexts. This would evolve LLM Canvas from a "developer tool" into a more creative "interaction platform," similar to the direction explored by products like Flowith.

In summary, LLM Canvas's future has both the breadth of being a foundational tool and the depth of becoming an innovative interaction platform. I'm very excited to see it flourish in both directions.

Some Key Takeaways

Looking back on this development journey, two things stood out:

On AI programming experience.
I found that as long as I clearly knew what I wanted the AI to do, it could be an incredibly reliable partner. The LLM Canvas project, with its 30,000 lines of code across the frontend and backend, was prototyped in just 7 days with AI's help. And 5 of those were weekdays when I was also juggling a full-time job, so I wasn't even coding at full capacity. This made me realize that AI programming is becoming a new form of core productivity.

On the cost of idea validation.
This also made me realize that in the age of AI, the cost of bringing an idea to life has become very low. A project that might have taken months to build in the past can now have a prototype in a week or less. So, if you have a new idea now, consider giving it a try.

Final Words

I want to say to all the friends who are interested in LLM Canvas:

If you've ever felt lost in linear logs and wished for a more intuitive way to understand and debug your LLM applications, welcome to Try it out!

LLM Canvas is an open-source project born from real problems. It's still developing and needs the community's strength to help it grow. Every piece of Feedback you provide is valuable, and of course, Pull Requests are welcome!

I look forward to exploring a clearer, more intuitive future for LLM development with everyone.

Build a Weather Chatbot with DeepSeek v3 and OpenAI SDK: A Step-by-Step Guide

Xiaoyun Zhang — Thu, 09 Jan 2025 11:35:18 +0000

In this tutorial, we'll walk through creating a weather chatbot using DeepSeek v3 and OpenAI SDK for .NET. The complete source code can be found in this GitHub repository

Prerequisites

.NET SDK (8.0 or later)
DeepSeek API key (You can sign up for an account with ￥10 credit at DeepSeek

Create a New Console Project

First, let's set up our project by creating a new console application and installing required OpenAI SDK packages:

dotnet new console -n WeatherChatbot
cd WeatherChatbot
dotnet add package OpenAI

Setting Up the DeepSeek Client

Instead of using the default OpenAI endpoints, we'll configure our client to use DeepSeek's API. The DeepSeek API is compatible with the OpenAI API, so we can use the OpenAI SDK to connect to it:

var option = new OpenAIClientOptions
{
    // Set the DeepSeek API endpoint
    Endpoint = new ("https://api.deepseek.com"),
};
var client = new OpenAIClient(new ApiKeyCredential(DEEP_SEEK_API_KEY), option);

// use deepseek-chat model which points to the latest v3 model
var chatClient = client.GetChatClient("deepseek-chat");

Defining Function Tools

One of the key features we'll be using is the ability to define "tools" - functions that the language model can call when it needs specific information. We'll create one tool to retreive dummy weather information for a given location:

var getCurrentWeatherTool = ChatTool.CreateFunctionTool(
    functionName: nameof(GetCurrentWeather),
    functionDescription: "Get the current weather in a given location",
    functionParameters: BinaryData.FromBytes("""
        {
            "type": "object",
            "properties": {
                "location": {
                    "type": "string",
                    "description": "The city and state, e.g. Boston, MA"
                },
                "unit": {
                    "type": "string",
                    "enum": [ "celsius", "fahrenheit" ],
                    "description": "The temperature unit to use. Infer this from the specified location."
                }
            },
            "required": [ "location" ]
        }
        """u8.ToArray())
);

Implementing the Chat Loop

The heart of our chatbot is the conversation loop. Here's how we handle the interaction:

Initialize the conversation with a user message
Process the message using the DeepSeek model
Handle any tool calls requested by the model
Continue the conversation based on the model's response

var messages = new List<ChatMessage>()
{
    new UserChatMessage("What's the weather like in San Francisco?"),
};

bool requiresAction;

do
{
    requiresAction = false;
    var useToolCall = messages.Last() is not ToolChatMessage;
    var completion = useToolCall
        ? await chatClient.CompleteChatAsync(messages, new () { Tools = {getCurrentWeatherTool}})
        : await chatClient.CompleteChatAsync(messages);

    // Handle different completion scenarios
    switch (completion.Value.FinishReason)
    {
        case ChatFinishReason.Stop:
            {
                messages.Add(new AssistantChatMessage(completion));
                Console.WriteLine(completion.Value.Content[0].Text);
                break;
            }
        case ChatFinishReason.ToolCalls:
            {
                // Handle tool calls
                // ...
            }
        // Handle other cases
    }
} while (requiresAction);

Implementing the Weather Functions

For this example, we've implemented simple placeholder functions for getting weather information:

static string GetCurrentWeather(string location, string unit = "celsius")
{
    // Call the weather API here
    return $"The weather in {location} is 72 degrees {unit}.";
}

In a production environment, you'd want to replace these with actual API calls to weather and location services.

Run the Chatbot

Now that we've set up our chatbot, we can run it and see the output:

Building a Multi-LLM Profanity Detector in C# using StepWise

Xiaoyun Zhang — Sat, 04 Jan 2025 13:42:22 +0000

This post demonstrates how to create a robust profanity detection workflow using multiple Large Language Models (LLMs) in C# with the StepWise framework. The complete code is available in ProfanityDetector.cs. You can visit this blog post to learn more about StepWise.

Profanity Detector Overview

The profanity detection workflow consists of the following steps:

Get input text from the user
Send the text to three different AI models for analysis
Collect votes from all models
Make a final determination based on majority voting

What makes this workflow interesting is that it runs multiple AI models in parallel and uses a voting system to increase reliability. Some models might be more sensitive to certain types of content than others, so using multiple models helps provide a more balanced assessment.

You can try-out this workfow at the online demo by clicking the below button

Create a new StepWise project

To create a new StepWise project, you can use the StepWise template provided by StepWise. You can create a new StepWise project using the following command from dotnet-cli:

// install the StepWise template
dotnet new -i LittleLittleCloud.StepWise.Template

// create a new StepWise project
dotnet new stepwise-console -n PrepareDinnerProject

Define the Workflow

In StepWise, we'll define our workflow as a C# class where each step is represented by a public async method with the Step attribute. Let's start by creating the ProfanityDetector class:

public class ProfanityDetector
{
    private readonly ChatClient _chatClient;

    public ProfanityDetector()
    {
        // Here we use deepseek v3 api for LLM inference because it's cost-effective.
        // You can also replace it with other LLM providers like OpenAI, Claude, etc
        _chatClient = ChatClientProvider.Instance.CreateDeepSeekV3();
    }

    [Step(description: """
        ## ProfanityDetector
        This workflow helps you determine if a given text contains rude or profane language.
        """)]
    public async Task<string> README()
    {
        return "README is Completed";
    }
}

Adding User Input

The first step is to get input from the user. We'll use StepWise's UI input capability:

[StepWiseUITextInput(description: "Please provide the text you want to check")]
public async Task<string?> Input()
{
    return null;
}

Implementing AI Analysis

Next, we'll add three parallel AI analysis steps. Each step will use the same LLM but with independent executions to provide different perspectives:

[Step(description: "Detect profane language using AI1")]
[DependOn(nameof(Input))]
public async Task<bool> AI1(
    [FromStep(nameof(Input))] string input)
{
    var prompt = $"""
        Please determine if the following text contains rude or profane language, 
        or is inappropriate to be shared in public:
        {input}

        Answer with 'yes' or 'no'
        """;

    var systemMessage = new SystemChatMessage(prompt);
    var response = await _chatClient.CompleteChatAsync(systemMessage);

    return response.Value.Content[0].Text.ToLower().Contains("yes");
}

The AI2 and AI3 methods follow the same pattern. Each AI step:

Depends on the Input step
Takes the input text as a parameter
Sends a prompt to the LLM
Returns a boolean indicating whether profanity was detected

Implementing the Voting System

Finally, we'll add the voting step that collects results from all three AI models and makes a final determination:

[Step(description: "Collect votes and determine the result")]
[DependOn(nameof(AI1))]
[DependOn(nameof(AI2))]
[DependOn(nameof(AI3))]
public async Task<string> Voting(
    [FromStep(nameof(AI1))] bool ai1,
    [FromStep(nameof(AI2))] bool ai2,
    [FromStep(nameof(AI3))] bool ai3)
{
    var votes = new[] { ai1, ai2, ai3 };
    var yesVotes = votes.Count(v => v);
    var noVotes = votes.Count(v => !v);

    return yesVotes > noVotes 
        ? "The text contains rude or profane language" 
        : "The text does not contain rude or profane language";
}

The Voting step:

Depends on all three AI analysis steps
Takes the results as parameters
Uses simple majority voting to make the final determination
Returns a human-readable result string

Add ProfanityDetector workflow to StepWise server

After the ProfanityDetector workflow is defined, we can add it to the StepWise server using the StepWiseClient instance. This will allow the workflow to be executed in the StepWise UI.

// Program.cs
// ...configure the StepWise server
var stepWiseClient = host.Services.GetRequiredService<StepWiseClient>();
var workflow = new ProfanityDetector();
stepWiseClient.AddWorkflow(Workflow.CreateFromInstance(workflow));

// Wait for the host to shutdown
await host.WaitForShutdownAsync();

The code above creates a StepWise server and adds the PrepareDinner workflow to the server. The server will be hosted on http://localhost:5123 by default. You can visit the URL to see the StepWise UI and execute the workflow.

Conclusion

This post demonstrated how to build a multi-LLM profanity detector in C# using the StepWise framework. By combining multiple AI models and using a voting system, we can create a more robust and reliable profanity detection workflow.

Create PrepareDinner workfow in StepWise

Xiaoyun Zhang — Mon, 30 Dec 2024 02:09:58 +0000

This post shows how to create a stepwise workflow in C# using PrepareDinner as an example. The complete code is available at PrepareDinner.cs. You can visit this blog post to learn more about StepWise.

You can try-out this workfow at the online demo by clicking the below button

Prepare dinner overview

The dinner preparation workflow consists of the following steps:

Where we can notice that some steps can be executed in parallel, such as CutVegetables() and BoilWater(), and some steps depend on the previous steps, such as CookVegetables() and CookMeat(). When all the steps are completed, the ServeDinner() step can be executed. This dependency can be automatically resolved by the StepWise engine after the prepare dinner workflow is defined.

Create a new StepWise project

To create a new StepWise project, you can use the StepWise template provided by StepWise. You can create a new StepWise project using the following command from dotnet-cli:

// install the StepWise template
dotnet new -i LittleLittleCloud.StepWise.Template

// create a new StepWise project
dotnet new stepwise-console -n PrepareDinnerProject

Define workflow

In StepWise, every workflow is represented as a basic C# class and every steps under that workflow is mapping to a public async method with Step Attribution.

Therefore, we can start by defining a PrepareDinner class with the following steps:

// file: PrepareDinner.cs
public class PrepareDinner
{
    // ...
}

Next, we can start adding the steps to the PrepareDinner class. We will add Start step first.

// file: PrepareDinner.cs
public class PrepareDinner
{
    [Step]
    public async Task<DateTime> Start()
    {
        Console.WriteLine("Start preparing dinner at " + DateTime.Now);

        return DateTime.Now;
    }
}

The Start step is a simple step that prints the current time to the console and returns the current time. This step is the first step in the workflow and does not depend on any other steps.

One thing to note is the Step attribute on the Start method. This attribute tells the StepWise engine that this method is a step in the workflow.

Next, we can add CutVegetables and BoilWaters steps to the PrepareDinner class. These steps will depend on the Start step, which is marked by the DependOn attribute.

// file: PrepareDinner.cs
public class PrepareDinner
{
    // ... previous steps

    [Step]
    [DependOn(nameof(Start))]
    public async Task<string> CutVegetables()
    {
        Console.WriteLine("Cutting vegetables...");
        await Task.Delay(2000);
        return "Vegetables are cut in 2 seconds";
    }

    [Step]
    [DependOn(nameof(Start))]
    public async Task<string> BoilWater()
    {
        Console.WriteLine("Boiling water...");
        await Task.Delay(3000);
        return "Water is boiled in 3 seconds";
    }
}

Next, we can add CookVegetables step. We want the CookVegetables step to depend on the CutVegetables step. This can be done by adding the DependOn attribute to the CookVegetables method. Other than the dependency, we also want to use the result of the CutVegetables step in the CookVegetables step. This can be done by adding the FromStep attribute to the CookVegetables method.

// file: PrepareDinner.cs
public class PrepareDinner
{
    // ... previous steps

    [Step]
    [DependOn(nameof(CutVegetables))]
    [DependOn(nameof(BoilWater))]
    public async Task<string> CookVegetables(
        [FromStep(nameof(CutVegetables)) string cutVegetablesResult)
    {
        Console.WriteLine("Cooking vegetables...");
        await Task.Delay(3000);

        var result = $"Vegetables are cooked in 3 seconds. {cutVegetablesResult}";

        return result;
    }
}

Finally, we can add the ServeDinner step to the PrepareDinner class. This step will depend on the CookVegetables and CookMeat steps. We can add the DependOn attribute to the ServeDinner method to specify the dependencies. We also want to use the results of the CookVegetables and CookMeat steps in the ServeDinner step. This can be done by adding the FromStep attribute to the ServeDinner method.

// file: PrepareDinner.cs
public class PrepareDinner
{
    // ... previous steps

    [Step]
    [DependOn(nameof(CookVegetables))]
    [DependOn(nameof(CookMeat))]
    public async Task<string> ServeDinner(
        [FromStep(nameof(CookVegetables)) string cookVegetablesResult,
        [FromStep(nameof(CookMeat)) string cookMeatResult)
    {
        Console.WriteLine("Serving dinner...");
        await Task.Delay(1000);

        var result = $"Dinner is served in 1 second. {cookVegetablesResult} {cookMeatResult}";

        return result;
    }
}

At this point, we have defined the PrepareDinner workflow with all the necessary steps. The workflow is ready to be executed.

Add PrepareDinner workflow to StepWise server

After the PrepareDinner workflow is defined, we can add it to the StepWise server using the StepWiseClient instance. This will allow the workflow to be executed in the StepWise UI.

// Program.cs
// ...configure the StepWise server
var stepWiseClient = host.Services.GetRequiredService<StepWiseClient>();
var prepareDinner = new PrepareDinner();
stepWiseClient.AddWorkflow(Workflow.CreateFromInstance(prepareDinner));

// Wait for the host to shutdown
await host.WaitForShutdownAsync();

Understanding Event-Based and Process-Based Workflows

Xiaoyun Zhang — Mon, 16 Dec 2024 08:09:58 +0000

When managing tasks and automating processes, whether in software development or business operations, two primary workflow types come into play: event-based and process-based workflows. Recognizing their differences helps in choosing the right approach for your needs.

Event-Based Workflows

Definition:

Event-based workflows are driven by external events or triggers, causing specific actions to occur often asynchronously. These workflows are ideal for systems requiring adaptability and quick response to real-time stimuli.

Simple Example:

Consider a web page where a user clicks the "Submit" button. This action initiates several asynchronous tasks:

Validating the form data.
Sending the data to a server.
Displaying a confirmation message to the user.

Real-World Applications:

Social Media Reactions: Liking or commenting on a post triggers notifications and updates for users in real-time.
Online Notification Systems: Such as whenever a new email is received or a milestone is achieved on a software platform.

Analogy with Reactive Programming:

Think of a restaurant where orders come in randomly. Chefs (reactive system) start preparing each order as it arrives, adapting to the inbound requests without a fixed order, akin to event-based workflows reacting to triggers.

Process-Based Workflows

Definition:

These workflows follow a strict, predefined sequence where each task is executed in order. This structured approach is beneficial for repeatable operations requiring consistency and control.

Simple Example:

In a document approval process, a document undergoes a clear sequence:

Submission by the originator.
Initial review by a manager.
Subsequent approval by a department head.
Final approval by an executive.

Real-World Applications:

Loan Processing Systems: Loans pass through set stages of assessment and approval.
Manufacturing Assembly Lines: Products are assembled through a fixed sequence to ensure quality.

Analogy with Imperative Programming:

Similar to following a recipe for baking a cake, each step (mix ingredients, bake, cool) must be performed in order for successful completion.

Comparison Chart

Feature	Event-Based Workflow	Process-Based Workflow
Nature	Asynchronous, flexible	Sequential, structured
Trigger	Events	Predefined process steps
Flow	Non-linear	Linear
Programming Style	Reactive Programming	Imperative Programming
Use Cases	Real-time notifications, dynamic interactions	Structured approvals, manufacturing processes

Visual Aids

Event-Based Workflow Diagram:

[User Clicks Button]
        |
    [Trigger Event]
    /      |       \
[Action 1][Action 2][Action 3]
(independent actions)

Process-Based Workflow Diagram:

[Start] -> [Step 1] -> [Step 2] -> [Step 3] -> [End]
(Sequential steps)

Conclusion:

Event-based workflows excel in environments that demand flexibility and quick adaptation to external triggers, such as real-time data operations, while process-based workflows are well-suited for tasks that require control and adherence to a linear progression, such as compliance and manufacturing processes. Understanding these distinctions helps in designing systems that optimize efficiency and response according to the specific context and requirements.

StepWise: A code-first, event-driven workflow framework for .NET

Xiaoyun Zhang — Thu, 07 Nov 2024 23:41:53 +0000

StepWise is an Open-Source .NET framework which assists you to code, visualize and execute event-base workflow. It is designed to help you build complex workflows in a simple and efficient way. StepWise comes with the following key features:

Code-First: Define workflows using C# code in your project.
WebUI Visualize and execute workflows from your favorite browser using StepWise WebUI.
Event-Driven: Execute steps in parallel and resolve dependencies automatically.
AI-Powered: Work with Geeno, a built-in AI assistant in StepWise WebUI to help you run and analyze workflows with ease.

To learn more about StepWise, checkout the project on Github. You can also try out the online samples by clicking the try-live-demo button below.

Building an AI-Powered Equation Solver with GPT-4o, AutoGen.Net and StepWise

Xiaoyun Zhang — Mon, 07 Oct 2024 05:33:05 +0000

Building an AI-Powered Equation Solver with AutoGen, GPT-4, and StepWise

In this post, we'll walk through building an AI-powered equation solver that can extract equations from images and solve them using the power of GPT-4. We'll be using AutoGen, OpenAI's GPT-4o, and the StepWise framework to create a workflow to resolve equation from user input image.

The complete source code for this project is available on GitHub

The Problem

Mathematical equations are everywhere, from academic papers to whiteboards in meeting rooms. But what if you could simply take a picture of an equation and have an AI solve it for you? That's exactly what we're going to build!

The Solution

We'll create a .NET application that:

Accepts an image input containing an equation
Uses GPT-4's vision capabilities to extract the equation
Converts the equation to LaTeX format
Solves the equation using GPT-4

Let's dive into the components and workflow of our solution.

Key Components

AutoGen: A framework for building AI agents and workflows
OpenAI's GPT-4o: A powerful language model with vision capabilities
StepWise: A framework for creating, visualizing and executing workflows

The Workflow

Our equation solver follows these steps:

Image Input: Accept an image containing an equation
API Key Validation: Ensure we have a valid OpenAI API key
Image Validation: Confirm the image contains exactly one equation
Equation Extraction: Extract the equation from the image and convert it to LaTeX
Equation Solving: Solve the extracted equation

Overview

Let's look at each step in more detail.

1. Image Input

We use StepWise's StepWiseUIImageInput attribute to create a user interface for image input:

[StepWiseUIImageInput(description: "Please provide the image of the equation")]
public async Task<StepWiseImage?> InputImage()
{
    return null;
}

2. API Key Validation

We provide two options for the OpenAI API key: environment variable or manual input:

[StepWiseUITextInput(description: "Please provide the OpenAI API key if env:OPENAI_API_KEY is not set, otherwise leave empty and submit")]
public async Task<string?> OpenAIApiKey()
{
    return null;
}

[Step(description: "Validate the OpenAI API key")]
public async Task<string> ValidateOpenAIApiKey(
    [FromStep(nameof(OpenAIApiKey))] string apiKey)
{
    // ... (key validation logic)
}

3. Image Validation

We use GPT-4 to confirm that the image contains exactly one equation:

[Step(description: "Valid image input to confirm it contains exactly one equation")]
public async Task<bool> ValidateImageInput(
    [FromStep(nameof(InputImage))] StepWiseImage image)
{
    // ... (image validation logic)
}

4. Equation Extraction

GPT-4's vision capabilities are used to extract the equation and convert it to LaTeX:

[Step(description: "Extract the equation from the image into LaTeX format")]
public async Task<string?> ExtractEquationFromImage(
    [FromStep(nameof(ValidateImageInput))] bool valid,
    [FromStep(nameof(InputImage))] StepWiseImage image)
{
    // ... (equation extraction logic)
}

5. Equation Solving

Finally, we use GPT-4 to solve the extracted equation:

[Step(description: "Solve the equation")]
public async Task<string?> SolveEquation(
    [FromStep(nameof(InputImage))] StepWiseImage image,
    [FromStep(nameof(ExtractEquationFromImage))] string equation)
{
    // ... (equation solving logic)
}

Putting It All Together

The magic happens in the EquationSolver class, which combines all these steps into a cohesive workflow. We use AutoGen to create an AI agent with GPT-4, and StepWise to manage the workflow:

public class EquationSolver
{
    private string _apiKey;
    private IAgent _agent;

    // ... (steps implementation)
}

The main application sets up a web server and initializes the workflow:

var host = Host.CreateDefaultBuilder()
    .ConfigureWebHostDefaults(webBuilder =>
    {
        webBuilder.UseUrls("http://localhost:5123");
    })
    .UseStepWiseServer()
    .Build();

await host.StartAsync();

var stepWiseClient = host.Services.GetRequiredService<StepWiseClient>();

var instance = new EquationSolver();
var workflow = Workflow.CreateFromInstance(instance);
stepWiseClient.AddWorkflow(workflow);

await host.WaitForShutdownAsync();

Conclusion

By combining the power of AutoGen, GPT-4o, and StepWise, we've created an equation solver prototype that can extract and solve equations from images. This approach demonstrates the potential of AI in simplifying complex tasks and opens up possibilities for similar applications in various fields.

Freeloading gpt-4o, llama, and many more llms on Github Model in AutoGen.Net

Xiaoyun Zhang — Wed, 11 Sep 2024 22:44:52 +0000

Github recently introduced Github Models, which allows us to try and test from a wide range of model types, sizes, and specializations FOR FREE. Considering the wide range of available models (gpt-4o, gpt-4o-mini, llama-3.1 8B, llama-3.1 405B), maybe now it's the second best time to have a github account. In this blog post, we'll explore how to freeload these models using AutoGen.Net.

Why GitHub Models?

Free Access: All you need is a GitHub account to get started.
Diverse Model Selection: From GPT-4 to Llama-3.1, GitHub offers a variety of models to suit different needs.

Getting Started

Create a GitHub Account: If you don't already have one, sign up for a free GitHub account.
Generate an API Key: Navigate to Settings > Developer settings > Token (classic) and generate a new token.
Step Up Your Environment: Save your API key securely in your local environment.

Integrating with AutoGen.Net

Here's a simple example of how to use the Llama-3.1 405B model with AutoGen.Net:

Create a dotnet console app, and add package `AutoGen 0.2.0`

dotnet new console -n GithubLLMFreeLoader
cd GithubLLMFreeLoader
dotnet add package AutoGen --version 0.2.0

Create a simple chat sample using `llama-3.1-405B`

using AutoGen.AzureAIInference;
using AutoGen.AzureAIInference.Extension;
using AutoGen.Core;
using Azure.AI.Inference;
var githubKey = Environment.GetEnvironmentVariable("GH_API_KEY") ?? "YOUR_KEY";
var endpoint = "https://models.inference.ai.azure.com";
var client = new ChatCompletionsClient(new Uri(endpoint), new Azure.AzureKeyCredential(githubKey));

var agent = new ChatCompletionsClientAgent(
    chatCompletionsClient: client,
    name: "assistant",
    modelName: "Meta-Llama-3.1-405B-Instruct")
    .RegisterMessageConnector()
    .RegisterPrintMessage();

await agent.SendAsync("Tell me a long joke");

The code snippet demostrates how to

Connect to Github Model
Create an AutoGen.Net agent using llama-3.1-405B
Start a conversation with agent.

Limitation

Okay, here's the not-so-great part: there's a limit on how much you can use this service. Since it's free, they don't let you go crazy with it. It's fine for trying things out or playing around, but it's not going to cut it if you need it for serious, heavy-duty work. But hey, it's free, so we can't really complain, right?

Tool call with local model using Ollama and AutoGen.Net

Xiaoyun Zhang — Mon, 29 Jul 2024 21:56:50 +0000

Ollama, starting from version 0.3.0, supports tool calls with popular models like llama 3.1 and mistral. This functionality allows LLM models to interact with external functions by incorporating tools within their prompts.

In this blog, we’ll demo the usage of tool call with local model by building a weather report agent using AutoGen.Net and Ollama. The weather report agent will query weather information from a dummy function and generate final weather information based on tool call result

Requirements

To run this example, ensure you have:

ollama >= 0.3.0
dotnet sdk 8.0

Step 1: Create a .NET Console App

First, create a new .NET console application:

dotnet new console -n weather-report-agent

Step 2: Add AutoGen.Net Packages

Next, add the necessary AutoGen.Net packages:

dotnet add package AutoGen
dotnet add package AutoGen.SourceGenerator # for type-safe function contract generation

Step 3: Add the `WeatherReportTool` Class

Create a dummy tool class to fetch the weather report for a given city. The API in this class can be accessed by the LLM through tool calls.

public partial class WeatherReportTool
{
    /// <summary>
    /// Get the weather report for the given city
    /// </summary>
    /// <param name="city">city</param>
    /// <returns>weather report</returns>
    [Function]
    public async Task<string> GetWeatherReport(string city)
    {
        return $$"""
        {
            "city": "{{city}}",
            "temperature": "25°C",
            "weather": "Sunny"
        }
        """;
    }
}

The [Function] attribute allows AutoGen.SourceGenerator to create a function contract from the structured comment.

Step 4: Create `WeatherReportTool` and `FunctionCallMiddleware`

Create a FunctionCallMiddleware with function definitions and function maps, then register it to an agent for consumption.

var weatherReportTool = new WeatherReportTool();
var weatherReportToolMiddleware = new FunctionCallMiddleware(
    functions: new[] { weatherReportTool.GetWeatherReportFunctionContract },
    functionMap: new Dictionary<string, Func<string, Task<string>>>
    {
        [nameof(weatherReportTool.GetWeatherReport)] = weatherReportTool.GetWeatherReport
    });

Step 5: Create `OpenAIChatAgent` and Connect to Ollama

Since Ollama supports the OpenAI-compatible completion API, we can use OpenAIChatAgent to connect to the Ollama service as a third-party endpoint.

First, create a CustomHttpClientHandler to redirect all requests to the Ollama server:

public sealed class CustomHttpClientHandler : HttpClientHandler
{
    private readonly string _modelServiceUrl;

    public CustomHttpClientHandler(string modelServiceUrl)
    {
        _modelServiceUrl = modelServiceUrl;
    }

    protected override Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        request.RequestUri = new Uri($"{_modelServiceUrl}{request.RequestUri.PathAndQuery}");
        return base.SendAsync(request, cancellationToken);
    }
}

Then create an OpenAIChatAgent and connect it to the Ollama endpoint:

var openAIClient = new OpenAIClient("api-key", new OpenAIClientOptions
{
    HttpClientHandler = new CustomHttpClientHandler("https://ollama.com/api/")
});

var model = "llama3.1";
var agent = new OpenAIChatAgent(
    openAIClient: openAIClient,
    name: "assistant",
    modelName: model,
    systemMessage: "You are a weather assistant.",
    seed: 0)
    .RegisterMessageConnector() // convert AutoGen message to OpenAI message
    .RegisterMiddleware(weatherReportToolMiddleware) // register function call middleware
    .RegisterMiddleware(async (msgs, option, agent, ct) => {
        var reply = await agent.GenerateReplyAsync(msgs, option, ct);
        if (reply is ToolCallAggregateMessage toolCallMessage)
        {
            var chatHistory = msgs.Append(toolCallMessage);
            return await agent.GenerateReplyAsync(chatHistory, option, ct);
        }

        return reply;
    })
    .RegisterPrintMessage(); // print message to console

Final Step: Ask for a Weather Report!

Once the weather report agent is created, you can ask it questions like "What's the weather in New York?"

var task = """
What is the weather in New York?
""";

await agent.SendAsync(task);

The response will be something like:

--------------------
TextMessage from assistant
--------------------
The current temperature in New York is 25°C (77°F), and it's a sunny day.

Conclusion and Further Reading

In this blog, we demonstrated how to enable tool calls using local models with Ollama and AutoGen.Net.

Thanks for reading! You can find the complete sample code in this repository.

How to write release note with LLM agents

Xiaoyun Zhang — Sun, 14 Jul 2024 07:29:47 +0000

TL;DR

This blog shows how to simplify the process of writing release notes using issue-helper and gpt agent in Agent ChatRoom. The issue-helper pulls issues in a milestone and gpt generates a release note based on these issues.

What is Agent ChatRoom

Agent ChatRoom is a multi-agent platform built on top of AutoGen.Net and Orleans. It comes with a web-based UI that provides built-in support for multi-agent group chat.

A release note on github usually includes the following sections

Improvements
Bug Fix
New Feature
API Break Change
…

To write a release note like the one above, the normal steps are

collect all issues in a milestone and classify these issues into corresponding sections
complete each section by summarizing the issues.

Manually completing these steps can be time-consuming. However, with the help of issue-helper and gpt agent, we no longer need to collect issues from github and summarize them one by one. Instead, we can ask issue-helper to find all issues in a milestone, and ask gpt to generate release note based on found issues.

In the rest of the blog, we will show a step-by-step guide on how to write release note using Agent ChatRoom, issue-helper and gpt.

Step 1: Install Agent ChatRoom and configure agents

Agent Chatroom is published as a dotnet tool package on nuget.org. To install Agent Chatroom, first make sure you install dotnet 8.x SDK, then run the following command:

dotnet tool install --global ChatRoom.Client --version 0.4.2

To start a chatroom, you need to provide a configuration file which includes credentials like openai-api-key, along with settings for the issue-helperand gpt agents. To simplify the process, you can begin with an empty configuration template by using the following command:

# create configuration from chatroom_empty template and save it to config.json
# You can also list all available templates using list-templates command
chatroom create -t chatroom_empty -o config.json

The create command will also generate a JSON schema file compatible with modern code editors like VS Code, providing code intellisense to assist you in completing the configuration.

To add issue-helper and gpt agent to the chatroom, you need to add chatroom_github_configuration and chatroom_openai_configuration section to config.json

Once you complete the configuration, save the file and start the chatroom using the following command. You will see output similar to the example below, which shows the url of web-based UI.

chatroom run -c config.json

Step 2: Create a group chat with issue helper and gpt

To create a group chat with agents, click on the (+) button on the top of the channel panel, and select the agents you want to add. In this case, both issue-helper and gpt are added for issue retrieval and writing release note. Other than that, we also select DynamicGroupChat which selects the next speaker using LLM as group chat orchestrator.

After clicking on save button, we can see the group chat ReleaseChannel is created and we can start asking agents to write release note.

Step 3: Ask issue-helper to retrieve issues in milestone

In this step, we will ask issue-helper to find all completed issues in milestone: 0.4.2, which is the most recent completed milestone in Agent-Chatroom.

As we can see, the issue-helper returns a list of issues and their summary, which is exactly what we need to write release note. Except for this time, we no longer need to pull them manually from github.

Final Step: Ask gpt to write release note

In the final step, we can ask gpt to create a release note. Because gpt and issue-helper are in the same group chat, they can share the context and reuse the issues returned from issue-helper.

Conclusion

In this blog, we show how to write release note using gpt and issue-helper agents in Agent ChatRoom.

Feedback and comments are welcome. If you found this blog useful, please give Agent ChatRoom a star on github.
Happy coding!

DEV Community: Xiaoyun Zhang

LLM Canvas: The Story Behind

The Beginning: The Struggle with Linear Logs

From Idea to Reality: The Seven-Day Challenge

Core Value: Managing Conversations Like Git

Looking Forward: From Tool to Interaction Paradigm

Some Key Takeaways

Final Words

Build a Weather Chatbot with DeepSeek v3 and OpenAI SDK: A Step-by-Step Guide

Prerequisites

Create a New Console Project

Setting Up the DeepSeek Client

Defining Function Tools

Implementing the Chat Loop

Implementing the Weather Functions

Run the Chatbot

Building a Multi-LLM Profanity Detector in C# using StepWise

Profanity Detector Overview

Create a new StepWise project

Define the Workflow

Adding User Input

Implementing AI Analysis

Implementing the Voting System

Add ProfanityDetector workflow to StepWise server

Conclusion

Create PrepareDinner workfow in StepWise

Prepare dinner overview

Create a new StepWise project

Define workflow

Add PrepareDinner workflow to StepWise server

Understanding Event-Based and Process-Based Workflows

Event-Based Workflows

Process-Based Workflows

Comparison Chart

Visual Aids

StepWise: A code-first, event-driven workflow framework for .NET

Building an AI-Powered Equation Solver with GPT-4o, AutoGen.Net and StepWise

Building an AI-Powered Equation Solver with AutoGen, GPT-4, and StepWise

The Problem

The Solution

Key Components

The Workflow

Overview

1. Image Input

2. API Key Validation

3. Image Validation

4. Equation Extraction

5. Equation Solving

Putting It All Together

Conclusion

Freeloading gpt-4o, llama, and many more llms on Github Model in AutoGen.Net

Why GitHub Models?

Getting Started

Integrating with AutoGen.Net

Create a dotnet console app, and add package AutoGen 0.2.0

Create a simple chat sample using llama-3.1-405B

Limitation

Tool call with local model using Ollama and AutoGen.Net

Requirements

Step 1: Create a .NET Console App

Step 2: Add AutoGen.Net Packages

Step 3: Add the WeatherReportTool Class

Step 4: Create WeatherReportTool and FunctionCallMiddleware

Step 5: Create OpenAIChatAgent and Connect to Ollama

Final Step: Ask for a Weather Report!

Conclusion and Further Reading

How to write release note with LLM agents

TL;DR

What is Agent ChatRoom

Step 1: Install Agent ChatRoom and configure agents

Step 2: Create a group chat with issue helper and gpt

Step 3: Ask issue-helper to retrieve issues in milestone

Final Step: Ask gpt to write release note

Conclusion

Create a dotnet console app, and add package `AutoGen 0.2.0`

Create a simple chat sample using `llama-3.1-405B`

Step 3: Add the `WeatherReportTool` Class

Step 4: Create `WeatherReportTool` and `FunctionCallMiddleware`

Step 5: Create `OpenAIChatAgent` and Connect to Ollama