DEV Community: Yogesh Choudhary

Using RAG to Build Your IDE Agents

Yogesh Choudhary — Tue, 18 Jun 2024 21:59:22 +0000

In the post-GPT revolution era, many of us developers have started using LLM-enabled tools in our development workflows. Nowadays, you can complete new and complex development tasks in a short span with the help of theses LLM tools when used correctly.

Until you start using them for anything related to new APIs or SDKs or their latest version, this is the place where they fall short.

Fixing the Shortcomings with RAG

At CommandDash (formerly Welltested), our team has been working in code generation. Like other organizations in this field, we recognized the challenges and have actively developed solutions.

Therefore, with the Dash Agent Framework, we initiated building a robust RAG system to tackle these issues from inception stage itself.

What is RAG?

Retrieval-Augmented Generation (RAG) is a powerful technique that combines the capabilities of LLMs with relevant references to enrich the responses. These relevant references are typically derived from external knowledge sources like document databases and more.

RAG significantly enhances the capabilities of LLMs, especially when working with new packages and frameworks. By accessing up-to-date information from documentation, code examples, and other sources, RAG-based LLMs can:

Provide accurate and contextual responses: Instead of relying solely on pre-trained data, LLMs can access the latest documentation and code examples to provide accurate and relevant information.
Adapt to evolving technologies: As APIs and SDKs evolve, RAG can keep pace by constantly updating its knowledge base from official sources.

In this blog, we will build a powerful IDE agent for PandasAI using Dash Agent. Then later on, we'll understand how using RAG can significantly improve LLM responses.

Building PandasAI Agent

PandasAI is a Python platform that makes it easy to ask questions about your data in a natural language. It integrates generative artificial intelligence capabilities into pandas to allow you to extract insights effortlessly.

Now that you're familiar with PandasAI. Let's start our journey to build our own PandasAI agent. The job of this agent will be to assist developers in building and integrating PandasAI code efficiently.

Prerequisite Steps

1. Install Dart
Dash Agent is built upon dart language. If you haven't already, follow the official Flutter installation instructions here.

2. Install dash_cli
Now, install the dash_cli command line tool that enables you to create and publish your agents at the CommandDash marketplace. Open your terminal and run the following command:

dart pub global activate dash_cli

Create PandasAI Project

Next, you will create the pandas_ai project. This is the place where you will define your agent configurations. Run the following command in the terminal:

dash_cli create pandas_ai

This will create a dash agent project that contains the template code agent building. Then, open the project in your preferred IDE where the flutter extension is installed.

Adding Agent Data Sources

The core of a RAG-based agent lies in its knowledge base, known as data sources. These sources provide the agent with context and information to understand and respond to user requests.

For our PandasAI agent, we will gather data from the following sources:

Official PandasAI Documentation: https://docs.pandas-ai.com, https://pandasai-docs.readthedocs.io/en/latest
Official Examples and Issues shared by PandasAI team: https://github.com/sinaptik-ai/pandas-ai
Other Open Source Examples: CSV Chatbot, GroqMultiCSVChatPandasAI , MutipleCSVChatllama3Pandasai, PandasAI-Tutorials

Navigate to the lib/data_sources file in your project and replace the existing code with:

import 'package:dash_agent/data/datasource.dart';
import 'package:dash_agent/data/filters/filter.dart';
import 'package:dash_agent/data/objects/file_data_object.dart';
import 'package:dash_agent/data/objects/project_data_object.dart';
import 'package:dash_agent/data/objects/web_data_object.dart';

// Indexes all the documentation related data
class DocsDataSource extends DataSource {
  @override
  List<FileDataObject> get fileObjects => [];

  @override
  List<ProjectDataObject> get projectObjects => [];

  @override
  List<WebDataObject> get webObjects => [
        WebDataObject.fromSiteMap('https://docs.pandas-ai.com/sitemap.xml'),
        WebDataObject.fromSiteMap(
            'https://www.xml-sitemaps.com/download/pandasai-docs.readthedocs.io-a2835e7d4/sitemap.xml?view=1'),
      ];
}

// Indexes all the example code and issues related data
class ExampleDataSource extends DataSource {
  final accessToken = 'your_personal_github_access_token';
  @override
  List<FileDataObject> get fileObjects => [];

  @override
  List<ProjectDataObject> get projectObjects => [];

  @override
  List<WebDataObject> get webObjects => [
        WebDataObject.fromGithub(
            'https://github.com/sinaptik-ai/pandas-ai', accessToken,
            codeFilter: CodeFilter(pathRegex: r'^examples\/.*')),
        WebDataObject.fromGithub(
            'https://github.com/ismailtachafine/PandasAI-CSV-Analysis',
            accessToken,
            codeFilter: CodeFilter(pathRegex: r'.*\.py$')),
        WebDataObject.fromGithub(
            'https://github.com/kBrutal/CSV_ChatBot', accessToken,
            codeFilter: CodeFilter(pathRegex: r'.*\.py$')),
        WebDataObject.fromGithub(
            'https://github.com/InsightEdge01/GroqMultiCSVChatPandasAI',
            accessToken,
            codeFilter: CodeFilter(pathRegex: r'.*\.py$')),
        WebDataObject.fromGithub(
            'https://github.com/InsightEdge01/MutipleCSVChatllama3Pandasai',
            accessToken,
            codeFilter: CodeFilter(pathRegex: r'.*\.py$')),
        WebDataObject.fromGithub(
            'https://github.com/TirendazAcademy/PandasAI-Tutorials',
            accessToken,
            codeFilter: CodeFilter(pathRegex: r'.*\.py$')),
      ];
}

The above code shared the references of sources that need to be indexed both the documentation and examples. Apart from the sources link, you have also provided accessToken and codeFilter:

accessToken: During processing, the CommandDash server indexes data for WebDataObject.fromGithub via Github's official API. To fetch the data from the GitHub API efficiently, the personal github token is required and can be easily generated by visiting the tokens page.
CodeFilter: This filter enables the framework to selectively index the code files based on the regex shared. This is optional.

Note: Your Personal Access Token is very sensitive data. Please make sure not it share it with anyone or push it to any public source.

You can learn more about WebDataObject and associated properties in detail at CommandDash documentation.

Adding Agent System Prompt and Metadata

Next, you'll add system prompt and agent metadata to the AgentConfiguration class. Navigate to lib/agent.dart file and replace the existing code with:

import 'package:dash_agent/configuration/metadata.dart';
import 'package:dash_agent/data/datasource.dart';
import 'package:dash_agent/configuration/command.dart';
import 'package:dash_agent/configuration/dash_agent.dart';
import 'data_sources.dart';

class PandasAI extends AgentConfiguration {
  final docsDataSource = DocsDataSource();
  final exampleDataSource = ExampleDataSource();

  // Add the metadata information about PandasAI agent
  @override
  Metadata get metadata => Metadata(
      name: 'Pandas AI',
      avatarProfile: 'assets/logo.jpeg',
      tags: ['LLM Framework', 'Data Analysis']);

  // Add the systemPrompt for dash agent's commandless mode (also know as chat mode). 
  // System prompt is a key component for conversational-style agents. As it provides 
  // the initial context and guidance regarding the agent's purpose and functionality to the LLM.
  @override
  String get registerSystemPrompt =>
      '''You are a Pandas AI assistant inside user's IDE. PandasAI is a Python library that makes it easy to ask questions to your data in natural language.

      You will be provided with latest docs and examples relevant to user questions and you have to help them achieve their desired results. Output code and quote links and say I don't know when the docs don't cover the user's query.''';

  // Add the data sources that needs to indexed for RAG purposes.
  @override
  List<DataSource> get registerDataSources =>
      [docsDataSource, exampleDataSource];

  @override
  List<Command> get registerSupportedCommands => [];
}

The above code basically glues everything all together for the PandasAI agent - data source, metadata, system prompt, commands, etc. that are needed to build the dash agent.

For more details related to the AgentConfiguration please read the dash_agent framework.

Finally, head to bin/main.dart file and replace the existing code with:

import 'package:dash_agent/dash_agent.dart';
import 'package:pandas_ai/agent.dart';

/// Entry point used by the [dash-cli] to extract your agent
/// configuration during publishing.
Future<void> main() async {
  await processAgent(PandasAI());
}

That's it. Your agent is now configured and ready to be used. Next, you'll publish it so that it can be tested and shared with other devs as well.

Publishing the PandasAI Agent

You need to be logged in to dash_cli using GitHub auth to publish your agent. Run the following command in the terminal to login:

dash_cli login

Finally, run the following command in the terminal from the root folder of your pandas_ai project to publish the agent:

dash_cli publish

This will validate the configuration and if all looks good. It will schedule your agent publication. Once your agent is ready to be used. You will get an email confirming the successful publication and PandasAI will be visible in the CommandDash Marketplace:

What's Next

Congratulations! 🎉 Now you know how to create powerful agents using the Dash Agent framework. These agents leverage the power of RAG and LLMs. We're excited to see the innovative agents you'll build with Dash Agent.

Also, don't forget to try out the PandasAI agent, which is currently live on the CommandDash extension for VS Code. Check it out here.

Next up, we will see in our upcoming blog, how well PandasAI perform. Stay tuned!

Understanding Dash Agents: A LangChain Case Study

Yogesh Choudhary — Wed, 12 Jun 2024 18:23:51 +0000

In software development, integrating APIs and SDKs is crucial. Yet, the process can be tedious and time-consuming. Documentation, syntax complexities, and debugging can quickly become time-consuming roadblocks. Thankfully, this is where Dash Agents comes in to ease these headaches.

This blog post explores the capabilities of Dash Agents. It shows how they simplify the integration of APIs and SDKs for tools and frameworks like LangChain.

What is LangChain?

LangChain is a framework that allows users to integrate LLMs into their projects. It provides a comprehensive toolkit for:

Model Handling: It supports various LLMs and allows for easy integration of custom models, offering prompt templates and engineering tools.
Data Processing and Pipelines: LangChain provides tools for data preprocessing and augmentation, along with the ability to create custom pipelines with a modular design.
Deployment and Monitoring: It supports cloud deployment and scaling, offers evaluation metrics and real-time monitoring, and ensures security and compliance measures are in place.

You can learn more about the LangChain framework on their Official site.

Introducing LangChain Agent

Imagine having an intelligent assistant in your IDE. It's always there to help you integrate any LangChain-based code into your project. That's precisely what the LangChain Dash Agent offers.

Why You'll Love LangChain Agent:

Extensive API Knowledge: It is shared with vast knowledge of LangChain framework documentation and sample codes, allowing it to provide nearly accurate code and insights.
Contextual Awareness: It is designed to respond to queries with the understanding of your codebase and project context. Thus, capable of providing better assistance and solutions.
Saving Time During Coding: It generates the required code for you and if required debugs the code for you, saving you a lot of time. Thus, forget the days of tedious manual API docs searching and then coding only to realise that you need to fix a bug by searching multiple sources.

Install LangChain Agent

Let's start by setting up your agent. If you've already LangChain installed please continue to the next section.

1. Install CommandDash Extension

Head over to the VS Code Marketplace and search for "CommandDash." Install the extension.

2. Sign in to CommandDash

Open the extension once installed. You'll need to sign in and provide your Gemini key.

3. Install LangChain Agent

Then, go to the CommandDash Marketplace within your Extension.

Search for "LangChain" and install the agent.

Next, it's time to put the installed LangChain agent to use.

Getting Started with LangChain Agent

Finally, it is time to put the agent to work with a compelling use case: building a creative text generator in Python.

This program takes the starting phrases of the poem as input. Then, it uses an LLM to extract the user's writing style and generate the remaining part of the poem.

Start by activating the LangChain agent. In your CommandDash extension, type @ in the bottom text field and select "LangChain" from the dropdown:

This activates the agent within your current workspace.

For demo purposes, we will attempt to use OpenAI's latest model - "GPT-4o". Type the following query into the CommandDash text field:

"Generate the Python code to create an application to generate poem with the user-supplied initial phrases using OpenAI gpt-4o model"

The agent will analyze your request and will respond something like:

from langchain_openai import ChatOpenAI
from langchain_core.prompts import ChatPromptTemplate
from langchain_core.messages import HumanMessage, SystemMessage

os.environ["OPENAI_API_KEY"] = "YOUR_OPENAI_API_KEY"

def generate_poem(initial_phrases, model="gpt-4o"):
    """Generates a poem using OpenAI's GPT-4o model.

    Args:
        initial_phrases (list): A list of initial phrases to start the poem.
        model (str, optional): The OpenAI model to use. Defaults to "gpt-4o".

    Returns:
        str: The generated poem.
    """

    llm = ChatOpenAI(model=model)

    prompt_template = ChatPromptTemplate.from_messages(
        [
            SystemMessage(content="You are a helpful and creative poet. Generate a poem using the following initial phrases as inspiration: "),
            HumanMessage(content=" ".join(initial_phrases))
        ]
    )

    messages = prompt_template.format_messages()
    response = llm.invoke(messages)

    return response.content

if __name__ == "__main__":
    initial_phrases = input("Enter your initial phrases separated by commas: ").split(",")
    poem = generate_poem(initial_phrases)
    print("Generated Poem:\n\n", poem)

Explanation:
Some explanation about the code
...

In our case, we haven't asked for any API integration request but rather an app itself. Thus, the agent supplied us with the whole end-to-end Python code.

Copy the code to a blank Python file and read the generated code. Right now, you can also update the code as needed. For instance, the current generated code is missing the import os statement. So, I will manually add the missing import.

Tip: At times, if you ever feel like the solution isn't straightforward, or you want to explore the problem more deeply or find a quick fix. Feel free to continue conversing with the agent.

Finally, add your OpenAI API key and run the app by executing the updated code:

Closing Thoughts

LangChain agent is not all that CommandDash has to offer. We have more agents for various APIs and SDKs, such as Firebase Genkit, Pandas AI, Vercel AI, and more.

Additionally, we also allow developers like you to create public agents for your teams and other developers. If interested, do visit the Dash Agents Docs.

That concludes this blog post. Now it's time to try it out yourself and materialize your LangChain use case to code instantly with CommandDash. We always love to hear what you build!

Why We Call Saving the Magical Feature

Yogesh Choudhary — Thu, 19 Oct 2023 08:42:23 +0000

Recently, the entire team at Welltested AI engaged in a discussion to understand what according to us is the most notable feature of Welltested at the moment. And, everyone agreed upon welltested save unit which we also call the magic command :)

While we internally love it, quick analytics into our database showed us that most of our users are not yet using it! This led to the question: are we talking about it enough? haha.

So, we decided to write this blog to share with our users what is so special about Welltested's save feature and how to use it effectively. Let's dive into it.

Setup Your Project

For starters, download the flutter twitter clone project that we will be using. You can also use your project or download any other open-source project of your choice to follow along. The steps that we will follow will mostly remain the same.

Next, open the project in the IDE. After opening the project, we will need to head to Welltested on pub.dev. Then, follow the instructions available on the page to set up Welltested. Since the setup process is fairly straightforward and well-documented, you will not face any issues following along. The general steps include:

Getting the API key and adding it to the project
Adding the Welltested packages
Activating Welltested CLI in your terminal

Once done we are ready to generate and save the tests :)

Generate Unit Tests

To generate the tests. We'll need to add the @Welltested annotation to the methods or classes for which we need to generate tests. This will help Welltested pick methods for whom test generation is requested.

Open lib/helper/utility.dart file in the project. We will find Utility class with a bunch of static methods declared inside it:

For this blog, we will generate tests for the methods present in the Utility class only. Add the Welltested annotation at the top of the Utility class:

@Welltested()

This annotation above the Utility class enables Welltested to know that we're interested in generating tests for the methods defined in this class.

Next, we will generate the tests by running the following chain of commands in the terminal:

welltested generate unit -m getPostTime2 && 
welltested generate unit -m getDob && 
welltested generate unit -m getJoiningDate && 
welltested generate unit -m getUserName

Upon execution of the above commands, Welltested will start generating the tests for the methods mentioned after the -m parameter that have the @welltested annotation too - either directly or on the class containing the methods.

Open the generated tests located in the tests/helper/utility directory after the execution of the generate commands is finished. Take a look at the generated files. Congratulations! You have successfully generated tests for your code in less than 5 minutes. Isn't it great :)

Looking at the generated tests. You will see that some of the tests have errors:

But don't worry, Welltested has got you covered here. And this is where the Welltested's save command shines 😎

Welltested's Save Command

Welltested AI has the ability to constantly learn and improve as you interact with it. Since this is our first interaction with the AI, it may make mistakes in the initial tests as it hasn't yet learned about your project and your coding style. However, fixing this is simple.

You can just simply fix the broken generated tests and save the tests using Welltested's save command. Welltested will automatically learn from them and provide much smarter and error-free test cases in the future.

To understand this in more detail. Let's look at an example by updating and saving one of the earlier generated unit tests. Afterwards, we will use Welltested's generate command to generate the remaining tests again.

Open test/helper/utility/getPostTime2.welltested_test.dart file and update it with the following:

import 'package:flutter_test/flutter_test.dart';
import 'package:flutter_twitter_clone/helper/utility.dart';

void main() {
  group('Utility getPostTime2', () {

    // 1.
    // Older version
    // test('When date is null, should return empty string', () {
    //   final mockUtility = MockUtility();
    //   when(mockUtility.getPostTime2(null)).thenReturn('');
    //   expect(mockUtility.getPostTime2(null), '');
    // });
    test('When date is null, should return empty string', () {
      expect(Utility.getPostTime2(null), equals(''));
    });

    test('When date is empty, should return empty string', () {
      expect(Utility.getPostTime2(''), equals(''));
    });

    test('When date is not in correct format, should throw FormatException',
        () {
      expect(() => Utility.getPostTime2('invalid date'), throwsFormatException);
    });

    test('When date is in correct format, should return formatted date', () {
      String date = '2022-12-01T20:18:04';
      String expectedDate = '08:18 PM - 01 Dec 22';
      expect(Utility.getPostTime2(date), equals(expectedDate));
    });

    // 2.
    // Older version
    // test('When date is in correct format but in different timezone, should return formatted date in local timezone', () {
    //   final mockUtility = MockUtility();
    //   String date = '2022-12-01T13:18:04Z'; // This is 20:18 in GMT+7
    //   String expectedDate = '08:18 PM - 01 Dec 22';
    //   when(mockUtility.getPostTime2(date)).thenReturn(expectedDate);
    //   expect(mockUtility.getPostTime2(date), equals(expectedDate));
    // });
    test(
        'When date is in correct format but in different timezone, should return formatted date in local timezone',
        () {
      String date = '2022-12-01T14:48:04Z'; // This is 20:18 in IST
      String expectedDate = '08:18 PM - 01 Dec 22';
      expect(Utility.getPostTime2(date), equals(expectedDate));
    });
  });
}

On comparing the updated code with the previously generated test code. You will find two major differences here:

The generated code utilizes MockUtility, which is a mock version of the Utility class. However, the updated code directly employs the Utility class. This is because our objective is to test the methods that are encapsulated within this class.
The final test case aims to verify that the getPostTime2 function accurately converts a date from a different time zone into the local time zone. Initially, the AI assumed GMT+7 was the local time zone, but this led to a failure because my actual local time zone is IST. Consequently, the code was adjusted to employ a date that, when processed by getPostTime2, would align with the date stored in the expectedDate variable for Indian Standard Time (IST).

Now, since we have only updated one test, we will delete all other generated tests. This is because when we execute the save test command, Welltested saves all the tests it has generated and that are present in the test directory. We want to ensure that our AI doesn't learn from code with errors. Don't you agree?

After deleting all the test files except /getPostTime2.welltested_test.dart, run the following command in the terminal:

welltested save unit

After a successful save, you'll receive the following message in the terminal:

✅ Succeeded. 1 were saved.

Now, add the following command on the terminal to generate tests again:

welltested generate unit -m getDob && 
welltested generate unit -m getJoiningDate && 
welltested generate unit -m getUserName

After the test generation is completed. Revisit the generated tests:

This time, the tests adhere to our coding style. Also, the generated code correctly uses the local time zone for my case based on the saved code which is fantastic. Don't you agree? I certainly do.

Conclusion

By now, I hope you can appreciate the power of saving tests in Welltested and how it can assist you in crafting more efficient tests tailored to your specific needs and coding style.

I encourage you to generate tests using Welltested and to save all of them. This will not only speed up the test generation process but also enhance the quality of your tests over time. Happy Testing :)