DEV Community: Makram El Timani

Creating a Free AI voice-to-text transcription Program using Whisper

Makram El Timani — Wed, 23 Oct 2024 23:39:04 +0000

In the recent Meta Connect, Mark Zuckerberg mentioned

I think that voice is going to be a much more natural way of interacting with AI than text.

I completely agree with this and it is also much faster than typing your question out, especially when most AI solutions today have some sort of chat baked into them.

In this blog, we will create an API and simple website that transcribes your recording into text using OpenAI’s Whisper model.

Let’s get started!

API Blueprint

First, we need to create the API. It will contain only one method that we will call transcribe_audio

Let’s install the following modules

pip install fastapi uvicorn python-multipart

The code for the blueprint will look like this:

from fastapi import FastAPI, File, UploadFile
from fastapi.responses import JSONResponse

app = FastAPI()

@app.post("/transcribe-audio/")
async def transcribe_audio(file: UploadFile = File(...)):
    try:
        transcribed_text = "Call Whisper here"

        return JSONResponse(content={"lang": "en", "transcribed_text": transcribed_text})

    except Exception as e:
        return JSONResponse(content={"error": str(e)}, status_code=500)

A single endpoint called transcribe-audio will return the language and the text transcribed
If there is an exception, an error is returned.

The AI part

Setup Whisper

I have opted to download the CUDA-specific version of PyTorch along with torchaudio

pip install torch==1.11.0+cu113 torchaudio===0.11.0+cu113 -f https://download.pytorch.org/whl/cu113/torch_stable.html

Another thing to note is that if you have an Nvidia GPU, you will need to install CUDA drivers for this application to use your GPU. Otherwise, it will target the CPU for execution and you will have slower results.

Next, we install FASTAPI as well as transformers, accelerate, and numpy<2; The reason we want to use a lower version of numpy is to avoid warnings while running with torch.

pip install transformers accelerate "numpy<2"

Finally, we download Whisper from git. I found this to be the easier method of installing the model:

pip install git+https://github.com/openai/whisper.git

With everything installed, we now set up the whisper model

Import necessary modules
Set a temp directory to upload the audio files to
Check if the current device is a CUDA device
Load the “medium” whisper model. For a full list of models you can check here

from fastapi import FastAPI, File, UploadFile
from fastapi.responses import JSONResponse

import torch

import whisper

from pathlib import Path
import os

# Directory to save uploaded files
UPLOAD_DIR = Path("./uploaded_audios")
UPLOAD_DIR.mkdir(parents=True, exist_ok=True)

# check if the device is a cuda device, else use CPU
device = "cuda:0" if torch.cuda.is_available() else "cpu"
print(f"Using Device: {device}")

# for a full list of models see https://github.com/openai/whisper?tab=readme-ov-file#available-models-and-languages
model = whisper.load_model("medium", device=device)

app = FastAPI()

Now if you run the application using uvicorn main:app --reload, you will see the model loaded successfully (it may take a while depending on the model chosen). Also, if you have installed CUDA drivers, you will see Using Device: cuda:0 in the logs

Transcription

We will edit the transcribe_audio method to perform the transcription using the model

Save the uploaded file to the uploads directory created above
Call the transcribe method on the model
Extract the text and language from the result and return it in the response
Finally, delete the file uploaded

@app.post("/transcribe-audio/")
async def transcribe_audio(file: UploadFile = File(...)):
    try:
       # Path to save the file
        file_path = f"{UPLOAD_DIR}/{file.filename}"

        # Save the uploaded audio file to the specified path
        with open(file_path, "wb") as f:
            f.write(await file.read())

        # transcribe the audio using whisper model
        result = model.transcribe(file_path)

        # Extract the transcription text from the result
        transcribed_text = result['text']

        return JSONResponse(content={"lang": result["language"], "transcribed_text": transcribed_text})

    except Exception as e:
        print(e.__cause__)
        return JSONResponse(content={"error": str(e)}, status_code=500)

    finally:
        # Optionally, remove the saved file after transcription
        os.remove(file_path)

Result

Now if you run the API and perform a POST request to /transcribe-audio with an audio file in the form data, you will get the following:

{
    "lang": "en",
    "transcribed_text": " Hello hello 123"
}

The reason why I chose “medium” is that it does a good job of punctuation. In the following example, it adds a comma

{
    "lang": "en",
    "transcribed_text": " Hey hey, this is a test"
}

It can also understand different languages

{
    "lang": "ar",
    "transcribed_text": " مرحبا مرحبا اتكلم عربي"
}

Adding a Simple Frontend Integration

AI is wonderful. I just asked it to create a simple website for me to test this API and it regurgitated a lot of code to help me. Thanks ChatGPT.

You can find the code in on my GitHub, but this is how the final product will look like:

Omitted GIF of a long text:

Me attempting Spanish:

Conclusion and Learnings

Using pre-trained models is very easy to implement if you have the hardware to support it. For me, I had an Intel 14700K CPU and a GTX 1080ti GPU. Even though the GPU is a little bit old, I still got some impressive results. It was a lot faster than I expected; transcribing 30 seconds of audio in about 4-5 seconds.

In the Meta Connect event, Mark Zuckerberg demoed his new AI-powered smart glasses by having a conversation with a Spanish speaker, and the glasses displayed subtitles for each person in their respective language. Pretty cool! This is possible using Whisper, however it is only possible with the large model. Alternatively, we can use external libraries like DeepL to perform the translation for us after we transcribe.

Now, how do we fit this inside glasses? 🤔

A nice project to add this to would be a service that gets installed on your OS that will listen to a macro key press or something similar to fill in a text field with an audio recording on demand. This could be a nice side project 🧐 Whisper medium isn't very large but you can also substitute this with Whisper "turbo" or "tiny". I imagine these can run even on mobile devices. Hah, another side project 😄

If you liked this blog, make sure to like and comment. I will be diving more into running AI models locally to see what are the possibilities.

Say Hi 👋 to me on LinkedIn!

Create an AI Customer Service Chatbot API

Makram El Timani — Wed, 02 Oct 2024 19:28:46 +0000

Today we will take a look at a basic example of a Customer Service Chatbot API implementation using Open AI’s Assistant API. Check Open AI’s documentation for information on how the Assistant API works. But to summarize, it is basically a specialized AI assistant that you give a specific system prompt. You can also upload files that it can use to perform retrieval. It will be very useful to us when we add the FAQs that will help the assistant answer customer questions.

This topic was inspired by one of my conversations with a friend.

I will create this tutorial using .NET but it will be simple enough to follow along and try to implement it in any framework/language. Therefore, go ahead and create a new .NET Web API project if you would like to follow along.

Setup Open AI repository
Create API blueprint
Load/Create Assistant on Application Load
Open Connection Endpoint
Receive User Query Endpoint
Close Connection Endpoint
Improvements
- Model Context Length
- Custom actions
- Response Analysis
Conclusion

Setup Open AI Repository

I have already created a repository that was inspired by the Node.js Open AI repository referenced in their documentation

The repository is in .NET and you can check it out on my GitHub.

Please note, that this is a very basic implementation for an application I was developing quite some time ago. I took this code out of it and pushed it to GitHub for the sake of this tutorial.

You might not find it very clean, so you can use the .NET official SDK for Open AI instead.

For the purpose of this tutorial, I have opted to use my own implementation, since it resembles the docs and will be easier to follow along. You can add the repository in Program.cs as follows:

// Add OpenAiClient
builder.Services.AddOpenAiClient(opt =>
{
    opt.ApiKey = builder.Configuration.GetValue<string>("OpenAiClientSecrets:ApiKey");
    opt.ApiUrl = builder.Configuration.GetValue<string>("OpenAiClientSecrets:ApiUrl");
});

builder.Configuration.AddEnvironmentVariables()
    .AddUserSecrets(Assembly.GetExecutingAssembly(), true);

Make sure to have the secrets in place in appsettings.json or configure your own secrets:

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*",
  "OpenAiClientSecrets": {
    "ApiKey": "test-key",
    "ApiUrl": "https://api.openai.com/v1/"
  }
}

The repository is then added as singleton and you can use it in any service like this:

var assistants = await _openAiClient.Beta.Assistants.GetAssistants();

var assistant = await _openAiClient.Beta.Assistants.Create(new CreateAssistant());

var file = await _openAiClient.Files.CreateFile(new CreateFile());

Create API Blueprint

Before we start, we need to make sure to understand the repository mentioned above.

Here is what the repository can do:

Retrieve an assistant that you created on the Open AI dashboard
Create a new assistant with parameters you specify
Create a thread for the assistant to run on(acts as a chat session)
Add messages to the thread and create runs
- A run is basically telling the assistant to process my request
- You have to wait till the run is finished processing
Read the message from the assistant, parse it into something useful, and send it back to the customer

The above image illustrates the basic flow that the API will follow. For this, we will create a service to handle this logic. Below is the boilerplate of the service (Will show you the AssistantResponse class in a later step):

using AICustomerServiceAPI.Models;
using OpenAiRepository;

namespace AICustomerServiceAPI.Services;

public interface ICustomerAssistantService
{
    /// <summary>
    /// Called on project startup to initialize the assistant with the system message
    /// Will query Open AI to the assistant and compare if it is the same message, otherwise will create it
    /// Will also add the files to the assistant if they are not already there
    /// </summary>
    Task InitializeAssistant();

    /// <summary>
    /// This method will return the thread ID for the assistant
    /// </summary>
    Task<string> OpenChatConnection();

    /// <summary>
    /// Requires the thread ID from the OpenChatConnection method
    /// Should return the response from the assistant based on the customer's question
    /// </summary>
    Task<AssistantResponse> SendUserMessage(string threadId, string question);

    /// <summary>
    /// Deletes the thread ID from the assistant
    /// </summary>
    Task CloseChatConnection(string threadId);

}

//TODO: move this to file `CustomerAssistanService.cs`
public class CustomerAssistantService: ICustomerAssistantService
{
    private readonly OpenAiClient _openAiClient;

    public CustomerAssistantService(OpenAiClient openAiClient)
    {
        this._openAiClient = openAiClient;
    }

    public Task InitializeAssistant()
    {
        throw new NotImplementedException();
    }

    public Task<string> OpenChatConnection()
    {
        throw new NotImplementedException();
    }

    public Task<AssistantResponse> SendUserMessage(string threadId, string question)
    {
        throw new NotImplementedException();
    }

    public Task CloseChatConnection(string threadId)
    {
        throw new NotImplementedException();
    }
}

We will create 3 Endpoints:

Open Connection: This endpoint will create a thread object on the Open Ai’s assistant and send it back to the requesting client
Send User Message: accepts **thread ID**; This endpoint will forward the message to the assistant and create a Run object
Close Connection: accepts **thread ID**; This endpoint will delete the thread from the assistant

Our Program.cs needs 2 changes

Adding the customer assistant service to the services:

// Add services to the container.
builder.Services.AddScoped<ICustomerAssistantService, CustomerAssistantService>();

Initializing the assistant when the program starts and mapping the endpoints:

// Resolve the ICustomerAssistantService and call InitializeAssistant on the program start
using (var scope = app.Services.CreateScope())
{
    var services = scope.ServiceProvider;
    var openAiClient = services.GetRequiredService<ICustomerAssistantService>();
    await openAiClient.InitializeAssistant();
}

app.MapPost("/chat", async ([FromServices] ICustomerAssistantService customerAssistantService) =>
{
    return await customerAssistantService.OpenChatConnection();
});

app.MapDelete("/chat/{threadId}", async ([FromServices] ICustomerAssistantService customerAssistantService, string threadId) =>
{
    await customerAssistantService.CloseChatConnection(threadId);
    return Results.NoContent();
});

app.MapPost("/chat/{threadId}", async ([FromServices] ICustomerAssistantService customerAssistantService, string threadId, string question) =>
{
    return await customerAssistantService.AskQuestion(threadId, question);
});

app.Run();

Load/Create Assistant on Application Load

In the previous step, we called await openAiClient.InitializeAssistant(); in the Program.cs file when the app is about to run. This is meant to initialize the Assistant on Open AI by creating it if it doesn’t exist OR updating it if it does.

You can choose to ignore this by creating the assistant on the Open AI dashboard and providing ALL settings and files required. You can then save the Assistant ID somewhere in your settings and use it for the next steps.

I like doing it this way because it makes it cleaner for me (as a developer).

First, let’s go to ChatGPT and generate some FAQs:

{
  "faq": [
    {
      "category": "Account Management",
      "questions": [
        {
          "question": "Can I pause my subscription temporarily?",
          "answer": "Yes, you can pause your subscription for a period of up to 3 months by accessing the 'Manage Subscription' section in your account. During this period, you will not receive any shipments, and your billing will also be paused."
        },
        {
          "question": "How do I change the frequency of my deliveries?",
          "answer": "You can change your delivery frequency by going to 'Subscription Settings' and selecting your preferred interval. Options include monthly, bi-monthly, and quarterly shipments."
        },
        {
          "question": "What happens if I move to a different address during my subscription?",
          "answer": "If you move, you can update your shipping address in your account settings. Be sure to update it at least 7 days before your next scheduled delivery to avoid shipping delays."
        }
      ]
    },
    {
      "category": "Payments and Invoices",
      "questions": [
        {
          "question": "Can I change my payment method after subscribing?",
          "answer": "Yes, you can change your payment method at any time by navigating to the 'Billing' section of your account. Changes made will apply to your next billing cycle."
        },
        {
          "question": "How do I access past invoices?",
          "answer": "You can view and download your past invoices by logging into your account and selecting the 'Invoice History' section under 'Billing'."
        },
        {
          "question": "What should I do if my payment fails?",
          "answer": "If a payment fails, you'll receive a notification via email. Please update your payment details or retry the transaction manually through the 'Billing' section of your account."
        },
        {
          "question": "Can I get a refund for a shipped box?",
          "answer": "Refunds are not available once a box has been shipped. However, if you are unsatisfied with the product, please contact customer service for potential resolutions such as replacements or store credit."
        }
      ]
    },
    {
      "category": "Orders and Shipping",
      "questions": [
        {
          "question": "How can I track my order?",
          "answer": "Once your order is shipped, you will receive a tracking number via email. You can also find this information in the 'Order History' section of your account."
        },
        {
          "question": "What happens if I miss a delivery?",
          "answer": "If a delivery attempt is unsuccessful, the courier will leave a notice with further instructions. In most cases, they will attempt to redeliver or provide details for collecting the package from a nearby location."
        },
        {
          "question": "Can I customize the items in my box?",
          "answer": "While our boxes are curated based on your preferences, we do not offer individual customization of items. However, you can update your style preferences and sizes in your account to ensure future boxes better suit your needs."
        },
        {
          "question": "Do you ship internationally?",
          "answer": "At the moment, we ship exclusively within the United States. We are actively working to expand our service to other countries, and you can sign up for updates on our website."
        }
      ]
    },
    {
      "category": "Business and Service Questions",
      "questions": [
        {
          "question": "How are the clothes in the box sourced?",
          "answer": "All clothing items are sourced from vetted suppliers that focus on sustainability and recycling. We ensure that each item has gone through quality checks and is suitable for reuse."
        },
        {
          "question": "What is your sustainability impact?",
          "answer": "Our service promotes sustainability by recycling gently used clothes and reducing textile waste. For every box purchased, we donate to eco-friendly initiatives and provide transparency in our annual impact reports."
        },
        {
          "question": "How can I partner with your business?",
          "answer": "We are open to partnerships with sustainable fashion brands and organizations. If you're interested, please reach out to our business development team through the contact form on our website."
        },
        {
          "question": "What do you do with clothes that don't make it into boxes?",
          "answer": "Clothes that do not meet our quality standards are either donated to charitable organizations or repurposed through textile recycling programs. We strive to ensure no item goes to waste."
        }
      ]
    },
    {
      "category": "Advanced Subscription Queries",
      "questions": [
        {
          "question": "Can I upgrade or downgrade my subscription plan?",
          "answer": "Yes, you can change your subscription tier at any time from your account settings. Upgrades or downgrades will take effect in the next billing cycle."
        },
        {
          "question": "Is there a limit to how many boxes I can receive per month?",
          "answer": "There is no strict limit on the number of boxes you can receive per month. You can subscribe to multiple plans or adjust your frequency to receive more frequent deliveries."
        },
        {
          "question": "Do you offer corporate or bulk subscription plans?",
          "answer": "Yes, we offer bulk subscription options for businesses looking to provide sustainable clothing for their employees or clients. Contact our sales team to learn more about custom plans."
        },
        {
          "question": "How do I cancel my subscription?",
          "answer": "You can cancel your subscription at any time through your account settings. Once canceled, you will not be charged for future billing cycles, and any pending orders will be fulfilled unless otherwise requested."
        }
      ]
    },
    {
      "category": "Returns and Exchanges",
      "questions": [
        {
          "question": "What is your return policy?",
          "answer": "Due to the nature of recycled clothing, we do not offer returns. However, if an item is damaged or does not meet your expectations, please contact customer service for assistance with exchanges or refunds in the form of store credit."
        },
        {
          "question": "Can I exchange an item if it doesn’t fit?",
          "answer": "Yes, you can request an exchange for a different size within 14 days of receiving your box. Log in to your account and submit an exchange request through the 'Orders' section."
        }
      ]
    },
    {
      "category": "Gift and Referral Program",
      "questions": [
        {
          "question": "Can I gift a subscription?",
          "answer": "Yes, we offer gift subscriptions that can be purchased for a set number of months. The recipient will receive an email with instructions on how to activate their subscription."
        },
        {
          "question": "How does the referral program work?",
          "answer": "For every friend you refer who subscribes, both you and your friend will receive a discount on your next box. Referral bonuses are automatically applied to your account."
        }
      ]
    }
  ]
}

Now that we have some data, let’s save it in /Data/faq.json in our project.

Next, let’s write the prompt that we will be using:

You are a customer service support agent

Your job is to answer customer questions with the help of the FAQ and documentation. 

You will find the answer to the question in the files. Retrieve the file, read it, and try your best to answer the question. If you can't find the answer, you can ask the customer to contact the support team.

Examples of questions you might get asked:
- How do I change my password?
- How can I access my invoice?
- Can I have a refund?
- Can you explain how the subscription works?

Examples of answers you might give:
- You can change your password by going to the settings page in your account. Check underneath the security section.
- You can access your invoice by going to the billing section of your account. It should be the first thing you see.
- I'm sorry, but you will need to contact the support team to get a refund. They will be able to help you with that.
- The subscription works by charging you every month automatically. You can cancel at any time.

Your goal is to make sure the customer is happy and satisfied with the service.

Make sure to always be polite and helpful. If the customer asks a question in a different language, translate and provide a translated response. 

If you notice any aggression from the customer, make sure to offer apologies.

If you don't know the answer, don't try to guess. It's better to tell the customer that you are not sure and that you will get back to them with the right answer. Then, you can ask for help from the support team.

Use the following JSON scheme to structure your answer:


(```

json
  {
    "responseMessage": "Your response here",
    "foundResponse": true, // true or false
    "customerAnxiety": 10, // 1 to 10 (1 being the lowest and 10 the 
highest)
  }


  ```)

Make sure to ONLY REPLY WITH JSON.

We also save this in a text file in /Data/assistant_system_message.txt

Lastly, let’s code the Initialize Assistant method. Here is how the logic is:

Create or update the FAQ file on the account reading the data from the faq.json file
Create a vector store and add the above file to it
- If the vector store already exists, check if the file is in the store, if not add it
Create the assistant with the above system message
Set a static Assistant object to be used for the other service methods

private static Assistant? _assistant;

public async Task InitializeAssistant()
{
    string faqFileId = await UpsertProjectFile();

    string vectorStoreId = await UpsertVectorStore(faqFileId);

    _assistant = await UpsertAssistant(vectorStoreId);
}

//helper methods
private async Task<string> UpsertProjectFile()
{
    //retrieve all files in the project's store
    string faqFileName = "faq.json";
    string faqFilePath = $"Data/{faqFileName}";
    ListResponse<OpenAiFile> files = await _openAiClient.Files.ListFiles();
    OpenAiFile? faqFile = files.Data.FirstOrDefault(f => f.Filename == faqFileName);
    bool uploadFile = faqFile is null;
    if (faqFile is not null)
    {
        int fileBytes = File.ReadAllBytes(faqFilePath).Length;
        if (faqFile.Bytes != fileBytes)
        {
            //delete the file on the store
            await _openAiClient.Files.DeleteFile(faqFile.Id!);

            uploadFile = true;
        }
    }
    if (uploadFile)
    {
        //create the file
        // create the file
        FileStream? fileStream = File.OpenRead(faqFilePath);
        if (fileStream is not null)
        {
            faqFile = await _openAiClient.Files.CreateFile(new()
            {
                File = fileStream,
                FileName = faqFileName,
            });
            fileStream.Close();
        }
    }
    return faqFile!.Id!;
}

private async Task<string> UpsertVectorStore(string faqFileId)
{
    string vectorStoreId = string.Empty;
    string vectorStoreName = "CustomerAssistantVectorStore";
    ListResponse<VectorStore> vectorStores = await _openAiClient.VectorStores.ListVectorStores();
    bool hasFAQVectorStore = vectorStores.Data.Exists(m => m.Name == vectorStoreName);
    if (!hasFAQVectorStore)
    {
        //create vector store
        var vectorStore = await _openAiClient.VectorStores.CreateVectorStore(new CreateVectorStore
        {
            Name = vectorStoreName,
            FileIds = [faqFileId]
        });
        vectorStoreId = vectorStore.Id!;
    }
    else
    {
        //check if the file is in the vector store
        vectorStoreId = vectorStores.Data.First(m => m.Name == vectorStoreName).Id!;
        var vectorStoreFiles = await _openAiClient.VectorStores.LirstVectorStoreFiles(vectorStoreId);
        if (!vectorStoreFiles.Data.Exists(vectorStoreFiles => vectorStoreFiles.Id == faqFileId))
        {
            //add file to vector store
            await _openAiClient.VectorStores.CreateVectorStoreFile(vectorStoreId, new CreateVectorStoreFile
            {
                FileId = faqFileId,
            });
        }
    }
    return vectorStoreId;
}

private async Task<Assistant> UpsertAssistant(string vectorStoreId)
{
    //Read assistant system message from file
    string filePath = "Data/assistant_system_message.txt";
    string assistantSystemMessage = File.ReadAllText(filePath);

    string assistantTitle = "AI Customer Service API";
    string assistantModel = "gpt-4o-mini";

    ListResponse<Assistant> allAssistants = await _openAiClient.Beta.Assistants.GetAssistants();
    bool hasDefaultAssistant = allAssistants.Data.Exists(m => m.Name == assistantTitle);
    Assistant assistant;
    if (hasDefaultAssistant)
    {
        //check if the instructions match
        assistant = allAssistants.Data.First(m => m.Name == assistantTitle);
        ResponseFormat? responseFormat = null;
        if (!string.IsNullOrEmpty(assistant.ResponseFormat) && assistant.ResponseFormat != "auto")
        {
            responseFormat = JsonSerializer.Deserialize<ResponseFormat>(assistant.ResponseFormat);
        }
        if (AssistantPropertiesNeedUpdate(assistant, assistantSystemMessage, assistantModel, vectorStoreId))
        {
            //update assistant with correct instructions
            assistant = await _openAiClient.Beta.Assistants.ModifyAssistant(new UpdateAssistant
            {
                Instructions = assistantSystemMessage,
                Name = assistantTitle,
                Model = assistantModel,
                Tools = [new() { Type = AssistantToolType.FileSearch }],
                ToolResources = new()
                {
                    FileSearch = new()
                    {
                        VectorStoreIds = [vectorStoreId]
                    }
                },
                ResponseFormat = "auto",
            }, assistant.Id!);
        }
    }
    else
    {
        //create default assistant
        assistant = await _openAiClient.Beta.Assistants.CreateAssistant(new()
        {
            Instructions = assistantSystemMessage,
            Name = assistantTitle,
            Model = assistantModel,
            Tools = [new() { Type = AssistantToolType.FileSearch }],
            ToolResources = new()
            {
                FileSearch = new()
                {
                    VectorStoreIds = [vectorStoreId]
                }
            },
            ResponseFormat = "auto",
        });
    }

    return assistant;
}

private bool AssistantPropertiesNeedUpdate(Assistant assistant, string assistantSystemMessage, string assistantModel, string vectorStoreId)
{
    if (assistant.Instructions != assistantSystemMessage // doesn't have same instructions
            || assistant.Model != assistantModel // doesn't have same model
            || assistant.Tools.Length == 0 || assistant.Tools.First().Type != AssistantToolType.FileSearch // doesn't have file search tool
            || assistant.ToolResources is null || assistant.ToolResources.FileSearch is null || assistant.ToolResources.FileSearch.VectorStoreIds.Length == 0 || assistant.ToolResources.FileSearch.VectorStoreIds.First() != vectorStoreId // doesn't have the correct file search tool resource
            || assistant.ResponseFormat is null || assistant.ResponseFormat != "auto" // doesn't have the correct response format
            )
    {
        return false;
    }

    return true;
}

Now if you run the program and check your Playground on your OpenAI API Account, you will see the following assistant created:

You can now chat with it and test it.

Open Connection Endpoint

This endpoint is fairly simple. All it does is create a thread in the project that the assistant will run on in the next step. Think of it as creating a new chat window in ChatGPT.

public async Task<string> OpenChatConnection()
{
    var thread = await _openAiClient.Beta.Threads.Create(new CreateThread());
    return thread.Id!;
}

We will be returning the thread ID in the response so we can use it in subsequent calls.

Receive User Query Endpoint

This method is also simple.

We will create a message on the assistant with the role “User”

Next, create a Run object on OpenAI which basically starts the assistant’s process

This Run will take some time to finish, so we need to wait for it to become of status “Completed”

Finally, we get the messages from the assistant and try casting the first one to our custom class AssistantResponse

The code will look like this:

public async Task<AssistantResponse> SendUserMessage(string threadId, string messageText)
{
    // send message from user to assistant
    var message = await _openAiClient.Beta.Threads.Messages.CreateMessage(new CreateMessage
    {
        ThreadId = threadId!,
        Content = messageText,
        Role = "user",
    });

    //create the run on the assistant and thread
    var run = await _openAiClient.Beta.Threads.Runs.Create(new CreateRun
    {
        AssistantId = _assistant!.Id!,
        ThreadId = threadId,
    });

    //wait until the run is completed and the assistant has finished answering
    while (run!.Status != Repository.OpenAi.Models.RunStatus.completed)
    {
        await Task.Delay(500);
        run = await _openAiClient.Beta.Threads.Runs.Retrieve(threadId!, run.Id!);
    }

    //get the messages from the assistant and return the first one 
    var responseMessage = await _openAiClient.Beta.Threads.Messages.List(threadId!, run.Id);
    if (responseMessage.Data.Count == 0)
    {
        throw new InvalidDataException("No response from the assistant");
    }
    string content = responseMessage.Data.First().Content[0].Text!.Value!;
    content = content.Replace("```
{% endraw %}
json", "").Replace("
{% raw %}
```", ""); //needed to fix the json format
    AssistantResponse assistantResponse = JsonSerializer.Deserialize<AssistantResponse>(content)!;
    return assistantResponse;
}

And the custom class under /Models/AssistantResponse.cs

using System.Text.Json.Serialization;

namespace AICustomerServiceAPI.Models;

public class AssistantResponse
{
    [JsonPropertyName("responseMessage")]
    public string ResponseMessage { get; set; } = string.Empty;

    [JsonPropertyName("foundResponse")]
    public bool FoundResponse { get; set; }

    [JsonPropertyName("customerAnxiety")]
    public int CustomerAnxiety { get; set; }
}

Notice the line to replace JSON? This is needed since the response coming from the assistant will be wrapped by this text. It is a precaution to remove it. Below is a screenshot to view the response from the assistant

The response from the API will look like this:

Close Connection Endpoint

This is the simplest. We want to make sure to not have orphaned threads. Even though the Thread objects will persist for 60 days

If you plan on using the data from the questions and answers, you should save the messages in a persistent database before deleting them using the list method for the messages.

The code will look like this:

public async Task CloseChatConnection(string threadId)
{
   await _openAiClient.Beta.Threads.Delete(threadId);
}

That’s it. The repository makes it easy :)

Improvements

There are multiple ways we can improve this project. We can use better Dependency Injection, we need to handle the errors better, and we need to add missing features.

I will leave those for another time. However, it is nice to talk about some possible features this project can benefit from.

Model Context Length

The faq.json file we uploaded is included in the context the assistant uses for every retrieval. This means that the bigger the file, the more tokens you will use per message and therefore the more expensive this project can become.

To solve this issue, we need to implement fine-tuning. This is meant to create more specified assistants by specifying many examples. Open AI recommends using prompt engineering to create better prompts instead of fine-tuning, however, in this situation, it has advantages and can save a lot of money on chat responses.

Custom Actions

These are actions we can give the assistant to perform on the user’s behalf. For example, we can let the assistant expect an action like “Track My Order”, it will then reply with a message that prompts the user to enter their tracking code.

After the user enters their tracking code and sends it to the assistant, we can use some JSON parsing magic to figure out which API call to perform and retrieve the relevant information.

It might seem easy on paper, however, implementing it might be difficult.

Another example is the action to cancel a subscription. This might require more work on the security side since we need to make sure it is the user who is requesting the cancelation of his subscription.

Response Analysis

You might have noticed that I have added a customerAnxiety field to the response. This can be used to analyze the responses given by the assistant and how anxious/aggressive the tone of the user is. We can save the chats we get from the customer and have a secondary assistant analyze them and create a better FAQ for the users. You can also categorize the FAQs by anxiety level.

The possibilities are great with this one since AI now can read the data and analyze its results to detect information we might not see easily as humans.

Conclusion

Here is a link to the repo if you want to check it out and edit it yourself.

This has been a really fun project to work on. If you are still here, thank you so much for reading

Feel free to contact me, if you would like to see how we can turn this to use a self-hosted LLM without relying on OpenAI

Leave a like and comment with what you think and if you would like to see a part 2 of this blog where I go through the improvements and how to create them!

7 Projects I Would Do To Learn Web Dev From Scratch

Makram El Timani — Wed, 18 Sep 2024 23:21:06 +0000

A few weeks ago my brother asked me how he should learn web development. He is still a second year in university studying computer science and wanted my advice. I obviously didn't waste this opportunity to boast and talk about my experience building and managing websites for 10 years 🤓 but I told him that web dev experience comes best with practice. You can sit and watch tutorials for weeks but until you work on some projects, you won't retain any of this experience.

So I got inspired to write up these 7 projects to help him and potentially you 🫵 practice web dev from scratch.

KEEP IN MIND: I will not be holding your hand in this post. This isn't a tutorial. I will simply suggest ideas for you to turn into code. Use whatever means you are used to in order to learn. Use google searches, tutorial inspiration, or ask AI to explain to you the task you are trying to accomplish. DON'T COPY-PASTE but try to understand the solution in the context of your task.

Post screenshots of your progress 📷
I will love to see it!

Here is a preview:

Blog Page
Image Slideshow
Tic-tac-toe
Cat Quote Generator
Professional Redesign
Movie Theater
Modernize The Movie Theater

Note: Some projects are intentionally small in order to maximize your learning time. If you feel they are easy, add your own twists to them

Blog Page

The best way to learn html elements is with a blog IMO. You tackle almost all important elements you would see on blogs such us the one you're reading now.

First you would need content. You can copy content from somewhere or write it yourself; can be about anything, your favorite game, your local museum, feet. Gather some text to display on a website.

Use the following elements to organize and build your blog in the best way you see fit. Again, I am encouraging your creativity here but you can also use inspirations from your favorite blog site or this one maybe 😉
Elements to use:

article
headings 1 to 6 (not necessary to use all of them)
paragraphs
links
ordered lists
unordered lists
images
stylized text (bold, italic, strike-through)

You can always use W3Schools to reference the html elements. It's a great place to start.

Next add a little sparkle to your blog with styles. Can either use inline styles, <style> tag, or (my preferred way) cascading stylesheet file linked in the html file you are working on

Experiment with the following styles:

background color
font color
font size
font family
font weight
text alignment
image width
link hover color
any other styles you might want to do

Nice work lil' bro! You have successfully created an html page that looks subjectively cool.

Learnings: html, css

Image Slideshow

Now it's time to level up your html and css and introduce a little bit of javascript.

Gather up a list of 4+ images. They can be whatever you want; your friend group, your pets, your feet. You will be building the following:

Things to notice:

They are a list of images with only one "active" at a time
When an image becomes active, there is a fade-in animation
The next and previous buttons are located center right and center left above the image
There are small dots below the slideshow to indicate which image is currently active, when you click one, it shows the corresponding image at that index

In this exercise, you will learn the uses of the <div> and <span> tags.
You will also explore new css properties such as position and border-radius in addition to animations.

But now it is time to write your first javascript code

Create a <script> tag inside the html file, or use an external file with extension .js referenced in the html file (my preferred way).

Create the necessary functions for the next button, the previous button and little dots underneath the images.
HINT: do not forget to link the buttons to the functions

🤔 Question: What do you think should happen if you click next on the last image? Or when you click previous on the first image?

There you have it! Your first code in javascript. The possibilities are endless from here.

Learnings: html, css, js

Tic-tac-toe

😬 Grit your teeth my brother, you will be creating AI... sort of.

Remember when I said the possibilities in js are endless? You're about to hate me now.

See this GIF? I am controlling only the X and the script is figuring out the best place of the O

The html is simple. You can use either a <div> or a <table> which ever you feel is simpler.

This will test your js logical coding skills. You can use either js objects, arrays, 2d arrays or whatever you want to create the functionality of the above GIF. Whenever the player(you) clicks a grid item, place an X there and save it somewhere in js. After than the AI will find the ideal place to put the O. Up until someone loses.

I don't expect it to look pretty, when the game is over you can just display the winner underneath (player or computer)

Bonus points if you make the user pick if he wants X or O

🚀🚀You have just launched your journey into web dev. Can't turn back now that you have created a sentient being 🤖.

Learnings: js, programming logic, js objects

Cat Quote Generator

My brother, you are about to venture into a quest into dark territories, mystical lands, and unknown error responses. Yes, this is API calls. 😨

For this small project, we will build a page that calls 2 api requests.

The page will use the data from these 2 endpoints to display an image with a quote underneath it or on top of it. You choose. Use your creativity to design this page. Here are the main requirements:

Image of the cat
Quote text displayed somewhere
A button to refresh the image and the quote

The goal of this exercise is to get an introduction to API calls in js using fetch.
One page load, load a random image and a random zen quote and populate the html with them.
Then if the user clicks on the refresh button, do it again

Simple, easy, and gets you started where all internet starts: Cat Memes 😸

Now get to typin'!

Professional Redesign

We are done playing games. This is a big one. I introduced you to basic html and css, javascript, and even accessing the scary web. Now we put our big boy pants and we create a functioning website.

Take a look at this Free HTML Template

Notice above the preview page there is a Demo Preview button. If you click it, you will see the preview with a top where you can select the display device; mobile, ipad, etc...

The goal of this exercise is to learn using a UI Framework, namely Bootstrap to learn responsive design.

Create this exact same webpage using components from bootstrap.

See how there is a space between the content and the side edges? That is called a container in bootstrap

Did you notice the menu items and image are side by side? These are columns in bootstrap

A lot of people might disagree with bootstrap, and say you should learn Tailwind. The fact of the matter is, it doesn't matter. The point here is to learn how to use ANY UI Library and bootstrap in my opinion is the easiest.

Granted, you can accomplish this UI without the use of bootstrap. However, reading their documentation will help you understand mobile first

DO NOT COPY AND PASTE

Learnings: Responsive web development

Movie Theater

If the last challenge was not hard for you, this one might interest you 😈

I am a huge movie buff. I would love it if you would create a movie website for me.

We will tackle this similarly to the previous challenge; except now pick you own design to replicate. You can chose your local/favorite theater or you can use The Movie DB as inspiration

Same requirements as last challenge, make it responsive for smaller screens.

The Movie DB also has an API that you can use to get the information from.

Here are the requirements for this website

Create the UI

a. Header with links to other pages
b. Intro section with a search bar; no need to make the search bar functional now
c. Create a listing of movies by thumbnail, title release date and rating
d. Create an example move details page with the details of an example movie from the API
Create the functionality; make it dynamic

a. Using the Movie DB API, add functionality to the search; here you should utilize the <form> and <input> tags to understand how to submit website forms
- When the user searches, redirect to a search page to view search results
- Use the same display from (1.c) for the listing
b. Use the API to find the movies by category and display them in the home page (from step 1.c)
c. When you click on any movie in the list, redirect to the movie details page for that movie
You should not have a different html page for each movie; rather you should have a movie-details.html that reads the movie id from the query paramters and uses the API to populate the necessary information on the page

You will have issues.

You will encounter errors.

You will see a lot of logs you will not understand.

DO NOT GIVE UP. YOU CAN DO IT.

I will be watch with my popcorn.

Learnings: dynamic display of information on websites

Modernize The Movie Theater

Final stretch. If you have made it this far, I would like to congratulate you. You have what it takes to make it in the web dev world.

Your final challenge is to attempt to do the previous challenge using a Javascript Framework 👻

There are many to choose from. React JS, Angular, Vue, etc...

To be honest, it doesn't matter which one. They were all created to make developers life easier and harder at the same time.

I would say pick 2 of them; React and Angular. And do this exercise in both

The concept you should be learning in the challenge is how to think of your website in components. Think about the previous challenge and try to find which areas in your code were duplicated a lot. These can be changed into reusable components that will make your website more data driven.

This is a wonderful exercise that is a welcome addition to your portfolio for a potential starting position as a Frontend Developer.

Research a lot, by now you would know the basics and following along Youtube tutorials should be much simpler.

I will not oversee this project or add any hints. Practice your problem solving and creative skills as they are skills that are needed in a developer before even having Web Dev skills. When you are satisfied with your results, you can move on to further studies.

Learnings: Modern Web Dev Frameworks

By completing these projects, you're not just learning web development—you're building a foundation for a future full of opportunities. Whether you're just starting out or refining your skills, these challenges will help you grow as a developer. Remember, the key to success is persistence and creativity. Don't be afraid to experiment, make mistakes, and learn along the way.

Now, it's your turn! Dive into the projects, share your progress, and let's grow together. 🚀

Thank you for reading and don't hesitate to contact me!