Create an AI Customer Service Chatbot API

Today we will take a look at a basic example of a Customer Service Chatbot API implementation using Open AI’s Assistant API. Check Open AI’s documentation for information on how the Assistant API works. But to summarize, it is basically a specialized AI assistant that you give a specific system prompt. You can also upload files that it can use to perform retrieval. It will be very useful to us when we add the FAQs that will help the assistant answer customer questions.

This topic was inspired by one of my conversations with a friend.

I will create this tutorial using .NET but it will be simple enough to follow along and try to implement it in any framework/language. Therefore, go ahead and create a new .NET Web API project if you would like to follow along.

Table of Contents

• Setup Open AI repository
• Create API blueprint
• Load/Create Assistant on Application Load
• Open Connection Endpoint
• Receive User Query Endpoint
• Close Connection Endpoint
• Improvements
  • Model Context Length
  • Custom actions
  • Response Analysis
• Conclusion

Setup Open AI Repository

I have already created a repository that was inspired by the Node.js Open AI repository referenced in their documentation

The repository is in .NET and you can check it out on my GitHub.

Please note, that this is a very basic implementation for an application I was developing quite some time ago. I took this code out of it and pushed it to GitHub for the sake of this tutorial.

You might not find it very clean, so you can use the .NET official SDK for Open AI instead.

For the purpose of this tutorial, I have opted to use my own implementation, since it resembles the docs and will be easier to follow along. You can add the repository in Program.cs as follows:

// Add OpenAiClient
builder.Services.AddOpenAiClient(opt =>
{
    opt.ApiKey = builder.Configuration.GetValue<string>("OpenAiClientSecrets:ApiKey");
    opt.ApiUrl = builder.Configuration.GetValue<string>("OpenAiClientSecrets:ApiUrl");
});

builder.Configuration.AddEnvironmentVariables()
    .AddUserSecrets(Assembly.GetExecutingAssembly(), true);

Make sure to have the secrets in place in appsettings.json or configure your own secrets:

{
  "Logging": {
    "LogLevel": {
      "Default": "Information",
      "Microsoft.AspNetCore": "Warning"
    }
  },
  "AllowedHosts": "*",
  "OpenAiClientSecrets": {
    "ApiKey": "test-key",
    "ApiUrl": "https://api.openai.com/v1/"
  }
}

The repository is then added as singleton and you can use it in any service like this:

var assistants = await _openAiClient.Beta.Assistants.GetAssistants();

var assistant = await _openAiClient.Beta.Assistants.Create(new CreateAssistant());

var file = await _openAiClient.Files.CreateFile(new CreateFile());

Create API Blueprint

Before we start, we need to make sure to understand the repository mentioned above.

Here is what the repository can do:

• Retrieve an assistant that you created on the Open AI dashboard
• Create a new assistant with parameters you specify
• Create a thread for the assistant to run on(acts as a chat session)
• Add messages to the thread and create runs
  • A run is basically telling the assistant to process my request
  • You have to wait till the run is finished processing
• Read the message from the assistant, parse it into something useful, and send it back to the customer

The above image illustrates the basic flow that the API will follow. For this, we will create a service to handle this logic. Below is the boilerplate of the service (Will show you the AssistantResponse class in a later step):

using AICustomerServiceAPI.Models;
using OpenAiRepository;

namespace AICustomerServiceAPI.Services;

public interface ICustomerAssistantService
{
    /// <summary>
    /// Called on project startup to initialize the assistant with the system message
    /// Will query Open AI to the assistant and compare if it is the same message, otherwise will create it
    /// Will also add the files to the assistant if they are not already there
    /// </summary>
    Task InitializeAssistant();

    /// <summary>
    /// This method will return the thread ID for the assistant
    /// </summary>
    Task<string> OpenChatConnection();

    /// <summary>
    /// Requires the thread ID from the OpenChatConnection method
    /// Should return the response from the assistant based on the customer's question
    /// </summary>
    Task<AssistantResponse> SendUserMessage(string threadId, string question);

    /// <summary>
    /// Deletes the thread ID from the assistant
    /// </summary>
    Task CloseChatConnection(string threadId);

}

//TODO: move this to file `CustomerAssistanService.cs`
public class CustomerAssistantService: ICustomerAssistantService
{
    private readonly OpenAiClient _openAiClient;

    public CustomerAssistantService(OpenAiClient openAiClient)
    {
        this._openAiClient = openAiClient;
    }

    public Task InitializeAssistant()
    {
        throw new NotImplementedException();
    }

    public Task<string> OpenChatConnection()
    {
        throw new NotImplementedException();
    }

    public Task<AssistantResponse> SendUserMessage(string threadId, string question)
    {
        throw new NotImplementedException();
    }

    public Task CloseChatConnection(string threadId)
    {
        throw new NotImplementedException();
    }
}

We will create 3 Endpoints:

• Open Connection: This endpoint will create a thread object on the Open Ai’s assistant and send it back to the requesting client
• Send User Message: accepts **thread ID**; This endpoint will forward the message to the assistant and create a Run object
• Close Connection: accepts **thread ID**; This endpoint will delete the thread from the assistant

Our Program.cs needs 2 changes

• Adding the customer assistant service to the services:

// Add services to the container.
builder.Services.AddScoped<ICustomerAssistantService, CustomerAssistantService>();

• Initializing the assistant when the program starts and mapping the endpoints:

// Resolve the ICustomerAssistantService and call InitializeAssistant on the program start
using (var scope = app.Services.CreateScope())
{
    var services = scope.ServiceProvider;
    var openAiClient = services.GetRequiredService<ICustomerAssistantService>();
    await openAiClient.InitializeAssistant();
}

app.MapPost("/chat", async ([FromServices] ICustomerAssistantService customerAssistantService) =>
{
    return await customerAssistantService.OpenChatConnection();
});

app.MapDelete("/chat/{threadId}", async ([FromServices] ICustomerAssistantService customerAssistantService, string threadId) =>
{
    await customerAssistantService.CloseChatConnection(threadId);
    return Results.NoContent();
});

app.MapPost("/chat/{threadId}", async ([FromServices] ICustomerAssistantService customerAssistantService, string threadId, string question) =>
{
    return await customerAssistantService.AskQuestion(threadId, question);
});

app.Run();

Load/Create Assistant on Application Load

In the previous step, we called await openAiClient.InitializeAssistant(); in the Program.cs file when the app is about to run. This is meant to initialize the Assistant on Open AI by creating it if it doesn’t exist OR updating it if it does.

You can choose to ignore this by creating the assistant on the Open AI dashboard and providing ALL settings and files required. You can then save the Assistant ID somewhere in your settings and use it for the next steps.

I like doing it this way because it makes it cleaner for me (as a developer).

First, let’s go to ChatGPT and generate some FAQs:

{
  "faq": [
    {
      "category": "Account Management",
      "questions": [
        {
          "question": "Can I pause my subscription temporarily?",
          "answer": "Yes, you can pause your subscription for a period of up to 3 months by accessing the 'Manage Subscription' section in your account. During this period, you will not receive any shipments, and your billing will also be paused."
        },
        {
          "question": "How do I change the frequency of my deliveries?",
          "answer": "You can change your delivery frequency by going to 'Subscription Settings' and selecting your preferred interval. Options include monthly, bi-monthly, and quarterly shipments."
        },
        {
          "question": "What happens if I move to a different address during my subscription?",
          "answer": "If you move, you can update your shipping address in your account settings. Be sure to update it at least 7 days before your next scheduled delivery to avoid shipping delays."
        }
      ]
    },
    {
      "category": "Payments and Invoices",
      "questions": [
        {
          "question": "Can I change my payment method after subscribing?",
          "answer": "Yes, you can change your payment method at any time by navigating to the 'Billing' section of your account. Changes made will apply to your next billing cycle."
        },
        {
          "question": "How do I access past invoices?",
          "answer": "You can view and download your past invoices by logging into your account and selecting the 'Invoice History' section under 'Billing'."
        },
        {
          "question": "What should I do if my payment fails?",
          "answer": "If a payment fails, you'll receive a notification via email. Please update your payment details or retry the transaction manually through the 'Billing' section of your account."
        },
        {
          "question": "Can I get a refund for a shipped box?",
          "answer": "Refunds are not available once a box has been shipped. However, if you are unsatisfied with the product, please contact customer service for potential resolutions such as replacements or store credit."
        }
      ]
    },
    {
      "category": "Orders and Shipping",
      "questions": [
        {
          "question": "How can I track my order?",
          "answer": "Once your order is shipped, you will receive a tracking number via email. You can also find this information in the 'Order History' section of your account."
        },
        {
          "question": "What happens if I miss a delivery?",
          "answer": "If a delivery attempt is unsuccessful, the courier will leave a notice with further instructions. In most cases, they will attempt to redeliver or provide details for collecting the package from a nearby location."
        },
        {
          "question": "Can I customize the items in my box?",
          "answer": "While our boxes are curated based on your preferences, we do not offer individual customization of items. However, you can update your style preferences and sizes in your account to ensure future boxes better suit your needs."
        },
        {
          "question": "Do you ship internationally?",
          "answer": "At the moment, we ship exclusively within the United States. We are actively working to expand our service to other countries, and you can sign up for updates on our website."
        }
      ]
    },
    {
      "category": "Business and Service Questions",
      "questions": [
        {
          "question": "How are the clothes in the box sourced?",
          "answer": "All clothing items are sourced from vetted suppliers that focus on sustainability and recycling. We ensure that each item has gone through quality checks and is suitable for reuse."
        },
        {
          "question": "What is your sustainability impact?",
          "answer": "Our service promotes sustainability by recycling gently used clothes and reducing textile waste. For every box purchased, we donate to eco-friendly initiatives and provide transparency in our annual impact reports."
        },
        {
          "question": "How can I partner with your business?",
          "answer": "We are open to partnerships with sustainable fashion brands and organizations. If you're interested, please reach out to our business development team through the contact form on our website."
        },
        {
          "question": "What do you do with clothes that don't make it into boxes?",
          "answer": "Clothes that do not meet our quality standards are either donated to charitable organizations or repurposed through textile recycling programs. We strive to ensure no item goes to waste."
        }
      ]
    },
    {
      "category": "Advanced Subscription Queries",
      "questions": [
        {
          "question": "Can I upgrade or downgrade my subscription plan?",
          "answer": "Yes, you can change your subscription tier at any time from your account settings. Upgrades or downgrades will take effect in the next billing cycle."
        },
        {
          "question": "Is there a limit to how many boxes I can receive per month?",
          "answer": "There is no strict limit on the number of boxes you can receive per month. You can subscribe to multiple plans or adjust your frequency to receive more frequent deliveries."
        },
        {
          "question": "Do you offer corporate or bulk subscription plans?",
          "answer": "Yes, we offer bulk subscription options for businesses looking to provide sustainable clothing for their employees or clients. Contact our sales team to learn more about custom plans."
        },
        {
          "question": "How do I cancel my subscription?",
          "answer": "You can cancel your subscription at any time through your account settings. Once canceled, you will not be charged for future billing cycles, and any pending orders will be fulfilled unless otherwise requested."
        }
      ]
    },
    {
      "category": "Returns and Exchanges",
      "questions": [
        {
          "question": "What is your return policy?",
          "answer": "Due to the nature of recycled clothing, we do not offer returns. However, if an item is damaged or does not meet your expectations, please contact customer service for assistance with exchanges or refunds in the form of store credit."
        },
        {
          "question": "Can I exchange an item if it doesn’t fit?",
          "answer": "Yes, you can request an exchange for a different size within 14 days of receiving your box. Log in to your account and submit an exchange request through the 'Orders' section."
        }
      ]
    },
    {
      "category": "Gift and Referral Program",
      "questions": [
        {
          "question": "Can I gift a subscription?",
          "answer": "Yes, we offer gift subscriptions that can be purchased for a set number of months. The recipient will receive an email with instructions on how to activate their subscription."
        },
        {
          "question": "How does the referral program work?",
          "answer": "For every friend you refer who subscribes, both you and your friend will receive a discount on your next box. Referral bonuses are automatically applied to your account."
        }
      ]
    }
  ]
}

Now that we have some data, let’s save it in /Data/faq.json in our project.

Next, let’s write the prompt that we will be using:

You are a customer service support agent

Your job is to answer customer questions with the help of the FAQ and documentation. 

You will find the answer to the question in the files. Retrieve the file, read it, and try your best to answer the question. If you can't find the answer, you can ask the customer to contact the support team.

Examples of questions you might get asked:
- How do I change my password?
- How can I access my invoice?
- Can I have a refund?
- Can you explain how the subscription works?

Examples of answers you might give:
- You can change your password by going to the settings page in your account. Check underneath the security section.
- You can access your invoice by going to the billing section of your account. It should be the first thing you see.
- I'm sorry, but you will need to contact the support team to get a refund. They will be able to help you with that.
- The subscription works by charging you every month automatically. You can cancel at any time.

Your goal is to make sure the customer is happy and satisfied with the service.

Make sure to always be polite and helpful. If the customer asks a question in a different language, translate and provide a translated response. 

If you notice any aggression from the customer, make sure to offer apologies.

If you don't know the answer, don't try to guess. It's better to tell the customer that you are not sure and that you will get back to them with the right answer. Then, you can ask for help from the support team.

Use the following JSON scheme to structure your answer:


(```

json
  {
    "responseMessage": "Your response here",
    "foundResponse": true, // true or false
    "customerAnxiety": 10, // 1 to 10 (1 being the lowest and 10 the 
highest)
  }


  ```)

Make sure to ONLY REPLY WITH JSON.

We also save this in a text file in /Data/assistant_system_message.txt

Lastly, let’s code the Initialize Assistant method. Here is how the logic is:

1. Create or update the FAQ file on the account reading the data from the faq.json file
2. Create a vector store and add the above file to it
   • If the vector store already exists, check if the file is in the store, if not add it
3. Create the assistant with the above system message
4. Set a static Assistant object to be used for the other service methods

private static Assistant? _assistant;

public async Task InitializeAssistant()
{
    string faqFileId = await UpsertProjectFile();

    string vectorStoreId = await UpsertVectorStore(faqFileId);

    _assistant = await UpsertAssistant(vectorStoreId);
}

//helper methods
private async Task<string> UpsertProjectFile()
{
    //retrieve all files in the project's store
    string faqFileName = "faq.json";
    string faqFilePath = $"Data/{faqFileName}";
    ListResponse<OpenAiFile> files = await _openAiClient.Files.ListFiles();
    OpenAiFile? faqFile = files.Data.FirstOrDefault(f => f.Filename == faqFileName);
    bool uploadFile = faqFile is null;
    if (faqFile is not null)
    {
        int fileBytes = File.ReadAllBytes(faqFilePath).Length;
        if (faqFile.Bytes != fileBytes)
        {
            //delete the file on the store
            await _openAiClient.Files.DeleteFile(faqFile.Id!);

            uploadFile = true;
        }
    }
    if (uploadFile)
    {
        //create the file
        // create the file
        FileStream? fileStream = File.OpenRead(faqFilePath);
        if (fileStream is not null)
        {
            faqFile = await _openAiClient.Files.CreateFile(new()
            {
                File = fileStream,
                FileName = faqFileName,
            });
            fileStream.Close();
        }
    }
    return faqFile!.Id!;
}

private async Task<string> UpsertVectorStore(string faqFileId)
{
    string vectorStoreId = string.Empty;
    string vectorStoreName = "CustomerAssistantVectorStore";
    ListResponse<VectorStore> vectorStores = await _openAiClient.VectorStores.ListVectorStores();
    bool hasFAQVectorStore = vectorStores.Data.Exists(m => m.Name == vectorStoreName);
    if (!hasFAQVectorStore)
    {
        //create vector store
        var vectorStore = await _openAiClient.VectorStores.CreateVectorStore(new CreateVectorStore
        {
            Name = vectorStoreName,
            FileIds = [faqFileId]
        });
        vectorStoreId = vectorStore.Id!;
    }
    else
    {
        //check if the file is in the vector store
        vectorStoreId = vectorStores.Data.First(m => m.Name == vectorStoreName).Id!;
        var vectorStoreFiles = await _openAiClient.VectorStores.LirstVectorStoreFiles(vectorStoreId);
        if (!vectorStoreFiles.Data.Exists(vectorStoreFiles => vectorStoreFiles.Id == faqFileId))
        {
            //add file to vector store
            await _openAiClient.VectorStores.CreateVectorStoreFile(vectorStoreId, new CreateVectorStoreFile
            {
                FileId = faqFileId,
            });
        }
    }
    return vectorStoreId;
}

private async Task<Assistant> UpsertAssistant(string vectorStoreId)
{
    //Read assistant system message from file
    string filePath = "Data/assistant_system_message.txt";
    string assistantSystemMessage = File.ReadAllText(filePath);

    string assistantTitle = "AI Customer Service API";
    string assistantModel = "gpt-4o-mini";

    ListResponse<Assistant> allAssistants = await _openAiClient.Beta.Assistants.GetAssistants();
    bool hasDefaultAssistant = allAssistants.Data.Exists(m => m.Name == assistantTitle);
    Assistant assistant;
    if (hasDefaultAssistant)
    {
        //check if the instructions match
        assistant = allAssistants.Data.First(m => m.Name == assistantTitle);
        ResponseFormat? responseFormat = null;
        if (!string.IsNullOrEmpty(assistant.ResponseFormat) && assistant.ResponseFormat != "auto")
        {
            responseFormat = JsonSerializer.Deserialize<ResponseFormat>(assistant.ResponseFormat);
        }
        if (AssistantPropertiesNeedUpdate(assistant, assistantSystemMessage, assistantModel, vectorStoreId))
        {
            //update assistant with correct instructions
            assistant = await _openAiClient.Beta.Assistants.ModifyAssistant(new UpdateAssistant
            {
                Instructions = assistantSystemMessage,
                Name = assistantTitle,
                Model = assistantModel,
                Tools = [new() { Type = AssistantToolType.FileSearch }],
                ToolResources = new()
                {
                    FileSearch = new()
                    {
                        VectorStoreIds = [vectorStoreId]
                    }
                },
                ResponseFormat = "auto",
            }, assistant.Id!);
        }
    }
    else
    {
        //create default assistant
        assistant = await _openAiClient.Beta.Assistants.CreateAssistant(new()
        {
            Instructions = assistantSystemMessage,
            Name = assistantTitle,
            Model = assistantModel,
            Tools = [new() { Type = AssistantToolType.FileSearch }],
            ToolResources = new()
            {
                FileSearch = new()
                {
                    VectorStoreIds = [vectorStoreId]
                }
            },
            ResponseFormat = "auto",
        });
    }

    return assistant;
}

private bool AssistantPropertiesNeedUpdate(Assistant assistant, string assistantSystemMessage, string assistantModel, string vectorStoreId)
{
    if (assistant.Instructions != assistantSystemMessage // doesn't have same instructions
            || assistant.Model != assistantModel // doesn't have same model
            || assistant.Tools.Length == 0 || assistant.Tools.First().Type != AssistantToolType.FileSearch // doesn't have file search tool
            || assistant.ToolResources is null || assistant.ToolResources.FileSearch is null || assistant.ToolResources.FileSearch.VectorStoreIds.Length == 0 || assistant.ToolResources.FileSearch.VectorStoreIds.First() != vectorStoreId // doesn't have the correct file search tool resource
            || assistant.ResponseFormat is null || assistant.ResponseFormat != "auto" // doesn't have the correct response format
            )
    {
        return false;
    }

    return true;
}

Now if you run the program and check your Playground on your OpenAI API Account, you will see the following assistant created:

You can now chat with it and test it.

Open Connection Endpoint

This endpoint is fairly simple. All it does is create a thread in the project that the assistant will run on in the next step. Think of it as creating a new chat window in ChatGPT.

public async Task<string> OpenChatConnection()
{
    var thread = await _openAiClient.Beta.Threads.Create(new CreateThread());
    return thread.Id!;
}

We will be returning the thread ID in the response so we can use it in subsequent calls.

Receive User Query Endpoint

This method is also simple.

We will create a message on the assistant with the role “User”

Next, create a Run object on OpenAI which basically starts the assistant’s process

This Run will take some time to finish, so we need to wait for it to become of status “Completed”

Finally, we get the messages from the assistant and try casting the first one to our custom class AssistantResponse

The code will look like this:

public async Task<AssistantResponse> SendUserMessage(string threadId, string messageText)
{
    // send message from user to assistant
    var message = await _openAiClient.Beta.Threads.Messages.CreateMessage(new CreateMessage
    {
        ThreadId = threadId!,
        Content = messageText,
        Role = "user",
    });

    //create the run on the assistant and thread
    var run = await _openAiClient.Beta.Threads.Runs.Create(new CreateRun
    {
        AssistantId = _assistant!.Id!,
        ThreadId = threadId,
    });

    //wait until the run is completed and the assistant has finished answering
    while (run!.Status != Repository.OpenAi.Models.RunStatus.completed)
    {
        await Task.Delay(500);
        run = await _openAiClient.Beta.Threads.Runs.Retrieve(threadId!, run.Id!);
    }

    //get the messages from the assistant and return the first one 
    var responseMessage = await _openAiClient.Beta.Threads.Messages.List(threadId!, run.Id);
    if (responseMessage.Data.Count == 0)
    {
        throw new InvalidDataException("No response from the assistant");
    }
    string content = responseMessage.Data.First().Content[0].Text!.Value!;
    content = content.Replace("```
{% endraw %}
json", "").Replace("
{% raw %}
```", ""); //needed to fix the json format
    AssistantResponse assistantResponse = JsonSerializer.Deserialize<AssistantResponse>(content)!;
    return assistantResponse;
}

And the custom class under /Models/AssistantResponse.cs

using System.Text.Json.Serialization;

namespace AICustomerServiceAPI.Models;

public class AssistantResponse
{
    [JsonPropertyName("responseMessage")]
    public string ResponseMessage { get; set; } = string.Empty;

    [JsonPropertyName("foundResponse")]
    public bool FoundResponse { get; set; }

    [JsonPropertyName("customerAnxiety")]
    public int CustomerAnxiety { get; set; }
}

Notice the line to replace JSON? This is needed since the response coming from the assistant will be wrapped by this text. It is a precaution to remove it. Below is a screenshot to view the response from the assistant

The response from the API will look like this:

Close Connection Endpoint

This is the simplest. We want to make sure to not have orphaned threads. Even though the Thread objects will persist for 60 days

If you plan on using the data from the questions and answers, you should save the messages in a persistent database before deleting them using the list method for the messages.

The code will look like this:

public async Task CloseChatConnection(string threadId)
{
   await _openAiClient.Beta.Threads.Delete(threadId);
}

That’s it. The repository makes it easy :)

Improvements

There are multiple ways we can improve this project. We can use better Dependency Injection, we need to handle the errors better, and we need to add missing features.

I will leave those for another time. However, it is nice to talk about some possible features this project can benefit from.

Model Context Length

The faq.json file we uploaded is included in the context the assistant uses for every retrieval. This means that the bigger the file, the more tokens you will use per message and therefore the more expensive this project can become.

To solve this issue, we need to implement fine-tuning. This is meant to create more specified assistants by specifying many examples. Open AI recommends using prompt engineering to create better prompts instead of fine-tuning, however, in this situation, it has advantages and can save a lot of money on chat responses.

Custom Actions

These are actions we can give the assistant to perform on the user’s behalf. For example, we can let the assistant expect an action like “Track My Order”, it will then reply with a message that prompts the user to enter their tracking code.

After the user enters their tracking code and sends it to the assistant, we can use some JSON parsing magic to figure out which API call to perform and retrieve the relevant information.

It might seem easy on paper, however, implementing it might be difficult.

Another example is the action to cancel a subscription. This might require more work on the security side since we need to make sure it is the user who is requesting the cancelation of his subscription.

Response Analysis

You might have noticed that I have added a customerAnxiety field to the response. This can be used to analyze the responses given by the assistant and how anxious/aggressive the tone of the user is. We can save the chats we get from the customer and have a secondary assistant analyze them and create a better FAQ for the users. You can also categorize the FAQs by anxiety level.

The possibilities are great with this one since AI now can read the data and analyze its results to detect information we might not see easily as humans.

Conclusion

Here is a link to the repo if you want to check it out and edit it yourself.

This has been a really fun project to work on. If you are still here, thank you so much for reading

Feel free to contact me, if you would like to see how we can turn this to use a self-hosted LLM without relying on OpenAI

Leave a like and comment with what you think and if you would like to see a part 2 of this blog where I go through the improvements and how to create them!