Otello: how to make Chat-GPT interact with your API

During my internship I had to build a chatbot capable of connecting to a private API. Building an LLM from scratch was out of question, so I made a little reseach and found out there is not much literacy about this task.
The two most promising services were OpenAI API and SuperagentAI.
I opted for the first one, because of a better documentation and greater support.

So here is how i built Otello:

What is it

First of all, let me specify what we are talking about:
Otello is a chatbot that enables the user to make questions about the information present in our company Web app.
It gives the user an easy and natural way to get the data without the need of navigating trough the interface.

How it's made

The chatbot is composed of three sequential components:

1. AskCompletion calls the OpenAI Completion to identify the resources requested by the user. You need to list all the possible resources and configure a function call tool

const completion = await openai.chat.completions.create({
        messages: [{ role: "system", content: context }, { role: "user", content: user_request }],
        model: "ft:gpt-3.5-turbo-0613:personal::XXXXXXXX",
        tools: [
            {
                "type": "function",
                "function": {
                    "name": "api_fetch",
                    "description": "fetches the API and returns the requested resource",
                    "parameters": {
                        "type": "object",
                        "properties": {
                            "resource": {
                                "type": "string",
                                "description": "requested resource name"
                            }
                        },
                        "required": [
                            "resources"
                        ]
                    }
                }
            }
        ],
        tool_choice: "auto",
    });

1. FetchAPI: makes a call to our API based on the resources identified, and retrive the data in JSON format

2. AskAssitant: send a request to a custom OpenAI Assistant with the user question and the data needed to aswer it.

Why Completion?

Because Completion is simple, fast and, with a small drawback, it allows you to use a fine tuned model: a base GPT model trained on your data, perfect to have more control on it. (yes, you need to create a dataset from scratch with a decent nubmer of chats to train the model)

{"messages": [{"role": "user", "content": "User question 1"},{"role": "assistant", "function_call": {"name": "logica_fetch", "arguments": "{\"resource\": \"presenze_bloccate\"}"}}],"functions": [{"name": "logica_fetch","description": "fetches the API and returns the requested resource","parameters": {"type": "object","properties": {"resource": {"type": "string","description": "requested resource name"}},"required": ["resources"]}}]}
{"messages": [{"role": "user", "content": "User question 2"},{"role": "assistant", "function_call": {"name": "logica_fetch", "arguments": "{\"resource\": \"presenze_bloccate\"}"}}],"functions": [{"name": "logica_fetch","description": "fetches the API and returns the requested resource","parameters": {"type": "object","properties": {"resource": {"type": "string","description": "requested resource name"}},"required": ["resources"]}}]}
{"messages": [{"role": "user", "content": "User question 3"},{"role": "assistant", "function_call": {"name": "logica_fetch", "arguments": "{\"presenze_bloccate\": \"presenze\"}"}}],"functions": [{"name": "logica_fetch","description": "fetches the API and returns the requested resource","parameters": {"type": "object","properties": {"resource": {"type": "string","description": "requested resource namea"}},"required": ["resources"]}}]}
...

But...

Ok, now that the architecture is clear, let's talk about the main issue with this approach: how do you make an OpenAI GPT answer question based on your data? or even better, how do you upload the data it needs?
I found two possible approaches:

• Big model - the easy way, just use a GPT which allows a large amount of input tokens (long questions) and add them to the user request. It is easy but definitly not cheap nor scalable for large quantities of data. Some examples are: gpt-3.5-turbo-16k, gpt-4-32k

const message = await openai.beta.threads.messages.create(thread_id, 
    {
        role: "user",
        content: user_request + ' The data you need are: ' + data,
    }
);
const run = await openai.beta.threads.runs.create(thread_id, { assistant_id });

• File assistant - the hard way, you can upload a file in a supported format (like .JSON) and tell the assistant to use its content in the answer. It is harder to implement and slower but far more cheap and supports up to 150 Gb files.

let file = await openai.files.create({ file: output_files[0].file, purpose: "assistants" });
file = await openai.beta.assistants.files.create(assistantF_id, { file_id: file.id });
file_ids.push(file.id);
redirection = `the data you need ${output_files[0].label} are in the file: ${file.id}`;

const thread = await openai.beta.threads.create();

await openai.beta.threads.messages.create(
    thread.id,
        {
            role: "user",
            content: user_request + redirection,
            content: `${user_request}. ${redirection}`,
            file_ids
        }
    );
let run = await openai.beta.threads.runs.create(thread.id, { assistant_id: assistantF_id });

I built the backend with Node.js, but Python could have been an alternative too, and after a simple frontent in pure Javascript, it was time to deploy it.

Deployment

The deployment of the backend had been in an AWS Lambda function, even if it is not suited well for projects like this, where is needed a high processing time.

Conclusion

Even if the technologies are still quite new, building a personalized chatbot might be easier than you think, if you don't want anything too fancy.

If this caught your attention and you want to know more, you'll find the complete code with all the possible variations here: github-repo

Hope this helps