Introduction
Chatbots are essential tools in various industries, providing automated interaction with users. There is no person in the world these days that have not tried at least once Chat GPT (or any other AI powered chatbot). Using OpenAI’s GPT models and the LangChain library, we can build a chatbot that handles sessions and processes user messages through a streaming response system, as in later post we will comunicate with our API's and make agents which will be specialized for certain things.
Here’s what we’ll cover:
- Setting up an Express server with middleware.
- Creating an `AgentManager` to handle chatbot agents.
- Creating an ChatAgent to handle chatbot agents.
- Streaming chatbot responses back to users in real-time.
Setting Up the Environment
First, we need a few key dependencies:
- Express for handling API requests.
- LangChain to manage GPT models and tools.
-
OpenAI for
GPT modelinteraction. We need to obtain token from Open AI in order to use spawn sessions and interact with chatbot
Install Dependencies
First thing we do is to initialize new project and install neccessarry modules which we will use.
npm init -Y
npm install express langchain openai uuid class-validator class-transformer mutex
Setting Up Express Routes
To begin, we'll define two main routes:
First route will create a new chat session, whereas second one will send messages to an existing session.
router.post('/session', APIKeyMiddleware, createSession);
router.post('/session/:id/message', APIKeyMiddleware, postMessage);
The APIKeyMiddleware ensures that only authenticated requests access these routes. Note that you can implement middleware which suits yours needs.
Creating Agent Manager
We’ll create an AgentManager class to handle chat agents. This class is responsible for creating new agents and managing active sessions, so imagine this class as the main entrypoint for our API's as it will hadle agents which will be responsible for chat. First user will need to create session and later on that session will be used for chatting.
export class AgentManager {
private __lock = new Mutex();
private __agents: Map<string, AgentInstance> = new Map();
async createAgent(authorization: string): Promise<string> {
const uuid = uuidv4();
const release = await this.__lock.acquire();
try {
this.__deleteExpiredAgentsLockless();
let agent: ChatAgent | null = agent = new GeneralChatAgent(authorization);
this.__agents.set(uuid, { agent, createdAt: Date.now() });
return uuid;
} finally {
release();
}
}
async getAgent(uuid: string): Promise<ChatAgent | null> {
const release = await this.__lock.acquire();
try {
this.__deleteExpiredAgentsLockless();
const agentInstance = this.__agents.get(uuid);
return agentInstance ? agentInstance.agent : null;
} finally {
release();
}
}
private __deleteExpiredAgentsLockless(): void {}
}
Creating General Agent
Now we need to create general chat agent, which will get parameters with lets say for example auth or any other you need and can be able to communicate with API, but for now we will extend existing ChatAgent and nothing more for this step.
export class GeneralChatAgent extends ChatAgent {
constructor() {
super();
}
}
The createAgent method initializes an agent, locks the process, and assigns it to a unique session ID. Agents expire after a specified session duration, which is handled by the __deleteExpiredAgentsLockless method but we will implement it in the next itteration you can avoid it for now.
Handling Sessions and Messages
Next, let's define our session creation and message handling routes:
export const createSession = async (req: Request, res: Response): Promise<void> => {
const authorization = req.headers['authorization'] as string;
try {
const sessionId = await agentManager.createAgent(authorization, AgentType.WEB);
res.json({ sessionId });
} catch (err) {
if (err instanceof Error) {
res.status(400).json({ error: err.message });
} else {
res.status(500).json({ error: 'An unknown error occurred' });
}
}
}
export const postMessage = async (req: Request, res: Response): Promise<void> => {
const { id } = req.params;
const { message } = req.body;
if (!id || !message) {
return res.status(400).json({ error: 'Bad request. Missing session ID or message' });
}
try {
const agent = await agentManager.getAgent(id);
if (!agent) {
return res.status(400).json({ error: `No agent found with id ${id}` });
}
const iterable = await agent.invoke(message);
await streamResponse(res, iterable);
} catch (err) {
res.status(500).json({ error: err instanceof Error ? err.message : 'An unknown error occurred' });
}
}
Here, createSession sets up a new session and postMessage sends a user’s message to the agent. If no session or message is provided, it returns a 400 Bad Request error.
Streaming Responses
Now, the key to making our chat bot feel responsive and interactive: streaming the response.
async invoke(input: string): Promise<AsyncIterable<Chunk>> {
const release = await this.__lock.acquire();
try {
const tool = this.determineTool(input);
if (tool) {
const toolOutput = await tool.call(input);
this.callbackQueue.enqueue({ type: ChunkType.TOKEN, value: toolOutput });
this.callbackQueue.enqueue({ type: ChunkType.FINISH, value: '' });
} else {
await this.chat.invoke([new HumanMessage(input)], {
callbacks: [
{
handleLLMNewToken: (token: string) => {
this.callbackQueue.enqueue({ type: ChunkType.TOKEN, value: token });
},
handleLLMEnd: () => {
this.callbackQueue.enqueue({ type: ChunkType.FINISH, value: '' });
},
handleLLMError: (error: Error) => {
this.callbackQueue.enqueue({ type: ChunkType.ERROR, value: error.message });
}
}
]
});
}
return this.createAsyncIterable(this.callbackQueue);
} finally {
release();
}
}
private createAsyncIterable(callbackQueue: AgentCallbackQueue): AsyncIterable<Chunk> {
return {
[Symbol.asyncIterator]: async function* () {
let finished = false;
while (!finished) {
const chunk = await callbackQueue.dequeue();
if (chunk) {
yield chunk;
if (chunk.type === ChunkType.FINISH || chunk.type === ChunkType.ERROR) {
finished = true;
}
} else {
await new Promise(resolve => setTimeout(resolve, 100));
}
}
}
};
}
In the invoke method, the agent processes the user’s input and streams back the response in chunks. Each chunk is either a token from the model or a message indicating the end of the stream.
The createAsyncIterable method allows us to generate these chunks one by one and stream them back to the client.
Streaming response
In the end, we want to stream response to client as we recieve it, dont want to wait for some time until completes and return the whole response, better solution is to stream response in chunks.
const delay = (ms: number): Promise<void> => new Promise(resolve => setTimeout(resolve, ms));
export async function streamResponse(res: Response, iterable: AsyncIterable<Chunk>) {
res.setHeader('Content-Type', 'application/x-ndjson');
res.setHeader('Transfer-Encoding', 'chunked');
try {
let buffer = '';
for await (const chunk of iterable) {
switch (chunk.type) {
case ChunkType.TOKEN:
buffer += chunk.value;
res.write(buffer);
if (res.flush) res.flush();
buffer = '';
break;
case ChunkType.ERROR:
console.error('Error chunk:', chunk.value);
if (!res.headersSent) {
res.status(500).json({ error: 'Streaming failed.' });
}
return;
case ChunkType.FINISH:
if (buffer.trim()) {
res.write(`${buffer.trim()}\n`);
}
return;
}
}
} catch (err) {
console.error('Error during streaming:', err);
if (!res.headersSent) {
res.status(500).json({ error: 'Streaming failed.' });
}
} finally {
res.end();
}
}
Conclusion
Congratulations! You now have a basic chatbot that handles chat sessions and streams responses back to the client. This architecture can be easily extended with additional tools, more sophisticated logic, or different GPT models, but for now we have skeleton for more complex chatbot.
By using OpenAI’s powerful language models and LangChain’s tool management, you can create more advanced and interactive chatbots for various domains.You can expand the chatbots capabilities and make it in a way that you want but on the other hand you dont need to use Langchain , you can use OpenAI and make even more simpler chat bot if you prefer that way.
Stay tuned for more , in next post we will talk about building tools for chat agent we made
Happy coding!
Feel free to check original post
Top comments (0)