DEV Community: Deola Adediran

Building a Continuous Voice Interface with the OpenAI Realtime API

Deola Adediran — Sun, 05 Apr 2026 16:04:44 +0000

A technical walkthrough of how the ABD Assistant voice command system works end-to-end, from raw microphone bytes to tool execution.

The Core Architecture

The system has three moving parts: a browser Web Audio capture layer, an Express WebSocket relay, and OpenAI's Realtime API as the voice brain.

The browser streams PCM audio directly to OpenAI via a WebSocket that stays open for the entire session. OpenAI performs server-side voice activity detection (VAD), transcribes speech incrementally, runs its LLM over the conversation history, and streams back audio tokens as they're generated. This means no client-side silence detection, no turn-management logic, and no separate transcription step — one pipeline, fully server-driven.

Audio Capture: The Hard Part

Capturing audio correctly is where most implementations fall apart. The key constraint: OpenAI's Realtime API expects mono PCM at 24kHz, 16-bit signed integers. Browser MediaRecorder produces audio/webm or audio/opus — a completely different format. The solution is a ScriptProcessorNode (or AudioWorklet):

const processor = audioContext.createScriptProcessor(4096, 1, 1);
processor.onaudioprocess = (ev) => {
  const inputData = ev.inputBuffer.getChannelData(0);
  const pcmData = new Int16Array(inputData.length);
  for (let i = 0; i < inputData.length; i++) {
    const s = Math.max(-1, Math.min(1, inputData[i]));
    pcmData[i] = s < 0 ? s * 32768 : s * 32767;
  }
  const base64 = btoa(String.fromCharCode(...new Uint8Array(pcmData.buffer)));
  ws.send(JSON.stringify({ type: "input_audio_buffer.append", audio: base64 }));
};

Each chunk gets base64-encoded and sent as a WebSocket message. No buffering, no batching — continuous streaming.

WebSocket Relay

The OpenAI Realtime API WebSocket URL requires your API key in a header, which browser WebSocket clients can't set. A thin Express server handles the relay:

wss.on("connection", (clientWs, req) => {
  const apiKey = new URL(req.url, `http://${req.headers.host}`).searchParams.get("api_key");
  if (apiKey !== process.env.VOICE_API_KEY) { clientWs.close(4401); return; }

  const openAiWs = new WebSocket("wss://api.openai.com/v1/realtime?model=gpt-realtime-2025-08-28", {
    headers: { Authorization: `Bearer ${process.env.OPENAI_API_KEY}` }
  });

  openAiWs.on("message", (data) => clientWs.send(data.toString()));
  clientWs.on("message", (data) => openAiWs.send(data.toString()));
  clientWs.on("close", () => openAiWs.close());
});

All message passing is raw — no transformation. The relay is stateless.

GA Realtime API Session Config

The Generally Available Realtime API uses a different session schema than the beta. The session.update payload must include type: "realtime" and fields are organized under audio.input and audio.output:

const sessionUpdate = {
  type: "session.update",
  session: {
    type: "realtime",
    model: "gpt-realtime-2025-08-28",
    audio: {
      input: {
        format: { type: "audio/pcm", rate: 24000 },
        turn_detection: { type: "server_vad", prefix_padding_ms: 1000, silence_duration_ms: 400 }
      },
      output: { format: { type: "audio/pcm", rate: 24000 }, voice: "alloy" }
    },
    output_modalities: ["audio"]
  }
};

Key fields that trip up implementations: session.type must be "realtime" (not "conversation"), modalities is not a top-level field (use output_modalities: ["audio"]), and voice lives under audio.output.

Event Handling

The GA API emits different event names than the beta. The critical ones to handle:

conversation.item.input_audio_transcription.delta → user transcript (word-by-word)
response.output_audio.delta → AI audio chunk (bytes in event.delta, not event.audio)
response.output_audio_transcript.delta → AI transcript
input_audio_buffer.speech_started / speech_stopped → VAD state changes
response.done → response complete, session continues

Tool Calling

The Realtime API supports function calling natively within the session. Define tools in the session.update:

sessionUpdate.session.tools = toolDefinitions.map(t => ({ type: "function", ...t }));

When the model emits response.done with output[0].type === "function_call", execute the tool client-side, then inject the result:

const result = await executeTool(name, args);
ws.send(JSON.stringify({
  type: "conversation.item.create",
  item: { type: "function_call_output", call_id, output: result }
}));
ws.send(JSON.stringify({ type: "response.create" }));

This triggers the model to respond with speech confirming the action.

Audio Playback: Sequential Queue

Streaming audio chunks arrive out-of-order and overlap if played immediately. A simple FIFO queue with isPlaying flag solves this — each chunk is decoded to an AudioBuffer and played through a BufferSourceNode; the onended callback triggers the next chunk.

Challenges & Trade-offs

ScriptProcessorNode deprecation: Use AudioWorklet in production; ScriptProcessorNode still works but Chrome shows a deprecation warning.
No multi-voice: output_modalities: ["audio"] forces a single voice model; there's no way to get text + audio simultaneously from the same model output.
No streaming transcript: Input transcription arrives as input_audio_transcription.delta, not streamed word-by-word like the audio output.
Tool execution timing: When a tool executes, recording pauses; the model waits for the response.create call before continuing.

For Your Own Project

The only ABD-specific parts are the tool definitions and executeTool mapping — swap those for your own API surface. Everything else — audio capture, WebSocket relay, session management, audio playback — is reusable infrastructure.

Machine learning for beginners with Autogon AI

Deola Adediran — Fri, 18 Aug 2023 17:12:24 +0000

Finally decided to learn machine learning and you are so confused with so many materials available to you While also, you are a big fan of no-code platforms, then have got you covered with Autogon AI, it’s a platform that helps you learn and build models without writing any code.

In this article, I will be explaining to you basic information about machine learning and building your first basic model on Autogon AI.

What is Machine learning?
According to BMI, machine learning is a branch of artificial intelligence (AI) and computer science which focuses on the use of data and algorithms to imitate the way humans learn, gradually improving its accuracy.

In simple terms, we could say,

Machine learning is a process of teaching computer machines how to make accurate predictions when given data to work or train on.

The best way to understand machine learning more clearly is with an example. So imagine you were giving data containing transactions of a financial institute and you were asked to either categorize the data either into fraud or not fraud and also asked to predict for a new set of data which category they fall into based on the previous data given, if the data is small in size (maybe 10 items or less) it could be easy for humans to perform the task but for an extremely large data, this is the point machine learning comes to play.

There are libraries and tools built for machine learning but we won’t be going that deep cause we are working with a no-code platform to achieve our goal.

We will be building our first basic model with a simple dataset and not the one stated in our example. This dataset is simple demo data generated randomly and not real-world data, the dataset can be downloaded from this link here and it contains information such as ‘age’, ‘gender’ and ‘genre’ with this dataset, we will be creating a model that helps recommend the type of genre a particular person would like or have much interest.

Here are the steps to creating or building our first basic model (music recommender):

Import the dataset
Clean the dataset
Split the data into training/test datasets
Create a model
Make prediction
Evaluate and improve

Note, make sure you have created an account on the Autogon AI console and you are currently seeing your dashboard then click on ‘My Projects’ and you should see something like this below for those with a new account, you will see a message indicating you have no project which its fine cause will be creating a project for to build our model.

Next, we create a project to indicate what we will be building in this case, I will call the project ‘Music Recommender’ (You can call the project any name you like) and I will type in some description. Here is how it looks

Before we launch the studio we need to upload the dataset to our Autogon account for easy usage on the studio, so let's navigate to ‘Dataset’ by clicking on it on the sidebar and we will have something like this below

The way we created the project, will do the same on ‘Dataset’ and you should have something similar to the image below

Now that we have our dataset saved on Autogon AI, we can go ahead and launch the studio to start building our model, so we navigate back to ‘My Projects’ and click on ‘View details’ of the project we created then click on ‘Launch Studio’ button and you should have this on your screen.

Once you are here (the image above), first we need to load up the dataset onto the studio and to do so we need to drag a block function called ‘Data Input’ under ‘Data Pre-processing’, next click on the block center and its properties at the right-side panel.

On the ‘Data Source’ input, we will be changing the value to ‘File’ and we have

Now we need to pass the dataset to the input labelled ‘File URL’, to get the dataset URL, click on the left panel icon with the name ‘Dataset’ and look for the ‘Music’ dataset and copy its URL and paste it inside the ‘File URL’ input as shown below

Click on the Play icon on the block function

Wait for the loading to stop and then click on the panel icon with the label ‘Variable’ and we are going to visualize the dataset like this

Now that we have loaded the dataset onto the studio we can move to the next step, which would be extracting the dataset into X and Y features (X stands for dependent variable and Y stands for independent variable). The Y feature will be what the model will be predicting while the X feature will be the input for the model.

Drag the ‘Feature Sampling’ and connect it to the ‘Data Input’ and also provide the block property with values before you run the block.

For the X boundaries, we will be using the value “:,:-1” and for Y boundaries will be using “:,-1”, now what these boundary values perform on the dataset will be, for the X boundary the first two columns on the dataset will be sliced to a new CSV file and for the Y boundary the last column will be sliced also to a new CSV file. For further information on how you can work with boundaries on Autogon Studio check out their documentation here. After clicking on the play button, you can also check the output under variables.

Here is my X Feature

Next, we are going to create/generate the training and test dataset from both features using the block function ‘Data Spliting’ (likewise we continue with the drag and drop).

From the image above, we could see the ‘Data Spliting’ block function has two properties, for this tutorial, I will only discuss ‘Test Data Size’ and for the ‘Random state’ we would ignore it.

‘Test Data Size’ takes a decimal value that indicates the size of how the algorithm should extract the train and test data from our dataset that is when you indicate a ‘Test Data Size’ of 0.1, the algorithm will split the dataset into a ratio of 90% train data and 10% of test data and if the ‘Test Data Size’ is 0.2 then we would get a train data of 80% and test of 20% and so on. Also note, the more train data your model has the better its accuracy.

At this point, we can now train the model with the training dataset and we will be using a model name ‘Decision Tree Classification’ block function and I will name my model ‘music-recommender’ and ‘Criterion’ with a value of ‘gini’ as shown below

Now, we need to make the model predict with the test dataset, so we will be dragging the predict block function ‘Decision Tree Classification predict’ and likewise, we will be filling the block properties with the same model name as the ‘Decision Tree Classification’ block function

On the highlighted panel, you notice we have a new variable called ‘Y_pred’, this is the CSV file that contains the prediction output and when you double-click on it, you see the output like this

If you would like to come up with your test data, you can use ‘Scalar to NDArray’ block function and I will be passing these test data (age: 21 and gender: male).

Note: in gender, female is zero and male is one

Click on the copy option to obtain the URL to the test dataset we generated and also note that the model has not seen this new test dataset we generated. We are going to paste the URL we copied to block function ‘Decision Tree Classification predict’ and added to ‘Test data’ property like this below

And when we run the block again, we have and checking the ‘Y_pred’

What this means is that the model is telling us that a 21-year-old male will like hip-hop compare to the rest of the genre. Now we want to see the accuracy of the model and to do that we need to drag the ‘Decision Tree Classification metric’ block function and we have

Note: You need to clear out the ‘Test data’ property and re-run the ‘Decision Tree Classification predict’ again

As you can see the accuracy is quite high which is a good thing for our model. We have successfully built our first model called ‘music-recommender’. The best part of using Autogon AI is that we don’t need to launch the studio all the time to make a prediction, to make another prediction, we are going to leave the studio and go to ‘Quick Predikt’ as shown below

Select the project ‘Music Recommender’ then select the model and either create a new dataset (test data) or select from the existing datasets but I will be using the same test data we generated earlier in the studio and we have this output

As you can see, we still get the same output as what we got from the studio.

This tutorial is based off Python Machine Learning Tutorial (Data Science)

How to create a floating label input with css-in-js and react

Deola Adediran — Thu, 18 Feb 2021 14:31:48 +0000

In case you are not familiar with JSS that is css-in-js. JSS is an authoring tool in css which allows you to use javascript to describe styles in a declarative, conflict-free and reusable ways, (as stated on JSS website).

What is Float Label?

These are input with different behaviour from the usual or normal way we understand input. To explain more on how it works; when an input is empty, the placeholder will act normal as expected, then when filled with text, it moves to the top as shown below

Create a new project

Let’s create a new react project (I’m using Yarn but you can also use npm)

$ yarn create-react-app floating-input

Once the project is created completely, remember to clear out unwanted files, remaining the App.js, we then create a div that contains the input and label like this

Now we need to style the above component with JSS but first we install react-jss

$ yarn add react-jss

Then we import createUseStyles from the package react-jss as shown below

Then we create our useStyle function after that we can now start styling our component, feel free to style the component according to your taste.

Note how I used useStyle() inside the App function to access the selector or object key as it’s in the form of an object. Then we create our JSS syntax like this and in my case

As you can see how the css in js is accessed on the jsx, classes is use to store the returned from useStyles function then inside the className, we pick out each css that should affect the html element. Now our input looks like this

Next we’ll be adding css animation to the above input, we start with the interaction, which consists of a transition and the behaviour of position absolute and relative combined. First we add position relative to JSS floatingLabelWrap

Then we add position absolute to our label and a transform to center the label similar to the way placeholder works.

We have our output of floating input as this

Now we complete the animation with transition and to use focus-within to apply the translate effect to change the position and scale label

Now we have this result but with a minor problem that needs to be fixed.

To fix the error as noticed above, we’ll be doing so in reactjs as follow:
Firstly we create a state using useState hooks as shown below

Next we add another state to manage the text inputted

Next we add a function handler on input onChange to toggle our JSS active which we haven’t yet written.

Finally, we add the JSS style for active

We have a full javascript enabled floating label input completed

Remember there are always a better way to achieve this and if you do have a better way, please i would love to see it. Thanks

Based on Creating Floating Label/Placeholder for Input with ReactJS - DEV Community 👩‍💻👨‍💻

How to use redux-form SubmissionError in react.

Deola Adediran — Mon, 15 Feb 2021 14:38:07 +0000

Redux-form is an open-source javascript, for managing form state in redux and a validation library that can be integrated into react web application so easy but i won’t be explaining how to integrate the library into a react project has i assume the reader is not a novice to redux-form or redux as a whole. In this tutorial will only look at how to use redux-form SubmissionError function to achieve something like this kind of error

Inside login presentation file, we’ll import reduxForm and Field from the redux-form library and also renderInput file

The above image is the Login presentation, we have handleSubmit, pristine, submitting props from redux-form. Next we create our renderInput file

The css selectors are from bootstrap. Next, we create our Login container, to perform data manipulation or whatever we need to use the data received from the presentation

We import connect from redux library, also our login action and the presentation file then create our function to be called when the submit button is clicked. Next we create our login action to send our values to the server or API or cloud function

From the above snippet, we imported SubmissionError and reset from redux-form library, from the beginning of this blog you know what we want to achieve with SubmissionError but reset looks new, so what reset does is to notify redux-form to reset the values inputted by the user in redux state.

At this stage we are not yet done, if you try to login and you receive an error from the server or from a third party API, you will notice that your app crashes with an error on the browser. To solve this we just need to add return or use async/await on our handleSubmit, like this

That’s all to SubmissionError from redux-form.

Conclusion

I hope you have a clear idea of how you can use redux-form SubmissionError in your react app. Hope to see more ways to implement this function.