OpenAI Text To Speech (TTS) Streaming with FastAPI + Python + React Frontend

Trollgen Studios — Tue, 31 Dec 2024 03:13:02 +0000

This code worked for me to get the chunks streaming in:
`await websocket.send_text('|AUDIO_START|')

            with openai_client.audio.speech.with_streaming_response.create(
                model="tts-1",
                voice="nova",
                response_format="mp3",  # Changed to mp3 format
                input=text_message,
            ) as response:
                for chunk in response.iter_bytes(chunk_size=1024):
                    await websocket.send_bytes(chunk)


            await websocket.send_text('|AUDIO_END|')`

Then, the challenge was figuring it out on the frontend on React.

// Handle incoming messages
    useEffect(() => {
      if (lastMessage?.data === '|AUDIO_START|') {
        setAudioBuffers([]); // Clear existing buffers
        setIsAudioStreaming(true);
        chunksRef.current = [];
      } else if (lastMessage?.data === '|AUDIO_END|') {
        setIsAudioStreaming(false);
        playAudioChunks();
      } else if (isAudioStreaming && lastMessage?.data) {
        console.log('trying to parse audio chunk:');
        console.log(lastMessage.data);
        chunksRef.current.push(lastMessage.data);
        // playAudioChunk(lastMessage.data);
      }
}, []);

const playAudioChunks = async () => {
      const audioBlob = new Blob(chunksRef.current, { type: 'audio/mp3' });
      const audioUrl = URL.createObjectURL(audioBlob);
      const audio = new Audio(audioUrl);

      try {
        await audio.play();
      } catch (err) {
        console.error('Error playing audio:', err);
      }

      // Clean up the URL after audio is done playing
      audio.onended = () => {
        URL.revokeObjectURL(audioUrl);
      };
    };

Sorry for the bad formatting, but hope this helps if you are struggling.

I am trying to figure out how to play chunk by chunk the best way. If I play each chunk as it comes in, the audio sounds weird so need to look into modifying this code (a larger chunk size? a delay?). Feel free to comment if any insights to share with the community.

const playAudioChunk = async (chunk) => {
      const audioBlob = new Blob([chunk], { type: 'audio/mp3' });
      const audioUrl = URL.createObjectURL(audioBlob);
      const audio = new Audio(audioUrl);

      try {
        await audio.play();
      } catch (err) {
        console.error('Error playing audio:', err);
      }

      // Clean up the URL after audio is done playing
      audio.onended = () => {
        URL.revokeObjectURL(audioUrl);
      };
    };

Google Sign In and Google Calendar API tutorial with client and server (iOS and Pyhton backend)

Trollgen Studios — Sat, 11 Nov 2023 06:12:59 +0000

I'll briefly show you how to do 2 things:
1) Logging in on iOS with Google
2) Getting the access and refresh tokens

Here are the steps:

Go to Google API and enable Google Calendar
Create OAuth Consent Screen
Generate a OAuth for your frontend (iOS / Android / Web)
Integrate sign in with Firebase
Make sure to add scope on sign in with Google (this is for iOS but similar for other frontends)

let additionalScopes = [
                "https://www.googleapis.com/auth/calendar.readonly",
                "https://www.googleapis.com/auth/userinfo.email",
                "https://www.googleapis.com/auth/userinfo.profile",
            ]
            let result2 = try await gidSignInResult.user.addScopes(additionalScopes, presenting: getRootViewController())

Then a serverAuthCode will be returned. Call a backend POST route to obtain access and refresh tokens (more details here: https://developers.google.com/identity/sign-in/ios/offline-access)

Use Google’s client libraries on backend to generate

credentials = client.credentials_from_clientsecrets_and_code(
            CLIENT_SECRET_FILE,
            ['https://www.googleapis.com/auth/calendar.readonly'],
            auth_code,
            redirect_uri="" # this must be there!
        )
if gcal_access_token:
  creds = Credentials.from_authorized_user_info(
   gcal_access_token,
   scopes=CALENDAR_SCOPES
  )
  print(creds.to_json())
 if not creds or not creds.valid:
  # If the credentials are expired, refresh them
  if creds and creds.expired and creds.refresh_token:
   creds.refresh(Request())
   print(creds.to_json())

You can then use the creds with the library to get events:


   calendar_event_result = service.events().list(
    calendarId="primary",
    singleEvents=False,
    orderBy="updated",
   ).execute()

I spent at least 10 hours of my life on navigating this maze due to improper documentation on how to authenticate with google sign in on frontend and then use Google Calendar API on backend, so hope this helps someone!

BY the way, if on the server you get a redirect_uri_mismatch bug, make sure to enable your localhost / production urls and then also have redirect_uri="" above

DEV Community: Trollgen Studios

OpenAI Text To Speech (TTS) Streaming with FastAPI + Python + React Frontend

Google Sign In and Google Calendar API tutorial with client and server (iOS and Pyhton backend)