babel_fish - real time language translation

This is a submission for the Cloudflare AI Challenge.

What we built

Douglas Adams came up with the idea of the Babel fish in Hitchhiker's Guide to the Galaxy. It's a fish that, once you've stuck it in your ear, allows any language to be translated to you in real time.

babel_fish is a real time language translation app built with Cloudflare, Svelte, and multiple language models. The app allows you to share a room with your friends in different languages. All messages get translated to your chosen language in real time!

@alicelovescake and I thought this would be a perfect app for trying out the fancy new Cloudflare Workers AI!

Demo

Le code

Frontend

The Svelte application contains the frontend code as well as all of our API endpoints interacting with the language models.

amorriscode / babel-fish

A real time translation app

babel_fish <><

This is a submission for the Cloudflare AI Challenge.

babel_fish is a real time language translation app built with Cloudflare, Svelte, and multiple language models. The app allows you to share a room with your friends in different languages. All messages get translated to your chosen language in real time!

This repo contains the Svelte frontend which includes API routes for interacting with the AI. The chat engine is handled by the backend.

Built with:

• 💅 Svelte
• ⚙️ Cloudflare Workers
• 🤖 Cloudflare Workers AI
• 📣 ElevenLabs

🚧 Requirements

• An ElevenLabs API key
• A running version of the backend

🛠️ Development

Note: wrangler is used to run the dev server so we have access to Cloudflare Workers AI

1. Create an .env file and fill in the variables

   cp .env.example .env

2. Clone the frontend

   git clone https://github.com/amorriscode/babel-fish

3. Install dependencies

   npm install

4. Run the wrangler dev server

   npm run…

View on GitHub

Backend

Our web socket backend uses Cloudflare Workers and Durable Storage. It's a simple logic engine that powers the chat rooms.

amorriscode / babel-fish-ws

Chat engine for the real time translation app babel_fish

babel_fish <>< ws

This is a submission for the Cloudflare AI Challenge.

babel_fish is a real time language translation app built with Cloudflare, Svelte, and multiple language models. The app allows you to share a room with your friends in different languages. All messages get translated to your chosen language in real time!

This repo contains the websocket backend that powers the babel_fish frontend;

Built with:

• ⚙️ Cloudflare Workers
• 📦 Cloudflare Durable Objects

🚧 Requirements

• A paid Cloudflare account to use Durable Objects

🛠️ Development

1. Clone the repo

   git clone https://github.com/amorriscode/babel-fish-ws

2. Install dependencies

   npm install

3. Run the dev server

   npm run wev

View on GitHub

Journey

Tired: trying out one new technology during a hackathon
Wired: trying out tons of new technology during a hackathon

I've been admiring Cloudflare from afar for a while now. I love what they've been doing with Pages, Workers, and now AI Workers. This hackathon finally gave me a good excuse to play around with the platform!

Aside from all the fancy tooling Cloudflare provide, this was also my first time using Svelte (sorry Rich!). I actually really enjoyed the experience. I'd love to spend more time writing Svelte; I have a lot left to learn.

So... AI?

We used a ton of Cloudflare's available AI models to build babel_fish.

• @cf/openai/whisper for voice-to-text
• @hf/thebloke/mistral-7b-instruct-v0.1-awq for language detection
• @cf/meta/m2m100-1.2b for language translation

Since a true Babel fish doesn't produce text we also used ElevenLabs for text-to-speech.

Challenges

We only had a single day to work on our project. The tight timeline and all new tech made it an exciting 24 hours.

The biggest challenge of this project actually ended up being the backend audio processing. With the initial POC, user messages weren't transcribed and sent over the wire until they were completely finished speaking. We were able to improve this by using AudioWorklets. Now we regularly check for gaps in a user's message and periodically process and broadcast messages.

The rest was easy because Cloudflare and Svelte made it so. 😎

Thanks to @cloudflaredev and @thepracticaldev for putting on this hackathon. If you're interested in this kind of thing, come work on experimental AI projects with me at Stripe!

<><