DEV Community: Isheanesu

How To Make An Insanely Fast AI App (Supabase, LLAMA 3 and Groq)

Isheanesu — Tue, 23 Apr 2024 16:31:14 +0000

By now you've probably seen one of those demos of AI apps that are so fast that you don't even need to stream the responses. Developing a full-stack AI app can be very complicated and has taken many startups millions of dollars and hundreds of talented engineers. Fortunately, because we have access to so many amazing tools not only can you make one by yourself, you can build one in a weekend.

Here's what you'll use:

Supabase (start for free)
A Javascript framework for frontend (just pick any you like, there are billions)
LLAMA 3 via Groq Cloud (they have insanely fast chips for running AI models)

Now you know what you'll use it's time to get started

First step to building an AI app

Drink some water and touch some grass. Seriously, you've been in front of your laptop for too long and it's not healthy. Please go outside.

Setting up a Supabase project

Now you're back from outside (or more likely, just ignored that part), you're ready to set up a Supabase project. A managed version is a few clicks on supabase.com. Self-hosting is a bit more involved but if you want to go down that route please read this guide.

Getting Groq

Calling Supabase

Assuming you have some kind of frontend project already set up with npm, you'll want to install Supabase with this command:

npm i @supabase/supabase-js

You'll then be able to create a Supabase client with this code:

import { createClient } from '@supabase/supabase-js'
const supabaseUrl = 'YOUR_URL'
const supabaseKey = 'YOUR_KEY'
const supabase = createClient(supabaseUrl, supabaseKey)

Now, you have access to Supabase in your project.

Calling Llama 3 From the Edge

We'll use an edge function in Supabase to call Llama 3 on the Groq API. They're a super easy way to write custom code for a Supabase project.

You'll need to install the Supabase CLI in different ways depending on your OS, check out this guide for details and come right back here.

Run supabase init in your project folder to initialize Supabase and then run supabase functions new groq-api which will create a function called 'groq-api.'.

Open up the index.ts file and add this code:

import { serve } from 'https://deno.land/std@0.177.0/http/server.ts'

export const corsHeaders = {
  'Access-Control-Allow-Origin': '*',
  'Access-Control-Allow-Headers': 'authorization, x-client-info, apikey, content-type',
}

serve(async (req) => {
    if (req.method === 'OPTIONS') {
        return new Response('ok', { headers: corsHeaders })
    }
})

This is very important because we want to call our function from a browser. So when a browser sends an OPTIONS request, the function will respond with an ok.

Now add this under the if statement:

const GROQ_API_KEY = 'GROQ_API_KEY'

const data = fetch('https://api.groq.com/openai/v1/chat/completions', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
            'Authorization': `Bearer ${GROQ_API_KEY}`
        },
        body: JSON.stringify({
            "messages": [
                {
                    "role": "user",
                    "content": "Hello Llama!"
                }
            ],
            "model": "llama3-8b-8192",
            "temperature": 1,
            "max_tokens": 1024,
            "top_p": 1,
            "stream": false,
            "stop": null
        })
    })

This will call the Groq API without streaming the response and it will pass a message from the user that just says, "Hello Llama!"

Now let's allow the user to send their own message by adding this to the start of the serve function:

const { message } = await req.json()

That will destructure message from the request JSON. Then return the response from the fetch operation as follows:

const completion = response.json()['data']['content']

return new Response(JSON.stringify({ "completion": completion}), {
        headers: { ...corsHeaders, 'Content-Type': 'application/json' },
        status: 200,
    })

Your code should now look like this:

import { serve } from 'https://deno.land/std@0.177.0/http/server.ts'

export const corsHeaders = {
  'Access-Control-Allow-Origin': '*',
  'Access-Control-Allow-Headers': 'authorization, x-client-info, apikey, content-type',
}

serve(async (req) => {

    const { message } = await req.json()

    // This is needed if you're planning to invoke your function from a browser.
    if (req.method === 'OPTIONS') {
        return new Response('ok', { headers: corsHeaders })
    }

    const GROQ_API_KEY = 'GROQ_API_KEY'

    const response = await fetch('https://api.groq.com/openai/v1/chat/completions', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json',
            'Authorization': `Bearer ${GROQ_API_KEY}`
        },
        body: JSON.stringify({
            "messages": [
                {
                    "role": "user",
                    "content": "Hello Llama!"
                }
            ],
            "model": "llama3-8b-8192",
            "temperature": 1,
            "max_tokens": 1024,
            "top_p": 1,
            "stream": true,
            "stop": null
        })
    })

    const completion = response.json()['data']['content']

    return new Response(JSON.stringify({ "completion": completion}), {
        headers: { ...corsHeaders, 'Content-Type': 'application/json' },
        status: 200,
    })
})

Deploy your function by running supabase functions deploy groq-api from your project folder.

Now, all you need to do is use your function in the front project. Import Supabase and invoke:

import { supabase } from "/path/to/supabase"

        const { data, error } = await supabase.functions.invoke('groq-api', {
            body: { message: 'The message value from the user' }
        })

All done! Depending on your framework you'll handle variables and states differently but that part above is framework agnostic.

A few more things to note:

We didn't implement chat history, you could that in your Supabase DB and pull dynamically depending on who the user is
Use environment variables instead of hard coding API keys when possible

For now, you've got a great start. Time to scale it to millions.

Letting Supabase Users Transcribe YouTube Videos

Isheanesu — Tue, 10 Oct 2023 19:38:46 +0000

Have you ever wanted to let your Supabase users transcribe YouTube videos using an open-source model deployed to a serverless GPU platform? Probably way too specific for a random scenario (fair enough) but it's a fun project anyway.

So first things, first: How do we transcribe a video?

While we could get into the intricacies of machine learning models all day but that might be a little out of scope for this article so we are going to use OpenAI's Whispher model. It's open source and pretty good.

We'll need a GPU to run the model (technically not 100% necessary but we want our inference speed to be really fast). The problem is that GPUs are expensive - as you'll know if you've seen NVIDIA's stock price lately...

Thankfully, serverless GPUs have really caught on lately and for this video we'll be using a platform called Beam to deploy our Whispher. They have a great article on this very topic. Our focus is what to do when the model is ready for inference by our Supabase users.

So first of all, how do we call it in Supabase? Let's use an edge function.

Start a Supabase project
Install the CLI and create a new function by calling supabase functions new transcribe-yt (transcribe-yt is our function name)

Our index.ts should look something like this:

// Follow this setup guide to integrate the Deno language server with your editor:
// https://deno.land/manual/getting_started/setup_your_environment
// This enables autocomplete, go to definition, etc.

import { serve } from "https://deno.land/std@0.168.0/http/server.ts"

console.log("Hello from Functions!")

serve(async (req) => {
  const { name } = await req.json()
  const data = {
    message: `Hello ${name}!`,
  }

  return new Response(
    JSON.stringify(data),
    { headers: { "Content-Type": "application/json" } },
  )
})

// To invoke:
// curl -i --location --request POST 'http://localhost:54321/functions/v1/' \
//   --header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZS1kZW1vIiwicm9sZSI6ImFub24iLCJleHAiOjE5ODM4MTI5OTZ9.CRXP1A7WOeoJeXxjNni43kdQwgnWNReilDMblYTn_I0' \
//   --header 'Content-Type: application/json' \
//   --data '{"name":"Functions"}'

Let's change the 'name' parameter to 'video_url' first:

  const { video_url } = await req.json()
  const data = {
    message: `Hello ${video_url}!`,
  }

Now let's actually call the model on Beam:

fetch("https://apps.beam.cloud/<ID_FROM_BEAM>", {
  method: "POST",
  headers: {
    "Authorization": "[YOUR_AUTH_TOKEN]",
    "Content-Type": "application/json"
  },
  body: JSON.stringify({
    video_url: video_url
  })
})
  .then(response => response.json())
  .then(data => console.log(data))
  .catch(error => console.error(error));

We will use .then today, and data['pred'] will return our transcription and we can now return our transcription to the user like this:

  .then(data => {
    const transcription = data['pred']

    return new Response(
      JSON.stringify({ transcription }),
      { headers: { "Content-Type": "application/json" } },
    )
  })

Congratulations, you've done it!

BUT...

One more thing, anyone who logins in can blow up your bill and bankrupt your cool new startup. We need to do something else and that's figuring out who is asking for a transcription. By 'logging in as the user' (in a sense), first we'll import our favourite SDK

import { createClient } from '@supabase/supabase-js'

Then we can get our user plan:

    const { user } = await supabaseClient.auth.getUser()
    const user_id = user.id

    // Now we can run any query to check user usage, billing plan, etc as them
    const { data, error } = await supabaseClient
      .from('user_plans')
      .select('plan')
      .eq('user_id', user_id)
      .single()

Hopefully, you're already using excellent Row Level Security policies and then if a user isn't on a supported plan you would need to cancel the request and send an error.

Here's how all the code looks together:

// Follow this setup guide to integrate the Deno language server with your editor:
// https://deno.land/manual/getting_started/setup_your_environment
// This enables autocomplete, go to definition, etc.

import { serve } from "https://deno.land/std@0.168.0/http/server.ts"

console.log("Hello from Functions!")

serve(async (req) => {
  const { video_url } = await req.json()

    // We use our client here
    const supabaseClient = createClient(
      // Use the URL and anon key (these will be in your Supabase project)
      Deno.env.get('SUPABASE_URL') ?? '',
      Deno.env.get('SUPABASE_ANON_KEY') ?? '',
      { global: { headers: { Authorization: req.headers.get('Authorization')! } } }
    )

    const { user } = await supabaseClient.auth.getUser()
    const user_id = user.id

    // Now we can run any query to check user usage, billing plan, etc as them
    const { data, error } = await supabaseClient
      .from('user_plans')
      .select('plan')
      .eq('user_id', user_id)
      .single()

  fetch("https://apps.beam.cloud/<ID_FROM_BEAM>", {
    method: "POST",
    headers: {
      "Authorization": "[YOUR_AUTH_TOKEN]",
      "Content-Type": "application/json"
    },
    body: JSON.stringify({
      video_url: video_url
    })
  })
  .then(response => response.json())
  .then(data => {
    const transcription = data['pred']

    return new Response(
      JSON.stringify({ transcription }),
      { headers: { "Content-Type": "application/json" } },
    )
  })
  .catch(error => console.error(error));

  return new Response(
    JSON.stringify(data),
    { headers: { "Content-Type": "application/json" } },
  )
})

// To invoke:
// curl -i --location --request POST 'http://localhost:54321/functions/v1/' \
//   --header 'Authorization: Bearer eyJhbGciOiJIUzI1NiIsInR5cCI6IkpXVCJ9.eyJpc3MiOiJzdXBhYmFzZS1kZW1vIiwicm9sZSI6ImFub24iLCJleHAiOjE5ODM4MTI5OTZ9.CRXP1A7WOeoJeXxjNni43kdQwgnWNReilDMblYTn_I0' \
//   --header 'Content-Type: application/json' \
//   --data '{"name":"Functions"}'

A super helpful hint to save you some research: If your users are invoking the functions from a browser use CORS.

There are some more things that can be done to optimise this function (like handling those tricky catch statements but for now this is enough to get you started on your AI adventure.

If you have any questions leave a comment and I (not AI) will answer.