Building an Image Recognition Website with SvelteKit and TensorFlow.js

Estelle-K — Tue, 20 Feb 2024 07:39:22 +0000

Introduction

In this article, I'll show you how to build a simple website that allows users to upload an image and get predictions about its content using MobileNet model from TensorFlow.js. This project is perfect for beginners in TensorFlow.js who want to get started with machine learning in the browser.

Prerequisites

Before you start, you should have some knowledge of web development and be familiar with JavaScript, TypeScript and Tailwind CSS. Make sure the following tools are also installed:

npm or yarn
VSCode (https://code.visualstudio.com/download)
Svelte extension (https://marketplace.visualstudio.com/items?itemName=svelte.svelte-vscode)

Step 1: Create a new SvelteKit project

Start by creating a new SvelteKit project using the following command in the terminal:



npm init svelte@next my-project

Step 2: Install Tailwind CSS and Flowbite Svelte

Tailwind CSS is a CSS framework. https://tailwindcss.com/
Flowbite Svelte is an official Flowbite component library for Svelte. https://flowbite-svelte.com/

Install Tailwind CSS in a SvelteKit project in the terminal:



npm install -D tailwindcss postcss autoprefixer
npx tailwindcss init -p

Update your tailwind.config.js file:



/** @type {import('tailwindcss').Config} */
export default {
  content: ['./src/**/*.{html,js,svelte,ts}'],
  theme: {
    extend: {}
  },
  plugins: []
};

Create ./src/app.css file:



@tailwind base;
@tailwind components;
@tailwind utilities;

Create a ./src/routes/+layout.sveltefle and importe the app.css file:



<script>
  import "../app.css";
</script>

<slot />

Install Flowbite Svelte in the terminal:



npm i -D flowbite-svelte flowbite

Update your tailwind.config.js file:



/** @type {import('tailwindcss').Config} */
export default {
    content: [
        './src/**/*.{html,js,svelte,ts}',
        './node_modules/flowbite-svelte/**/*.{html,js,svelte,ts}'
    ],
    theme: {
        extend: {
            colors: {
                // flowbite-svelte
                primary: {
                    50: '#FFF5F2',
                    100: '#FFF1EE',
                    200: '#FFE4DE',
                    300: '#FFD5CC',
                    400: '#FFBCAD',
                    500: '#FE795D',
                    600: '#EF562F',
                    700: '#EB4F27',
                    800: '#CC4522',
                    900: '#A5371B'
                }
            }
        }
    },
    plugins: [require('flowbite/plugin')]
};

Step 3: Install TensorFlow.js

TensorFlow.js documentation https://www.tensorflow.org/js
Pre-trained TensorFlow.js models https://github.com/tensorflow/tfjs-models#readme

Install TensorFlow.js in the terminal:



npm i @tensorflow-models/mobilenet @tensorflow/tfjs @tensorflow/tfjs-vis

Step 4: Import the MobileNet model

Go to ./src/routes/+page.svelte and delete everything there.

Create a <script lang="ts"></script> tag and add these imports:



<script lang="ts">
    import * as tf from '@tensorflow/tfjs';
    import * as mobilenet from '@tensorflow-models/mobilenet';
</script>

Step 5: Create a function to load the model

Create a function to load the MobileNet model and return it. Add this code in script tag under imports:



let model: mobilenet.MobileNet;

const loadModel = async () => {
        const version = 2;
        const alpha = 0.5;
        model = await mobilenet.load({ version, alpha });
        console.log('Model loaded successfully');
        return model;
    };

Step 6: Create a function to preprocess an image before using it in a machine learning model.

Add this code in script tag below the previous function:



const preprocessImage = (imageElement: HTMLImageElement) => {
        const imageTensor = tf.browser.fromPixels(imageElement);
        const resizedImageTensor = tf.image.resizeBilinear(
            imageTensor,
            [224, 224]
        );
        return resizedImageTensor;
    };

Explanation of the preprocessImage function
Feature: Preprocess an image before using it in a machine learning model.

Inputs:

imageElement: An HTMLImageElement representing an image in the HTML DOM.

Steps:

Convert the image to 3D tensor (height, width, color channels) with tf.browser.fromPixels.
Resize the image to 224x224 pixels using bilinear interpolation: tf.image.resizeBilinear.

Output:

A tensor representing the preprocessed image, ready for use in a machine learning model.

Step 7: Create a function to make a prediction

Add this code in script tag below the previous function:



const classifyImage = async (
        model: mobilenet.MobileNet,
        preprocessedImage: tf.Tensor3D
    ) => {
        const predictions = await model.classify(preprocessedImage);
        return predictions;
    };

Step 8: Put it all together

Now we need to combine the functions we created to load the model and make predictions. Add this code in script tag below the previous function:



let predictionsResult: Prediction[] = [];

const predictionImage = async (imgDisplay: HTMLImageElement) => {
        const model = await loadModel();
        const preprocessedImage = preprocessImage(imgDisplay);
        predictionsResult = await classifyImage(model, preprocessedImage);
    };

Step 9: Create an user interface with Flowbite Svelte and Tailwind CSS.

I put here the complete code of the +page.svelte file. I added some elements like a spinner or a TypeScript interface.



<script lang="ts">
    import * as tf from '@tensorflow/tfjs';
    import * as mobilenet from '@tensorflow-models/mobilenet';
    import {
        Label,
        Fileupload,
        Heading,
        List,
        Li,
        Card,
        Spinner
    } from 'flowbite-svelte';

    let fileuploadprops = {
        id: 'inputImage'
    };

    let predictionsResult: Prediction[] = [];
    let imgDisplay: string;
    let files: FileList;
    let imgtoprocess: HTMLImageElement;
    let model: mobilenet.MobileNet;
    let predictionsLoading = false;

    $: if (files) {
        imgDisplay = URL.createObjectURL(files[0]);
    }

    interface Prediction {
        className: string;
        probability: number;
    }

    const loadModel = async () => {
        const version = 2;
        const alpha = 0.5;
        model = await mobilenet.load({ version, alpha });
        console.log('Model loaded successfully');
        return model;
    };

    const preprocessImage = (imageElement: HTMLImageElement) => {
        const imageTensor = tf.browser.fromPixels(imageElement);
        const resizedImageTensor = tf.image.resizeBilinear(
            imageTensor,
            [224, 224]
        );
        return resizedImageTensor;
    };

    const classifyImage = async (
        model: mobilenet.MobileNet,
        preprocessedImage: tf.Tensor3D
    ) => {
        const predictions = await model.classify(preprocessedImage);
        return predictions;
    };

    const predictionImage = async (imgDisplay: HTMLImageElement) => {
        predictionsLoading = true;
        const model = await loadModel();
        const preprocessedImage = preprocessImage(imgDisplay);
        predictionsResult = await classifyImage(model, preprocessedImage);
        predictionsLoading = false;
    };

    const handleImageChange = () => {
        predictionImage(imgtoprocess);
        URL.revokeObjectURL(imgDisplay);
    };
</script>

<svelte:head>
    <title>ML Vision</title>
    <meta
        name="description"
        content="A simple ML Vision app"
    />
</svelte:head>

<section class="flex flex-col gap-12">
    <Heading tag="h1" class="text-center">ML Vision with MobileNet images Classifier</Heading>
    <div class="ml-auto mr-auto block w-1/2">
        <Label class="pb-2 text-lg">Select an image to classify</Label>
        <Fileupload
            {...fileuploadprops}
            bind:files
            on:change={handleImageChange}
        />
    </div>
    <Card
        size="md"
        padding="sm"
        class="ml-auto mr-auto"
    >
        <img
            hidden={!imgDisplay}
            src={imgDisplay}
            alt=""
            bind:this={imgtoprocess}
        />
        <div class="py-4">
            <Heading tag="h3">Predictions:</Heading>
            <List
                position="outside"
                class="p-4"
            >
                {#if predictionsLoading}
                    <span class="flex flex-col items-center"
                        ><Spinner color="blue" /></span
                    >
                {:else}
                    {#each predictionsResult as prediction}
                        <Li
                            >{prediction.className}: {Math.round(
                                prediction.probability * 100
                            )}%</Li
                        >
                    {/each}
                {/if}
            </List>
        </div>
    </Card>
</section>

Step 10: Add Svelte components

I create a Footer and Menu components and display them in +layout.svelte file.

Create a file ./src/lib/components/footer/Footer.svelte and add this code:



<script>
    import {
        Footer,
        FooterCopyright,
    } from 'flowbite-svelte';
</script>

<Footer class="rounded-none bg-black p-4 shadow md:px-6 md:py-8">
    <hr class="my-6 border-white sm:mx-auto lg:my-8" />
    <FooterCopyright
        aClass="text-[#6C70ED] text-base"
        spanClass="text-white block text-sm text-center"
        href="/"
        by="Flowbite"
    />
</Footer>

Create a file ./src/lib/components/menu/Menu.svelte and add this code:



<script lang="ts">
    import {
        Navbar,
        NavBrand,
        NavLi,
        NavUl,
        NavHamburger
    } from 'flowbite-svelte';
</script>

<Navbar
    let:hidden
    let:toggle
    class="bg-black text-white"
>
    <NavBrand href="/">
    </NavBrand>
    <NavHamburger on:click={toggle} class="hover:bg-[#6C70ED]" menuClass="outline-[#6C70ED]" />
    <NavUl {hidden} ulClass="bg-black flex flex-col p-4 mt-4 md:flex-row md:space-x-8 rtl:space-x-reverse md:mt-0 md:text-sm md:font-medium" divClass="w-full md:block md:w-auto">
        <NavLi
            href="/"
            active={true} class="md:text-white md:hover:text-[#6C70ED]" nonActiveClass="text-white">Home</NavLi
        >
    </NavUl>
</Navbar>

Then edit the ./src/routes/+layout.svelte file with this code:



<script>
    import Menu from '$lib/components/menu/Menu.svelte';
    import Footer from '$lib/components/footer/Footer.svelte';
    import '../app.css';
</script>

<div class="flex flex-col min-h-screen">
    <Menu />

    <main class="flex flex-col flex-1 p-4 w-full box-border max-w-5xl mt-0 mb-0 ml-auto mr-auto">
        <slot />
    </main>

    <Footer />
</div>

Step 11: Try your project

To run your project:



npm run dev -- --open

Choose an image with a dog or a vegetable. See the result!

Conclusion

In this article, you learned how to create a simple image recognition website with SvelteKit and TensorFlow.js. This project is a great starting point for learning about machine learning in the browser.

Project on Github

https://github.com/Estelle-K/ml-vision

DEV Community: Estelle-K