DEV Community

Marcostx7
Marcostx7

Posted on

Learning Languages through a web application with SashiDo and Teachable Machine

#machinelearning #tutorial #beginners #javascript

Alt Text

Table of contents

Introduction

Hi everyone! I and my friend Aluisio @amorim33 are Brazilian High-School students doing an internship at SashiDo and, in this tutorial, we will talk a little about our project and explain how we made it.

First of all, we are going to talk about our idea. Our project is called LangEduc, which is an abbreviation of the words "Language" and "Education". Basically, this project aims to help children to learn other languages, especially English. In Brazil, toddlers have lots of difficulties learning English, mostly because the educational system here is very theoretical and does not encourage children to learn English as well as other foreign languages dynamically.

Before developing the project, we made lots of searches and found out that toddlers can learn languages easily if they learn with images. Thus, we had the idea to develop an application using Teachable Machine, to make it easier for children to have their first contact with foreign languages and start their learning, something that will be very important if they decide to actually study deeply the chosen language.

Intentions

We would like to say that we tried to develop a simple application in order to make it easier for beginners to get in contact with the main programming languages as well as with SashiDo, a company that provides a very great back-end support to any project you have in mind.

Therefore, if you just learned HTML, CSS and JavaScript, we think that this project could be very interesting to you as a beginner.

Now, that we already explained our idea and intentions, let's get started with the tutorial!

Teachable Machine

The first part of our application should be made in the Teachable Machine. Do not feel afraid if you did not learn about Machine Learning yet because that machine is intended to make our work as simple as possible using an intuitive interface.

Organization

Making a very good project using the Teachable Machine can be very simple but it requires lots of organization. Therefore, if you would like to do your best working with lots of pictures, I suggest that you should create one folder for each class you intend to implement in your project.

For this project, we intended to work with animal pictures since knowing the names of animals in foreign languages is a very good way to start your learning. Thus, I created folders with the names of the animals I intended to create classes for on the Teachable Machine.

Alt Text

I think that this step is inextricably linked with a good development because as many images you register on the Teachable Machine, the better it recognizes standards. Thus, if you want to register lots of images, this type of organization will be very important.

Working with Teachable Machine

After that, you should go to Teachable Machine to actually start the work.

Basically, the process is simple, you just have to go to Google Images or any other image database you want to get the images you need to upload on Teachable Machine.

To get better organized, I recommend you to download in the folders all the images you need before starting to upload them on Teachable Machine. I think that you can work quickly by following those steps but feel free to follow the order you do your best.

After uploading all those images, it is time to name the classes we made. Since we aim to teach languages to our users we should name the classes with the name of the animal that appears on the pictures we uploaded. We also intend to teach more than 1 language, therefore, we will name the classes in this way: NameInEnglish,NameInPortuguese,NameInSpanish, e.g., Butterfly,Borboleta,Mariposa

Alt Text

OBS: The words "Borboleta" and "Mariposa" mean Butterfly in Portuguese and Spanish respectively.

Make sure you know the way you are spacing the names because we will use this character disposition when coding the app to show the name of the animal in the language chosen by the user when he/she uploads an image or opens his/her webcam.

Training and exporting your image model

For this step, you just have to click on the "Train Image" button on the right of the classes you already created. For our project, the standard settings in which Teachable Machine is trained were enough but we encourage you to play a little with the advanced settings to get better familiarized with the machine.

When the machine finishes its training you can test your model right in the Teachable Machine's interface or you can just export the generated code to implement it in your application's code.

Alt Text

For this project, we will use Tensorflow.js to implement the model on our application but feel free to export your project in Tensorflow or Tensorflow Lite if you already know how to work with them.

Now, you should click on "Upload my model". Copy the code and save it for later, in the Creating the functions and implementing Teachable Machine section, we will show how to implement the model on the application code.

Registration using SashiDo and Parse

For this process, we have to link our app to SashiDo

  • At first, go to your SashiDo dashboard

  • Click Create a New App and follow the instructions.

  • Now you may add a new column in your DB browser in class "Role" and name it as “Users” like us, or you may choose another name you prefer

Alt Text

  • Go to the “Getting Started” page, click on “JavaScript” tab and then copy the code at the bottom

Alt Text

  • Next, we will move up to the application code. First we have to create the form in the HTML file 
<div id="form-container">
        <form id="form">
            <label for="username">Username:</label>
            <input type="text" id="username" name="username">
            <label for="pass">Password:</label>
            <input type="password" id="pass" name="pass">
        </form>
        <div id="buttons=container">
            <button type="button" onclick="logIn()" id="login">LogIn</button>
            -
            <button type="button" onclick="signUp()" id="signup">SignUp</button>
        </div>
        <span id="returnMsg"></span>
</div>
Enter fullscreen mode Exit fullscreen mode
  • Add the Parse script at the bottom of your HTML file 
<script src="https://cdnjs.cloudflare.com/ajax/libs/parse/3.1.0/parse.min.js"></script>
Enter fullscreen mode Exit fullscreen mode
  • Finally, go to your JS file and paste here the code that we already copied in the SashiDo dashboard. That code will be responsible for connecting your web-app with SashiDo’s servers. 
Parse.initialize(
"QXm1DnE7daKvXNZSzWDqu4ETRQJjT9TFVRMVNSpS",
"nZWhqfsMjfdTtzXqwAWiHV9hTJfMNjWRTTUVMRNF");

Parse.serverURL = "https://pg-app-ux3nnb9n64wcjhjysie6zdc5fdd1x8.scalab1.cloud/1/";
Enter fullscreen mode Exit fullscreen mode
  • Create the functions to handle: “logIn” and “signUp”. Here we used Parse to make the requests. If you want to go deeper in the explanations, feel free to check Parse documentation 
async function logIn() {
  const username = document.getElementById("username").value;
  const pass = document.getElementById("pass").value;
  const formContainer = document.getElementById("form-container");
  const container = document.getElementById("container");
  let span = document.getElementById("returnMsg");

  const onFulfilled = () => {
    formContainer.parentNode.removeChild(formContainer);
    container.className = "";
  };

  const onRejected = () => {
    span.textContent = "Wrong user or password";
    span.className = "redSpan";
  };

  const user = await Parse.User.logIn(username, pass).then(
    onFulfilled,
    onRejected
  );
}
async function signUp() {
  const username = document.getElementById("username").value;
  const pass = document.getElementById("pass").value;
  let span = document.getElementById("returnMsg");
  const user = new Parse.User();
  user.set("username", username);
  user.set("password", pass);

  try {
    await user.signUp();
    span.textContent = "Successfully signed Up";
    span.className = "blueSpan";
  } catch (error) {
    span.textContent = "Error: " + error.code + " " + error.message;
    span.className = "redSpan";
  }
}
Enter fullscreen mode Exit fullscreen mode

Creating the functions and implementing Teachable Machine

Alt Text

  • The first step is to create the container in the HTML file, next we have to put all the game body elements inside of it. Please, pay attention to the scripts because they will allow you to use external functions from Teachable Machine and Parse. 
<div id="container" class = "d-none">
        <div id="header">
            <img src="./assets/LangEduc (3).png" id="logo"/>
            <button type="button" onclick="languageHandler()" id="lang">English</button>
        </div>

        <div id="buttons-container">
            <button type="button" onclick="startCamHandler()" id="camButton">Start Webcam</button>
            <button type="button" onclick="startUpHandler()" id="upButton">Upload Image</button>

        </div>

        <div id="game-container">
            <div id="webcam-container" class="d-none"></div>


            <input type="file" id="inp" class="d-none">
            <canvas id="canvas" class="d-none"></canvas>

            <div id="label-container" class="d-none"></div>
        </div>
        <script src="https://cdn.jsdelivr.net/npm/@tensorflow/tfjs@1.3.1/dist/tf.min.js"></script>
        <script src="https://cdn.jsdelivr.net/npm/@teachablemachine/image@0.8/dist/teachablemachine-image.min.js"></script>

        <script src="https://cdnjs.cloudflare.com/ajax/libs/parse/3.1.0/parse.min.js"></script>

        <script src="./index.js"></script>
</div>
Enter fullscreen mode Exit fullscreen mode
  • Go back to the JavaScript file and put the Teachable Machine model URL you created in a constant. 
//Teachable Machine model Url
const URL = "https://teachablemachine.withgoogle.com/models/0VVapoU7Y/";
Enter fullscreen mode Exit fullscreen mode
  • Now, we will create the variables we will need later 
let model, webcam, labelContainer, maxPredictions;
//is started Webcam flag
let startCamFlag = true;
//is started Upload flag
let startUpFlag = true;

let language = "ENG",
  languageCont = 0;

let camButton = document.getElementById("camButton"),
  upButton = document.getElementById("upButton");
Enter fullscreen mode Exit fullscreen mode
  • Here we have the functions to handle the buttons 
function startCamHandler() {
  if (startUpFlag) {
    if (startCamFlag) init();
    else stop();
    startCamFlag = !startCamFlag;
  }
}

function startUpHandler() {
  if (startCamFlag) {
    if (startUpFlag) openUploadImage();
    else closeUploadImage();
    startUpFlag = !startUpFlag;
  }
}
Enter fullscreen mode Exit fullscreen mode

Webcam initialization and finalization. 

// Load the image model and setup the webcam
async function init() {
  const modelURL = URL + "model.json";
  const metadataURL = URL + "metadata.json";

  // load the model and metadata
  // Refer to tmImage.loadFromFiles() in the API to support files from a file picker
  // or files from your local hard drive
  // Note: the pose library adds "tmImage" object to your window (window.tmImage)
  model = await tmImage.load(modelURL, metadataURL);
  maxPredictions = model.getTotalClasses();

  // Convenience function to setup a webcam
  const flip = true; // whether to flip the webcam
  webcam = new tmImage.Webcam(400, 400, flip); // width, height, flip
  await webcam.setup(); // request access to the webcam
  await webcam.play();
  window.requestAnimationFrame(loop);

  // append elements to the DOM
  document.getElementById("webcam-container").appendChild(webcam.canvas);
  labelContainer = document.getElementById("label-container");

  labelContainer.appendChild(document.createElement("div"));

  //Changing button text
  if (language == "ENG") camButton.textContent = "Stop";
  else if (language == "PORT") camButton.textContent = "Parar";
  else if (language == "SPA") camButton.textContent = "Detener";

  //Showing containers
  document.getElementById("webcam-container").className = "";
  document.getElementById("label-container").className = "";
}
Enter fullscreen mode Exit fullscreen mode 
async function stop() {
  await webcam.stop();
  document.getElementById("webcam-container").removeChild(webcam.canvas);

  labelContainer = document.getElementById("label-container");
  console.log(labelContainer.children);

  labelContainer.removeChild(labelContainer.children[0]);

  //Changing button text
  if (language == "ENG") camButton.textContent = "Start Webcam";
  else if (language == "PORT") camButton.textContent = "Começar Webcam";
  else if (language == "SPA") camButton.textContent = "Comenzar Webcam";

  //Hiding containers
  document.getElementById("webcam-container").className = "d-none";
  document.getElementById("label-container").className = "d-none";
}
Enter fullscreen mode Exit fullscreen mode
  • Now, you should create the loop in which the webcam frames and the predictions will be updated. Those predictions show the class that is most similar to the image in the webcam. 
async function loop() {
  webcam.update(); // update the webcam frame
  await predict();
  window.requestAnimationFrame(loop);
}
Enter fullscreen mode Exit fullscreen mode
  • Now, let’s create the “Predict()” function. At first, we added an argument that, if nothing is passed, it is by default defined as “webcam.canvas”, due to the fact we use this function to “predict” the "webcam" and the "upload images" input. Then, we call the predict function of Teachable Machine passing the webcam's canvas as a parameter, then, it returns the percentages of similarity. The following block of code was made to take the highest percentage class name in the selected language. Finally, we put this class name inside the label we have created. 
// run the webcam image through the image model
async function predict(imageModel = webcam.canvas) {
  let highestProbability;
  let lastProbability = 0;
  // predict can take in an image, video or canvas html element

  const prediction = await model.predict(imageModel);
  console.log(prediction);
  for (let i = 0; i < maxPredictions; i++) {
    if (prediction[i].probability.toFixed(2) > lastProbability)
      highestProbability = i;
    lastProbability = prediction[i].probability.toFixed(2);
  }
  const className = prediction[highestProbability].className;
  let classNameShow = "";
  if (language == "ENG") {
    for (let i = 0; i < className.length; i++) {
      if (className[i] == ",") break;
      classNameShow += className[i];
    }
  } else if (language == "PORT") {
    let auxCont = 0;
    for (let i = 0; i < className.length; i++) {
      if (className[i] == ",") {
        auxCont++;
      } else if (auxCont == 1) classNameShow += className[i];
    }
  } else if (language == "SPA") {
    let auxCont = 0;
    for (let i = 0; i < className.length; i++) {
      if (className[i] == ",") {
        auxCont++;
      } else if (auxCont == 2) classNameShow += className[i];
    }
  }
  labelContainer.childNodes[0].innerHTML = classNameShow;
}
Enter fullscreen mode Exit fullscreen mode

Alt Text
Alt Text

Uploading images

  • Let’s move to the upload part. 
function openUploadImage() {
  //Showing elements
  document.getElementById("inp").className = "";
  document.getElementById("canvas").className = "";

  //Changing button text
  if (language == "ENG") upButton.textContent = "Close";
  else if (language == "PORT") upButton.textContent = "Fechar";
  else if (language == "SPA") upButton.textContent = "Cerrar";
}
function closeUploadImage() {
  labelContainer = document.getElementById("label-container");
  let canvas = document.getElementById("canvas"),
    input = document.getElementById("inp");

  //Hiding input
  input.className = "d-none";
  input.value = null;

  //Removing Label
  labelContainer.className = "d-none";
  if (labelContainer.children.length > 0)
    labelContainer.removeChild(labelContainer.children[0]);
  canvas.className = "d-none";

  //Clear canvas
  const context = canvas.getContext("2d");
  context.clearRect(0, 0, canvas.width, canvas.height);

  if (language == "ENG") upButton.textContent = "Upload Image";
  else if (language == "PORT") upButton.textContent = "Enviar imagem";
  else if (language == "SPA") upButton.textContent = "Cargar imagen";
}
Enter fullscreen mode Exit fullscreen mode
  • Now, we will implement the upload image handler, the process is similar to the previous one. Basically, we caught the image on the input file, drew it into a canvas and then requested the Teachable Machine prediction. 
//Uploading Image

document.getElementById("inp").onchange = function (e) {
  var img = new Image();
  img.onload = draw;
  img.onerror = failed;
  img.src = window.URL.createObjectURL(this.files[0]);
};
async function draw() {
  var canvas = document.getElementById("canvas");
  canvas.width = this.width;
  canvas.height = this.height;
  var ctx = canvas.getContext("2d");
  ctx.drawImage(this, 0, 0);

  const modelURL = URL + "model.json";
  const metadataURL = URL + "metadata.json";

  model = await tmImage.load(modelURL, metadataURL);
  maxPredictions = model.getTotalClasses();

  labelContainer = document.getElementById("label-container");
  labelContainer.appendChild(document.createElement("div"));

  labelContainer.className = "";
  await predict(canvas);
}
function failed() {
  console.error("The provided file couldn't be loaded as an Image media");
}
Enter fullscreen mode Exit fullscreen mode

Alt Text

Language Handler

Alt Text

For the language handling part, we created a button and a counter, then, every time the button is clicked it increases the counter. When the counter reaches the maximum number of languages implemented it is zeroed. With conditionals we can check what is the current index and change the texts exhibited on the page to its respective language.

function languageHandler() {
  languageCont += 1;
  if (languageCont == 0) {
    language = "ENG";
  } else if (languageCont == 1) {
    language = "PORT";
    document.getElementById("lang").textContent = "Português";

    if (camButton.textContent == "Start Webcam") {
      camButton.textContent = "Começar Webcam";
    } else {
      camButton.textContent = "Parar";
    }

    if (upButton.textContent == "Upload Image") {
      upButton.textContent = "Enviar imagem";
    } else {
      upButton.textContent = "Fechar";
    }
  } else if (languageCont == 2) {
    language = "SPA";
    document.getElementById("lang").textContent = "Español";
    if (camButton.textContent == "Começar Webcam") {
      camButton.textContent = "Comenzar Webcam";
    } else {
      camButton.textContent = "Detener";
    }

    if (upButton.textContent == "Enviar imagem") {
      upButton.textContent = "Cargar imagen";
    } else {
      upButton.textContent = "Cerrar";
    }
  } else {
    language = "ENG";
    document.getElementById("lang").textContent = "English";
    if (camButton.textContent == "Comenzar Webcam") {
      camButton.textContent = "Start Webcam";
    } else {
      camButton.textContent = "Stop";
    }

    if (upButton.textContent == "Cargar imagen") {
      upButton.textContent = "Upload Image";
    } else {
      upButton.textContent = "Close";
    }
    languageCont = 0;
  }
}
Enter fullscreen mode Exit fullscreen mode

Alt Text
Alt Text

Video demo

Below is a quick video displaying the components of our app

Closing remarks

Initially, this project seemed to be a challenge to us because we had never got in contact with SashiDo when we started the development. However, the motivation and support SashiDo gave to us were the actual fuel we needed to put all our ideas into practice.

We very much recommend SashiDo to everyone interested in learning more about back-end development since it was a very great asset for this project. Teachable Machine was also a great feature we have the opportunity to work on and we certainly will use it again on future projects.

We also would like to thank SashiDo and Veselina Staneva for all the support they provided to us. We really hope this tutorial was very meaningful to all you readers!

A special thanks to my friend Aluisio @amorim33 who wrote this tutorial with me and gave a very huge contribution in all parts of our project, especially in the web-app code.

OBS: We also encourage all you readers to participate in hackathons because we only get accepted into this internship due to a hackathon we won with our friends. These competitions are a very good way to learn and get insights in lots of programming languages as well as to meet people and make friends from all around the world.

Useful links

Project in github
Web-app link
SashiDo
SashiDo Getting Started Guide
Teachable Machine
Parse docs
Teachable Machine Node
The Awesome Teachable Machine List

Top comments (1)

Collapse
 
veselinastaneva profile image
Vesi Staneva •

👏👏👏

Read next

ahgsql profile image

Building a Better RAG: A Practical Guide to Two-Step Retrieval with LangChain

ahgsql -

marcindigna profile image

Modern Data Quality: Navigating the Landscape

Marcin Chudeusz -

bytehide profile image

How to Use GitHub Copilot in Visual Studio 2024

ByteHide -

luca1iu profile image

Active Directory (AD) vs Azure Active Directory (AAD)

Luca Liu -