CarolineBaillie

Posted on Dec 3, 2020 • Edited on Aug 31, 2021

Fish Classification iOS App with SashiDo and Teachable Machine

#swift #machinelearning #ios #tutorial

I don’t consider myself A Fisher, but I do enjoy fishing occasionally. I thought it would be fun to create a classification and storage app. Using Xcode to create this app, Teachable Machine to identify the type of fish, and SashiDo to manage all the data and backend code, I finally have a finished product that I would like to share with you all. All these tools (Xcode, Teachable Machine, and SashiDo) enable you to create amazing apps, and I wanted to provide you the basic knowledge to get started. Before advancing, I recommend you have knowledge of swift as I will mainly be going over how to use SashiDo and how to integrate a Teachable Machine model. Using my app as an example, I want to show you the steps to creating your own and some important things I learned from creating mine.

First, for a quick overview of my app, once logged in, you can either take a picture, upload a picture, browse your catches, or browse others' catches. Below is a quick video displaying the components of my app:

Teachable Machine
Classification
Outline
SashiDo and Parse
Registration
- Signup
- Login
- Logout
SessionManager
Closing Remarks

Using Teachable Machine

Although it was not my actual first step, creating and importing the model really should be the first step. Unfortunately, Teachable Machine does not offer an exportation method using swift, so I had to find some other way to access the model. There are many different solutions, but I chose to export the model using Tensorflow Lite (which is made for Android development).

Here are steps to creating and downloading the model:
     1. Create and train your model with Teachable Machine.
     2. Click export model.
     3. Click on Tensorflow Lite at the top (right-most option).
     4. Download model (should be Floating point for Model conversion type).

This will download two files onto your computer (model_unquant.tflite and labels.txt). Save them for later as we will be inserting them to our Xcode project.

To actually get started with Xcode, generally you would create your own project and begin working from scratch, but in this case, because you must use the .tflite model in Xcode, it is best to download and use a template.

These are the following steps to set up the template:
     1. Download CatVsDogSample.
     2. In your downloaded file go to TensorFlow Deployment/Course 2 - TensorFlow Lite/Week 3/ Examples/iOS Apps/.
     3. You should see 3 files (cats_vs_dogs, image_classification, and object_detection); move the cats_vs_dogs folder into a separate place outside of all the other folders.
     4. Delete everything else (you only want the cats_vs_dogs folder).
     5. In terminal type sudo gem install cocoapods.
     6. Inside the cats_vs_dogs folder, right click on CatVsDogClassifierSample.xcworkspace and open with Xcode (you must open .xcworkspace, NOT .xcodeproj).
     7. Open the bottom-most folder Pods and open the Podfile.
     8. Add pod ‘Parse’ so that the contents are as such:

     9. Save and exit Xcode.
     10. In terminal navigate to the downloaded CatVsDogSample:
         - Use cd folderName.
         - You should see 6 other folders/files like this:

11. Type pod init.
12. Type pod install.

Now that you have downloaded everything, here are the steps to importing your model:
     1. Open up the CatVsDogClassifierSample.xcworkspace again.
     2. Under StoryBoards/Main.storyboard, delete the pre-existing ViewController.
     3. Under ViewController, delete the ViewController.swift file.
     4. Right click on the outermost CatVsDogClassifierSample.xcworkspace and click Add Files to “CatVsDogClassifierSample”...
     5. At the top select your model_unquant.tflite and click Add:

     6. Open ModelDataHandler.swift.
     7. Towards the top, look for this function:

     8. Replace “converted_model” with “model_unquant”.
     9. Scroll down a bit until you see this line of code:

     10. Replace “Cat”, “Dog” with your own labels.
         - To do this you must open the labels.txt file and IN THE SAME ORDER insert them into the array.

With the model imported, I'll show you how to use it for classification.

Classifying the image

There are several different ways you can get the image you want to classify (picture, upload, UIImage View, etc.), so in this example, we are just assuming that you have stored the image in a variable called image. This is the code to run the model:

let pixelBuffer = image.pixelBuffer()!
let inferenceResults = modelDataHandler?.runModel(onFrame:pixelBuffer)
guard let inference = inferenceResults?.first else {
   return
}

Now the classification label should be stored in inference.label, and you can choose what you want to do with this (display it on the controller, save it to the database, pass it to the next view controller, etc.).

Outline

One of the most important things to do when creating an iOS app is designing an outline for all the controllers you will need. This will look different for every app, but for mine, the outline looked like this:

With an outline in mind, it is now time to actually create these controllers.
For my specific app, I had 3 different types of controllers (viewController, TableViewController, and CollectionViewController), all highlighted in the specific colors (black, orange, blue) respectively.

Here are the steps to creating the controllers and their files:
     1. Add necessary controllers (View Controller, Table View Controller, and Collection View Controller, etc.).
         - Make sure you choose the controller option.
     2. Add navigation controller:
         - Set navigation controller as the initial view controller.
         - Set signup/login page (or whatever you want the first page of your app to be) as the navigation controller’s root view controller.
         - Add this code inside the settings class (or page/file after the signup and login pages).
         - Make sure 'settings' is changed to whatever you called your controller.

//keep with navigation controller
var objVC: UIViewController? = storyboard!.instantiateViewController(withIdentifier: "settings")
var aObjNavi = UINavigationController(rootViewController: objVC!)

     3. Add other elements (buttons, image views, labels, etc.).
     4. Create connections/paths through storyboard.
         - Hold control, click on button, drag to controller that you want to create path between.
         - Not all paths will be made through storyboard as some will be made through code.
         - The following were paths I did not connect through the storyboard: login and signup settings, upload and pic input info page, and table view collectionview description page.
     5. Add constraints to elements.

Here what my storyboard ended up looking like:

     6. Create files for all of the controllers:
         a. Right click in the ViewController folder and press New File...
         b. Choose swift file and click Next.
         c. Name the file and click Create.
         d. Write the necessary code:
             i. At the top of each file, import UIKit and import Parse (although you won’t need both of these for all controllers, it doesn’t hurt to have it).
             ii. Create a class with a name matching the name of the file and a type matching the type of the controller.
             iii. Inside the class override the viewDidLoad function and inside write super.viewDidLoad().
             iv. Eg. The code for settings.swift meant to connect with a viewController should look like this:

import Foundation
import UIKit
import Parse

class settings: UIViewController {
    override func viewDidLoad() {
        super.viewDidLoad()
    }
}

     7. Connect each file to its corresponding controller:
         a. Click on the top of a view controller.
         b. Click on the identity inspector on the top right.
         c. Under Custom Class and next to Class, click on the blue arrow.
         d. Choose the corresponding file name.
         e. Copy the name of the file (in this example settings) and paste it into StoryboardID as we may need to reference it later.

Using SashiDo and Parse

Next we have to link our app to SashiDo:
     1. Go to your SashiDo dashboard.
     2. Click Create New App and follow the instructions.
         - Once your app has been created you should be shown a page saying Connect Applications With SashiDo.
     3. Making sure swift is selected, copy the 6 lines at the bottom:
         - If you are not shown this page or accidentally move past it, you can find it again by clicking Getting Started at the top of the menu on the left.
         - If you had any trouble with this section, refer to the getting started guide.

4. Next go back to Xcode and open AppDelegate.swift.
5. Inside func application, paste the code you copied.

In order to communicate with SashiDo, we are going to be using Parse documentation. I will be going over some of the functions and calls I used, but it is also beneficial to refer to the Parse iOS documentation. Additionally, in order to gain a more in depth understanding, I found this video playlist on using parse for iOS development very useful.

Registration

One thing that is consistent and important in most apps are the login and signup pages. In this section, I will lead you through creating your own registration pages with Sashido and Parse.

Signup

     1. Connect text inputs (as outlet) and button (as action) to file.
         - If you do not know how to do this, refer to this thorough guide.
     2. Inside the button action function (I called it signupToggled) create a user.
         - Parse enables you to do this through a few simple lines of code.
         - See screenshot below for code.
     3. Save the user and use the navigation controller to go to the next page.
         - See code below, but if you don’t understand how to push the next page, visit this link #18.
     4. Override the function touchesBegan to dismiss the keyboard when you tap somewhere else.

import Foundation
import UIKit
import Parse

class signup: UIViewController {
    @IBOutlet weak var emailTextField: UITextField!
    @IBOutlet weak var usernameTextField: UITextField!
    @IBOutlet weak var passwordTextField: UITextField!
    @IBOutlet weak var buttonStyle: UIButton!

    override func viewDidLoad() {
        super.viewDidLoad()
    }

    //get rid of keyboard when touch screen
    override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
        usernameTextField.resignFirstResponder()
        passwordTextField.resignFirstResponder()
        emailTextField.resignFirstResponder()
    }

    @IBAction func signupToggled(_ sender: Any) {
        self.showSpinner()
        //check username and password field are not empty / nil
        let user = PFUser()
        user.username = usernameTextField.text!
        user.password = passwordTextField.text!
        user.email = emailTextField.text!
        user.signUpInBackground { (result, error) in
            if error == nil && result == true {
                //successfully signed up in
                let cUser = currentUser(username: user["username"] as! String, id: user.objectId!)
                sessionManager.shared.user = cUser
                //next screen
                let secondViewController = self.storyboard?.instantiateViewController(withIdentifier: "settings") as! settings
                self.navigationController?.pushViewController(secondViewController, animated: true)

            } else {
                //error display error message
            }
        }
    }

}

Note: The code about cUser is just to store a local variable that keeps track of the current user. I stored it in sessionManager, which is a file that I can access globally containing all of my database query functions.

Login

1. This is the same as the signup page, but change the code from signing up to logging in as shown below:

import Foundation
import UIKit
import Parse

class login: UIViewController {
    @IBOutlet weak var usernameTextField: UITextField!
    @IBOutlet weak var passwordTextField: UITextField!
    @IBOutlet weak var buttonStyle: UIButton!

    override func viewDidLoad() {
        super.viewDidLoad()
    }

    //get rid of keyboard when touch screen
    override func touchesBegan(_ touches: Set<UITouch>, with event: UIEvent?) {
        usernameTextField.resignFirstResponder()
        passwordTextField.resignFirstResponder()
    }

    @IBAction func loginToggled(_ sender: Any) {
        PFUser.logInWithUsername(inBackground:usernameTextField.text!, password:passwordTextField.text!) {
          (user, error) -> Void in
          if user != nil {
            var cUser = currentUser(username: user?["username"] as! String, id: user?.objectId as! String)
            sessionManager.shared.user = cUser
            // Do stuff after successful login.
            let secondViewController = self.storyboard?.instantiateViewController(withIdentifier: "settings") as! settings
            self.navigationController?.pushViewController(secondViewController, animated: true)
          } else {
            // The login failed
          }
        }
    }
}

Logout

To logout in Parse, all you do is add this line of code:

PFUser.logOut()

Session Manager

When creating the app, I had two major issues surrounding database querying or saving that I solved using a file called sessionManager.swift. The first was organization and the second was timing. Basically, sessionManager is a file (and class) that I created where I can write and store all my database query and saving functions while still being able to access them in other files. By doing this, I was able to easily keep track of my functions, solving my first problem. My second problem was a little more complicated. When I tried retrieving data from SashiDo without sessionManager, there was a timing issue and the page would sometimes load before the data got back. Because of this, I decided to store information locally when the user signed in. This enabled me to access information on the catches of the user very quickly and easily. sessionManager was important in that process because all the functions were in one place and accessible throughout the app - I could also create arrays that stored the information there, too. If the user uploaded a new catch (or page as I called them), I would save the information to the database but also append it to the necessary arrays. This is generally a good practice, so I want to show you how to set up sessionManager, some examples of using it, and some examples of communicating with SashiDo.

Setting up sessionManager

     1. Create a new file and call it sessionManager.
     2. import Parse at the top.
     3. Create a class of type NSObject.
     4. Write static let shared = sessionManager().
     5. Then override the init function.

import Foundation
import Parse
class sessionManager: NSObject {
    static let shared = sessionManager()
    override init() {
       super.init()
   }
}

Now you can write whatever code you want inside this class. The two examples I am going to show you are retrieving information from the database and saving information in the database.

requestGetPages

Before getting started, it might be beneficial to take a look at Parse’s basic querying and image querying.

This is a function I wrote that gets all the pages (or catches) of a user with the matching ID:
     1. Define function (I called it requestGetPages).
     2. Clear the array that local information is stored in.
         - You clear it at the beginning because this should only be called once for each user, so if it is called again it means they have logged out then logged back in.
     3. Create query and define its constraints.
     4. Find objects in the database, check for errors, and loop through returned objects.
     5. Save or use the information somehow.
         - I created an object of class page and appended it to the array.
         - The class page is one that I created similar to sessionManager and is of type NSObject as well.
         - To learn more about creating classes in swift look at this site.

func requestGetPages (completion:@escaping (_ success:Bool) -> ()) {
    AllPages.removeAll()
    let query = PFQuery(className: "page")
    query.whereKey("userID", equalTo:user.id)
    query.findObjectsInBackground { (objects, error) in
        // no errors
        if error == nil {
            // if there are objects in the array
            if let returnedObjects = objects {
                // loop through all objects in array
                for object in returnedObjects {
                    let file = object["Image"] as! PFFileObject
                    file.getDataInBackground { (data, error) in
                        if error == nil {
                            if let imageData = data {
                               let image = UIImage(data: imageData) //can then display on screen
                               let fishData = page(fishType: object["fishType"]! as! String, Image: image!, desc: object["description"]! as! String, location: object["location"]! as! String, catchDate: object["catchDate"]! as! String, weight: object["weight"]! as! String, dimensions: object["dimensions"]! as! String)
                                    self.AllPages.append(fishData)
                            }
                        }
                    }
                }
            }
            completion(true)
        }
        else {
            //return false completion if fails
            completion(false)
        }
    }
}

Note: The functions I used for this are completion functions, meaning they will not move on until the data query has been completed.

6. In my settings.swift file (or whenever you want to call this function), INSIDE viewDidLoad you would put the following code:

sessionManager.shared.requestGetPages { (success) in
   if success {
      // query has completed - do what you want
   }
}

requestCreateNewPage

Before getting started it might be beneficial to take a look at saving data and images with Parse.

This is a function I wrote that saves a new page:
     1. Define function (requestCreateNewPage).
     2. Create a new object and add all necessary ‘columns’.
     3. Resize image
         - This step is not necessary, but I found I was trying to save too much information to the database and this was a solution to that.
     4. Save the information in the database and do whatever else you would like.
         - I appended the new page to the AllPages array.

func requestCreateNewPage (page:page, completion:@escaping (_ success:Bool) -> ()) {
    let pg = PFObject(className:"page")
    pg["fishType"] = page.fishType
    pg["description"] = page.desc
    pg["location"] = page.location
    pg["catchDate"] = page.catchDate
    pg["userID"] = user.id
    pg["weight"] = page.weight
    pg["dimensions"] = page.dimensions

    // reducing image size
    let image = page.Image
    let actualHeight:CGFloat = image.size.height
    let actualWidth:CGFloat = image.size.width
    let imgRatio:CGFloat = actualWidth/actualHeight
    let maxWidth:CGFloat = 1024.0
    let resizedHeight:CGFloat = maxWidth/imgRatio
    let compressionQuality:CGFloat = 0.5
    let rect:CGRect = CGRect(x: 0, y: 0, width: maxWidth, height: resizedHeight)
    UIGraphicsBeginImageContext(rect.size)
    image.draw(in: rect)
    let img: UIImage = UIGraphicsGetImageFromCurrentImageContext()!
    let imageData:Data = img.jpegData(compressionQuality: compressionQuality)!
    UIGraphicsEndImageContext()
    let imageFinal = UIImage(data: imageData)!

    // preping to save image
    let imgData = imageFinal.pngData()
    let imageFile = PFFileObject(name:"image.png", data:imgData!)
    pg["Image"] = imageFile

    pg.saveInBackground { (succeeded, error)  in
        if (succeeded) {
            // The object has been saved.
            self.AllPages.append(page)
            self.UpdateFishCat { (success) in }
            completion(true)
        } else {
            // There was a problem
            completion(false)
        }
    }
}

     5. In my info input page, I have an action of the submit button:
         a. First, I created a new page element using the template of a class page.
         b. Then I called my function and passed in the page I created.
         c. Finally, if it is successful I move onto the next controller.

@IBAction func submitToggled(_ sender: Any) {
    //create new page
    var newPage = page(fishType: inference, Image: image, desc: descriptionField.text, location: locationField.text!, catchDate: dateField.text!, weight:weightField.text!, dimensions: dimensionsField.text!)
    sessionManager.shared.requestCreateNewPage(page: newPage) { (success) in
        if success {
            //go to diff viewController
            let customViewController = self.storyboard?.instantiateViewController(withIdentifier: "CatchTypeTable") as! CatchTypeTable
                self.navigationController?.pushViewController(customViewController, animated: true)
       }
       else {
            //error
       }
    }
}

Closing Remarks

When I started this project, I had never before interacted with SashiDo and Parse. Like most new things, it was challenging at first, but after sifting through lots of resources and trying to implement the concepts myself, it began to get easier and my understanding grew. I really enjoyed creating this project, and I hope you can learn from the knowledge and skills I gained in doing so. There are a lot of great tips and good practices within this tutorial, and I would love to see posts about other projects with Teachable Machine and SashiDo! Overall, SashiDo is a great backend platform that was really useful in creating this app, and I really recommend it for anyone who is interested in app development. Additionally, Teachable Machine makes machine learning and classification super easy and is great for cool projects like this one!

A big thanks to SashiDo and to all you readers! I hope this was helpful and can’t wait to see where your journey takes you!