Latent Variables in Computer Vision

Moses Odhiambo — Wed, 25 Aug 2021 08:23:19 +0000

Imagine prisoners chained together in a cave, with all they can see being a wall in front of them. Behind the prisoners is a fire, and between the fire and the prisoners are people carrying sculptures that cast shadows on the wall. The prisoners watch these shadows, believing them to be real.

If an object (a book, let us say) is carried past behind them, and it casts a shadow on the wall, and one of them says “I see a book", he thinks that the word “book” refers to the very thing he is looking at. But he would be wrong. He’s only looking at a shadow. The real referent of the word “book” he cannot see. To see it, he would have to turn his head around.

In a lot of computer vision tasks, images to neural networks are like shadows to the prisoners. In order to gain an understanding of an image's contents, they need to find and use variables that are not exactly observable to them but are the true explanatory factors behind the image. These factors are known as latent variables.

What is a Latent Space

A latent space or vector is basically a distribution of the latent variables above. It is also commonly referred to as a feature representation.

Think of a latent vector as a collection of an image's 'features', I.e. variables that describe what is going on in an image, such as the setting (medieval or modern), the time of day, and so on. This is not exactly how it works - it is just an intuition. The idea is that the latent variables represent high level attributes, rather than raw pixels with little meaning.

Where are Latent Variables Used

Latent variables can be used when connecting computer vision models to models from other domains that do not deal with image data. For example, a common task in natural language processing is image captioning. To generate a caption for an image, an NLP model would require the image's latent variables. That is where they get the understanding necessary to describe what is going on in the image.

Latent variables are also used in image manipulation. These variables when learned well can be used to adjust higher level properties about an image. A common application of this is the creation of deepfakes.

Face recognition is a problem that heavily leverages latent variables. A model can be trained on face recognition data and used to obtain latent variables for every face it encounters. The latent space can then be compared to that of another image to see if it matches.

Another application of latent vectors is debiasing of computer vision systems. By learning a latent space a computer vision model will gain a high level understanding of the data and can be used to tell which features are underly represented, for instance, in the context of face recognition, faces of a certain race or skin complexion.

How is a Latent Space Learned?

A popular model used to learn a latent space for images of a given distribution is the autoencoder, which consists of two parts - an encoder and a decoder. The encoder portion of the network maps an image to its latent space and the decoder samples from this latent space to reconstruct the original image. By training the model end to end to a point where the reconstruction is as close as possible to the original image, the latent space would be learned.

Feature Disentanglement

Ideally, we want learned latent variables to be as independent as possible and not correlated with each other at all such that when we vary a given latent variable, only the aspect of the image the variable represents is changed. There are many ways to enforce this - most of them are beyond the scope of this article.

TinyML: Deploying TensorFlow models to Android

Moses Odhiambo — Tue, 24 Aug 2021 17:21:30 +0000

What is TinyML?

Tiny machine learning (TinyML) is a field that focuses on running machine learning (mostly deep learning) algorithms directly on edge devices such as microcontrollers and mobile devices. The algorithms have to be highly optimized to be able to run on such systems since most of them are low powered.

Wait, what do you mean by 'edge devices'?

An edge device is the device which makes use of the final output of machine learning algorithms, for instance, a camera that displays the result of image recognition, or a smartphone that plays speech synthesized from text. Most practitioners run machine learning models on more powerful devices, then send the output to edge devices, but this is starting to change with the advent of TinyML.

Why TinyML?

The need to run machine learning directly on edge devices and the convenience that comes with this has made TinyML become one of the fastest growing fields in deep learning.

How does one go about deploying ML to edge devices?

Train a machine learning model on a more powerful environment such as a cloud virtual machine or a faster computer.
Optimize the model, say, by reducing the number of parameters, or by using low precision data types such as 16 bit floats. This will make the model smaller and the inference faster and more power efficient at the cost of accuracy, which is a compromise you'll have to take.
Run the model 'on the edge'!

TensorFlow Lite Quick Start

TensorFlow Lite is TensorFlow's take on TinyML.

Converting a saved model from TensorFlow to TensorFlow Lite

import tensorflow as tf
model=tf.keras.models.load_model("/path_to_model.h5")
converter = tf.lite.TFLiteConverter.from_keras_model(model)
tflite_model = converter.convert()
open("/tflite_model.tflite", "wb").write(tflite_model)

As you can see, it only takes a few lines of code 😊.

Running a TensorFlow Lite model in TensorFlow Lite's Python Interpreter

import tensorflow as tf
interpreter = tf.lite.Interpreter(model_path="/tflite_model.tflite")    #initialize interpreter with model
interpreter.allocate_tensors()
input_details = interpreter.get_input_details()
output_details = interpreter.get_output_details()

inputs = []  #list of input tensors
for index,item in enumerate(inputs):
    interpreter.set_tensor(input_details[index]['index'], item)

interpreter.invoke()    #run model

outputs = [] #output tensors
num_outputs = len(output_details) #number of output tensors
for index in range(num_outputs):
    outputs.append(interpreter.get_tensor(output_details[index]['index']))

Running a TensorFlow Lite model in an Android application

1. Create a new Android Studio Project

2. Import the model into Android Studio

Copy the .tflite model to app/src/main/assets/ - create the assets folder if it does not exist.

3. Import TensorFlow Lite into your project

Add the following dependency to your app-level build.gradle

implementation 'org.tensorflow:tensorflow-lite:+'

4. Load Model

Load the .tflite model you placed in your assets folder as a MappedByteBuffer.

private MappedByteBuffer loadModelFile(Context c, String MODEL_FILE) throws IOException {
    AssetFileDescriptor fileDescriptor = c.getAssets().openFd(MODEL_FILE);
    FileInputStream inputStream = new FileInputStream(fileDescriptor.getFileDescriptor());
    FileChannel fileChannel = inputStream.getChannel();
    long startOffset = fileDescriptor.getStartOffset();
    long declaredLength = fileDescriptor.getDeclaredLength();
    return fileChannel.map(FileChannel.MapMode.READ_ONLY, startOffset, declaredLength);
}

model = loadModelFile(this, 'name_of_model.tflite')

5. Initialize Interpreter

try {
    interpreter = new Interpreter(model);
}
catch (IOException e) {
    e.printStackTrace();
}

6. Run Model

Object[] inputs = {input1, input2, ...} //the objects in inputs{} are jagged arrays - what in TensorFlow would be considered tensors

Map<Integer, Object> outputs = new HashMap<>(); //same for outputs
outputs.put(0, output1);    //add outputs to the map

interpreter.runForMultipleInputsOutputs(inputs, outputs);   //get inference from interpreter

And there you go.

Sample project

Here is source code for a GAN deployed to an Android app with TensorFlow Lite. Here is the android app for you to play with.

DEV Community: Moses Odhiambo