Machine Learning Filters: How social media incorporates Machine Learning

Sometimes, in the special moments of boredom, you may play around with Instagram, Snapchat or Tiktok filters just for the fun of it — putting on digital-makeup, fake freckles, sticking out your tongue to become a cute dog or swapping your face with a celebrity or your friend. And maybe sometimes, in those special moments, you may wonder how all of this works on a technical level. How all kinds of social media manage to match your face with the filters.

Brief History
The history of Machine Learning Filters has begun with different face changing startups developing the face modifying technology. Basically, those startups were simultaneously working on an application that allowed users to modify their facial features on photos and videos and during video calls. Around 2015 this application was adapted by various social media apps.

Those filters that can change facial features are part of the huge and cutting edge field of computer vision. Computer vision creates a 3D space from image data with Image processing technology. With Image processing, mathematical operations are performed on each individual pixel on the provided image to transform a picture. Computer vision technology is used not only by Snapchat, but also by other social media apps like TikTok, Instagram, Asian face modifier Snow and others.

To create new filters and digital face modifiers, Meta Studio and Lens Studio technologies were developed. Those studios are a tool for creating different lenses and filters. It allows you to create both Face Lenses for the front camera experiences and World Lenses for the rear camera experiences. Any person: from a 3D professional to a Photoshop amatuer — can use this tool to create their own customized filter.

From an outside perspective, it sounds almost futuristic. But, surprisingly, it can be even more advanced. Last June the newer Machine Learning-based technology was launched. It's an update to Meta and Lens Studios that allows the developers to use Machine Learning algorithms to create more detailed, realistic and vivid filters.

What Machine Learning Filters do
Broadly speaking, Machine Learning filters can:

1. create and train a neural network in an external tool (these are Tensorflow or Pytorch)
2. insert the model created in Meta and Lens Studio projects
3. test the filter with any smartphone
4. distribute the created Lens to social media users

How Machine Learning Filters work
Engineers, development teams, Photoshop hobbyists and other casual creators can upload custom Machine Learning models directly into social media apps. Surely, these models must be compatible with the ONNX model format or any other format suitable for specific apps but it’s a rather simple and convenient way to drop your customized Lens.

Machine Learning filters manage not only Computer Vision algorithms but also others, for example Style-Transfer, and other implementable algorithms Filters Development studios came up with. They can be used as templates for the potential filters.

There are 6 different templates that are available (via Jupyter Notebook) and can be used for a number of cases:

1. Classification
   That is an algorithm that is able to recognize different particular situations (for example, a person wearing glasses or a person having a textured cultural hairstyle) and consequently carry out some action.

2. Object Detection
   This algorithm can recognize the presence of an object the camera is shooting. The algorithm has a car and food detection model and can visually call it out.

3. Style Transfer
   This algorithm allows you to transform images by applying a particular graphic or art style (for example, transform the camera feed into a Van Gogh painting or a comic book style)

4. Custom & Ground Segmentation
   This algorithm is used for identifying areas or objects and replacing them with custom textures and objects (for example, the algorithm comes with a pizza segmentation texture and uses Material Editor to make it look sizzling).

5. Keyword Detection
   This algorithm is used for basic audio related Machine Learning models. Given a spectrogram analysis of the audio, it can return the probability of a spoken word.

6. Multi Object Detection
   This algorithm can detect 7 different classes of objects: cat, dog, potted plant, TV, car, bottle and cup.

Those algorithms can be used by anyone as a starting point to create your own Lens. Those algorithms are flexible and can be modified so you can use them as a basis for creating your own template.

What Machine Learning Filters contribute to Machine Learning and what they mean for ML engineers and developers
Machine Learning Lenses are a promising distribution channel with great functionality. Above that, Machine Learning Filters have other advantages and props:

• Machine Learning filters reduce barriers to entry
  As was previously stated, with Machine Learning filters, anyone can drop custom neural networks into Filters Development studios and then distribute it to millions of social media users and it's a promising and exciting possibility. With this technology it's no longer required for Machine Learning Development teams and engineers to create an entire mobile application from scratch to develop an algorithm.

• Machine Learning filters allow easier experimentation
  Without the need for teams and engineers to work through a full app release cycle to see their on-device models in action, it will be much easier to experiment with and test brand- and product-based Lenses powered by immersive Machine Learning features.

• Machine Learning filters democratize on-device Machine Learning
  For an overall development of Machine Learning technology it is important to democratize said technology. One of early developers and creators of Machine Learning Lenses technology, Hart Woolery noted in one of his interviews that the potential of Machine Learning filters reminds him, in some ways, of how YouTube helped democratize video creation.

Next Step
With all of it being said, Machine Learning Filters are a great leap forward for Mobile Machine Learning technology and AI technology overall. So from now on, when you're playing around with TikTok or Snapchat filters, remember that it's a cutting edge technology and, in a way, you're incorporating Machine Learning in your everyday life just by pressing a couple of buttons and having fun.